Skip to main content

Hadoop sharp edges annoy

1. Pig vs. Hive

You cannot use Hive UDFs in Pig. You have to use HCatalog to access Hive tables in Pig. You cannot use Pig UDFs in Hive. Whether it's one little extra functionality I need while in Hive, but don’t really feel like writing a full-on Pig script or it's the “gee, I could easily do this if I were just in Hive” while I’m writing Pig scripts, I frequently think, “Tear down this wall!” when I’m writing in either.

2. Being forced to store all my shared libraries in HDFS

This is a recurring theme in Hadoop. If you store your Pig script on HDFS, then it automatically assumes any JAR files will be there as well (I’m working on fixing that myself). This general theme repeats in Oozie and other tools. It's usually sensible, but at times, having an organization-wide forced shared library version is painful. Besides, more than half the time, these are the same JAR files you installed everywhere you installed the client, so why store them twice?

3. Oozie

Debugging you is not fun, so the docs have lots of examples with the old schema. When you get an error, it usually has nothing to do with whatever you did wrong. It may be a “protocol error” for a configuration typo or a schema validation error for a schema that validates using the schema validator but fails on the server. To a great degree, Oozie is like Ant or Maven, except distributed, with no tooling and a bit brittle.

4. Error messages

You’re joking, right? Speaking of error messages. My favorite is the one where any of the Hadoop tools say, “failure, no error returned,” which translates to “something happened, good luck finding it.”

5. Kerberos

If you want to secure Hadoop in a way that was relatively thought out, you get to use Kerberos. Remember Kerberos and how much fun and antiquated it is? So you go straight LDAP, except that nothing in Hadoop is integrated: no single sign-on, no SAML, no OAuth, and nothing passes the credentials around (instead, it re-authenticates and re-authorizes). Even more fun, each part of the Hadoop ecosystem wrote its own LDAP support, so it's inconsistent.

6. Knox

Because writing a proper LDAP connector needs to be done at least 100 more times in Java before we get it right. Gosh, go look at that code. It doesn’t really pool connections properly. In fact, I kind of think Knox was created out of a zeal for Java or something. You could do the same with a well-written Apache config, mod_proxy, mod_rewrite. In fact, that's basically what Knox is, except in Java. To boot, after it authenticates and authorizes, it doesn’t pass the information on to Hive or WebHDFS or whatever you’re accessing, and gets to do it again.

7. Hive won't let me have my external table and delete it too

If you let Hive manage tables, it automatically deletes them if you drop the table. If you have an external table, it does not. Why can’t there be a “drop external table too” or something? Why do I have to do this outside if I really want to? Also, while Hive is practically evolving into an RDBMS, why doesn’t it have Update and Delete?

8. Namenode fail

Oozie, Knox, and several other parts of Hadoop do not obey the new Namenode HA stuff. You can have HA Hadoop, so long as you don’t use anything else with it.

9. Documentation

It’s cliche to complain, but check this out. Line 37 is wrong -- worse, it is wrong in every post all over the Internet. This proves that no one even bothered to run the example before checking it in. The Oozie documentation is even more dreadful, and most of the examples won’t pass schema validation on the version it's meant for.

10. Ambari coverage

I have trouble criticizing Ambari; given what I know about Hadoop architecture, it's amazing Ambari works at all. That said, where Ambari has shortcomings, they can be annoying. For example, Ambari doesn’t install -- or in some cases, doesn’t install correctly -- many items, including various HA settings, Knox, and much, much more. I’m sure it will get better, but “manually install afterward” or “we’ll have to create a puppet script for the rest” shouldn’t appear in my emails or documentation any longer.

11. Repository management

Speaking of Ambari, have you ever done an install while the Repositories were being upgraded? I have -- it does not behave well. In fact, sometimes it finds the fastest (and most out of date) mirror. It doesn't care if what it pulls down is in any way compatible. You can configure your way out of that part, but it's still annoying the first time you install incoherent pieces of Hadoop across a few hundred nodes.

12. Null pointer exceptions

I seem to find them. Often they are parse errors or other faults I've caused. That said, they still should not be exposed as NPEs in Pig, Hive, HDFS, and so on.
The response to any similar list of complaints will of course be “patches welcome!” or "hey, I’m working on it." Hadoop has come a long way and is certainly one of my favorite tools, but boy, those sharp edges annoy me.

Comments

Post a Comment

Popular posts from this blog

Real-Time Talk: Windows 10 IoT Core Background Tasks and ASP.NET Core Web Apps

Display useful information from your Windows 10 IoT Core application in an ASP.NET Core web app, essential for integrating IoT data into a solution. Windows 10 IoT background task talk with a web application using WebSockets. Problems As my path to this solution has been troublesome, I am listing here the main problems I faced so my dear readers have a better idea of dead-end streets along the way: I was not able to make the ASP.NET Core web application run under a Windows 10 IoT background service. I found no information about when or if it will be supported in the near future. ASP.NET MVC and ASP.NET Core have different SignalR implementations. I was not able to make a SignalR client for .NET Core work with SignalR hosted on a web application. I was able to make things work by directly using a WebSocket. It’s not as nice a solution as I had in my mind, but it works until things get better. Making the Background Task and Web Application Talk I worked out sim...
Post a Comment
Read more

Python and Parquet Performance

In Pandas, PyArrow, fastparquet, AWS Data Wrangler, PySpark and Dask. This post outlines how to use all common Python libraries to read and write Parquet format while taking advantage of  columnar storage ,  columnar compression  and  data partitioning . Used together, these three optimizations can dramatically accelerate I/O for your Python applications compared to CSV, JSON, HDF or other row-based formats. Parquet makes applications possible that are simply impossible using a text format like JSON or CSV. Introduction I have recently gotten more familiar with how to work with  Parquet  datasets across the six major tools used to read and write from Parquet in the Python ecosystem:  Pandas ,  PyArrow ,  fastparquet ,  AWS Data Wrangler ,  PySpark  and  Dask . My work of late in algorithmic trading involves switching between these tools a lot and as I said I often mix up the APIs. I use Pandas and PyArrow for in-RAM comput...
Post a Comment
Read more

DANGER (STILL) LURKS IN THE INTERNET OF THINGS (IOT)

All of the media, both “mainstream” and “tech”, has gushed over all of the new appliances and devices that are now in the category of what we would call the Internet of Things.  Items like home security,  home lighting, and refrigerators, to name a few. There are many advantages to having connected appliances and devices, Threats that can and will be exploited if unsuspecting users don’t secure them.   Two “layers” of security : The first layer of offering  is a security API that will provide [a way] to easily do a virtual patch, to prevent a remote attack, for example . . . the third layer is cloud: IoT cannot do anything without the cloud.  Most data is sent to the cloud and you will need to have proper protection and make sure the cloud is always available. In both situation Users are vulnerable, mostly due to their own apathy.  Users often either don’t know how to patch their own machines (and in this case, devices) or have glanced...
Post a Comment
Read more