CATALLAXY SERVICES

Abstract

Hadoop is a hot topic right now, and Microsoft is betting that Hadoop and SQL Server will go together like peanut butter and chocolate. This presentation will spend a few minutes on the theory behind Hadoop, but the rest will be a demo-driven look at how to get a handle on Hadoop before somebody introduces it into your enterprise.

Slides

The slides are available in HTML 5 format. All modern browsers (including tablets and phones) should be able to navigate the slides successfully.

The slides are licensed under Creative Commons Attribution-ShareAlike.

Demo Code

The demonstration code is available on my GitHub repository. The solution and scripts give you an opportunity to play with Hadoop. You should first install the HDInsight Emulator.

The source code is licensed under the terms offered by the GPL. The slides are licensed under Creative Commons Attribution-ShareAlike.

Links And Further Information

Hadoop Implementations

If you want to get started with Hadoop, there are a number of pre-built sources available to you. Here are some that I can recommend through personal experience; I'm sure there are a lot of others out there (especially in the cloud services space), but these are ones I've at least played with.

Local sandboxes:

• HDInsight Emulator. This version gives you the ability to write .NET code and run Hadoop jobs. It includes a number of tools and add-ins, including YARN, Hive, and Pig.
• Hortonworks Sandbox. This is probably the easiest way to get started with Hadoop. They give you a full VM which is already installed and configured with a number of tools, such as HBase, Hive, Pig, and Hue.
• Cloudera QuickStart VM. Like the Hortonworks Sandbox, Cloudera's offering is a fully-featured single-node VM.

Online services:

• Azure HDInsight. It's fairly pricey--data nodes can run you a couple hundred dollars per month apiece--but if your team really wants to integrate .NET code with Hadoop, this is your best option.
• Elastic MapReduce. Amazon's offering ties to S3 and is extremely inexpensive: approximately 1/5 the Azure cost. If your company is integrated with Amazon already, this is a good service.

Interesting Links

In my demo, I make mention of a tool by Darrin Ward to plot by latitude and longitude on Google Maps. It also saves your work, so you can investigate more closely the results from my demo.

I make use of an excellent image from the MapReduceFoundation at Universität Passau. They have a few academic papers that I admittedly haven't read but which do sound interesting.

I make an off-hand mention of a Serializer/De-serializer (SerDe) for Hive which processes CSVs better than Hive's default. You can get csv-serde from Github and plug it into your Hadoop cluster.

Books and Other Resources

At this time, I can recommend the following books on Hadoop. Please note that with a fast-evolving technology, a book which is even a couple years old can feel aged.

• Programming Hive (review)

At this time, I have not read the following books, but I do own copies of them. As I get through these books, I'll shift the ones I recommend up to the recommended section.

• Hadoop Beginner's Guide.
• Hadoop Operations.
Pro Apache Hadoop.
• Pig Design Patterns.
• Instant MapReduce Patterns.

I learned a good deal from the Hortonworks tutorials, which include both written and video tutorials. They are a good place to start.