Using Kafka for Real-Time Data Ingestion with .NET

I presented a version of this talk at NDC Oslo in 2018. You can get the recording on the NDC Youtube channel.

Click here to access demo code for this presentation.

This includes all of the console applications and Kafka scripts to work with topics.

To get the flight data we loaded and the airport data we used to enrich our data sets, check out the Data expo '09 contest's set of flight data, including the 2008 flight data and airports CSV file

The source code is licensed under the terms offered by the GPL.

Kafka Links

• This all started as an eight-part blog series.
• If you want an idea of how many queue and broker technologies are out there, Queues.io is a start.
• The technology probably most closely oriented with Kafka is Amazon Kinesis. They compare well, in that Kinesis is easier to get into, whereas Kafka scales out better.
• The official Kafka quick-start documentation is very good.
• Jun Rao explains how to determine how many partitions you need for a topic.
• Getting the number of messages in a Kafka topic.
• Getting started with Kafka and .NET.
• If you're using the Hortonworks Data Platform, note that the default Kafka broker port is 6667.
• Datadog has a series on monitoring Kafka servers.
• Cloudera has some good administration hints, including an article on optimizing producers and consumers.
• The Kafka Offset Monitor gives you an idea of how quickly your consumers are going through topics.
• Kafka Tool is an interesting administrative GUI for Kafka.
• Gary Kaiser digs into TCP window size, which is vital for understanding how to optimize network throughput.
• Yeva Byzek has a whitepaper on tuning Kafka deployments.

Kafka .NET Providers

At this point, I highly recommend using the Confluent .NET client for Kafka due to its official support. Nevertheless, I think it's valuable to have an idea of other Kafka libraries which are available, as until late 2016, there were several viable alternatives.

• Confluent's Kafka .NET client, which is what I used for this project.
• Kafunk is another F#-friendly Kafka provider, which developers at Jet maintain.
• Kafka Sharp
• Franz, a Kafka provider written in F#.
• Kafka4net
• Microsoft's C# client for Kafka
• If you're feeling ambitious, write your own Kafka client.
• Jonathan Leaver builds a microservice which publishes and consumes from Kafka topics, all in F#. He uses the older RDKafka-DotNet provider, so some of the code is out of date, but the principles hold.

F# Links

• A big part of the Confluent client involves event handling, so it's a good topic to learn. Here is how to perform event handling in F#.
• Understanding F# exceptions.
• Jamie Dixon shows how to do list manipulation in F#.
• To improve F# performance, look into using Nessos Streams (not used in this project).
• Understanding Seq versus Stream.
• File.ReadLines is an efficient way of streaming a large file through .NET code. You can see other streaming methods in the answers to this Stack Overflow question.
• Understanding types in F#. They aren't quite the same as classes.
• The ParallelSeq (PSeq) library is great for writing multi-threaded code. Just don't use it for console output!
• F# has a JSON type provider to make JSON handling easier. I didn't use it for this project, but it can be useful for generic JSON parsing.