Kafka is an open-source distributed stream-processing platform that is capable of handling over trillions of events in a day. This massive platform has been developed by the LinkedIn Team, written in Java and Scala, and donated to Apache.
Kafka is a publish-subscribe messaging system. So, in this article, we are going to learn how Kafka works and how to use Kafka in our .NET Application. To learn how to install, configure, and run Kafka, please read this article.

How Kafka works?

Before going into details, we will discuss here a little bit of Kafka Architecture.

Kafka Architecture

In the above image, we can see the Producer, Consumer, and Topic. Basically, Kafka producers write to the Topic and consumers read from the Topic. Kafka runs on a cluster on the server and it is communicating with the multiple Kafka Brokers and each Broker has a unique identification number.
Kafka stores messages as a byte array and it communicates through the TCP Protocol.
Topics
As we know Kafka is a pub-sub model, Topic is a message category or, you can say, a logical channel. Topic defines the message stream of data and a Topic should have a unique id.
Partitions
All the messages are sequentially stored in one partition and the Topics are split into partitions. Basically, Kafka uses those partitions for parallel consumers. Kafka maintains all the records in order as a structured way, called log. There are no limitations to the number of partitions in a Topic and all the Topics are divided into a number of partitions.
Brokers
Kafka broker allow the fetching of messages for consumers, it’s known as Kafka server and Kafka node. Each broker has a unique Id that contains more than one Topic partition. All the Topics are divided into a number of partitions.

How do we use Kafka in .Net?

Here I am going to demonstrate a Kafka messaging service in a .Net Windows Application.
What do you need on your development computer?
Here, I have created a Windows Application Project.
After creating the Application project, download and install Kafka-net package from NuGet.
  1. Install-Package kafka-net -Version 0.9.0.65
Here I have designed a Windows Form,
You need to define the uri where Kafka is running and mention the topic name that you have created.
  1. Uri uri = new Uri("http://localhost:9092");
  2. string topic = "chat-message";
How to create a Kafka Topic? Please read this article,
  • Topic Name is “chat-message”. We are running it into only one Kafka server.
  • Go to your Kafka installation directory: For me, it’s D:\kafka\kafka_2.12-2.2.0\bin\windows.
  • Open a command prompt and run the following command.
    1. kafka-topics.bat --create --zookeeper localhost:2181 -replication-factor 1 --partitions 1 --topic chat-message
Code snippet
  1. private void btnSend_Click(object sender, EventArgs e)
  2. {
  3. if(txtMessage.Text ==string.Empty)
  4. {
  5. MessageBox.Show("Please Enter Message","Warning", MessageBoxButtons.OK,MessageBoxIcon.Warning);
  6. return;
  7. }
  8. string payload = txtMessage.Text.Trim();
  9. var sendMessage = new Thread(() => {
  10. KafkaNet.Protocol.Message msg = new KafkaNet.Protocol.Message(payload);
  11. var options = new KafkaOptions(uri);
  12. var router = new BrokerRouter(options);
  13. var client = new Producer(router);
  14. client.SendMessageAsync(topic, new List<KafkaNet.Protocol.Message> { msg }).Wait();
  15. });
  16. sendMessage.Start();
  17. this.Clear();
  18. }
In the above code snippet, you can see, I have put the code for sending the message into a particular Kafka Topic, for me it is "chat-message". You can create Kafka Topic by any preferred name.
Before sending the message you need to start the Zookeeper. Know about more Zookeeper, Please read this article.
  • Go to your Kafka installation directory: For me, it’s D:\kafka\kafka_2.12-2.2.0\bin\windows.
  • Open a command prompt and run the following command.
    1. zookeeper-server-start.bat D:\Kafka\kafka_2.12-2.2.0\config\zookeeper.properties
After starting the Zookeeper you need to run Kafka Server. Know about more Kafka Server, Please read this article.
  • Go to your Kafka installation directory: For me, it’s D:\kafka\kafka_2.12-2.2.0\bin\windows.
  • Open a command prompt and run the following command,
    1. kafka-server-start.bat D:\Kafka\kafka_2.12-2.2.0\config\server.properties
After running all the Services you need to consume the topic from the server, so that follow the below Steps.
  • Go to your Kafka installation directory: For me, it’s D:\kafka\kafka_2.12-2.2.0\bin\windows.
  • Open a command prompt and run the following command,
    1. kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic chat-message --from-beginning
You can get all the Kafka messages by using the following code snippet.
Code snippet
  1. Uri uri = new Uri("http://localhost:9092");
  2. string topicName = "chat-message";
  3. var options = new KafkaOptions(uri);
  4. var brokerRouter = new BrokerRouter(options);
  5. var consumer = new Consumer(new ConsumerOptions(topicName, brokerRouter));
  6. foreach (var msg in consumer.Consume())
  7. {
  8. Console.WriteLine(Encoding.UTF8.GetString(msg.Value));
  9. }
  10. Console.ReadLine();

Conclusion

If you want to use a highly scalable and high-performance messaging system with .Net Application, you easily develop that system by using Kafka-net package.