Building Real-Time Streaming Applications with Go and Kafka

In today’s fast-paced digital world, real-time data processing is becoming increasingly crucial for businesses to gain insights, make informed decisions, and provide dynamic user experiences. To meet these demands, developers are turning to technologies like Apache Kafka and programming languages like Go (also known as Golang) to build efficient, scalable, and reliable real-time streaming applications.

Building Real-Time Streaming Applications with Go and Kafka

In this article, we will explore the combination of Go and Kafka to build real-time streaming applications. We’ll dive into the fundamental concepts of Kafka and demonstrate how to leverage Go’s concurrency features to create high-performance streaming applications.

1. Understanding Kafka: The Backbone of Real-Time Streaming

Before we dive into the technical details, let’s understand the role of Apache Kafka in building real-time streaming applications.

1.1. Kafka: A Distributed Event Streaming Platform

Apache Kafka is an open-source distributed event streaming platform that is designed to handle high-throughput, fault-tolerant, and real-time data streaming. It allows you to publish and subscribe to streams of records (events) in a fault-tolerant and scalable manner. Kafka’s architecture is based on the publish-subscribe model, making it ideal for building real-time streaming applications.

1.2. Kafka Components

Kafka consists of several key components:

  • Producer: Producers are responsible for sending records to Kafka topics. These records can be thought of as individual events or messages that are produced by applications.
  • Broker: Kafka brokers are the servers that store and manage the published records. Brokers work together to form a Kafka cluster.
  • Topic: A topic is a named stream of records in Kafka. Producers write records to topics, and consumers read records from topics.
  • Consumer: Consumers subscribe to topics and read records from them. Each consumer group can have multiple consumers to parallelize processing.
  • Consumer Group: A consumer group is a group of consumers that work together to consume records from topics. Kafka ensures that each record in a topic is consumed by only one consumer within a group.

1.3. Why Go for Real-Time Streaming with Kafka?

Go (Golang) is a statically typed, compiled programming language designed for building efficient and concurrent applications. It has gained popularity in recent years due to its simplicity, performance, and built-in support for concurrency through goroutines and channels. These features make Go a great choice for building real-time streaming applications that require high concurrency and efficiency.

2. Building Real-Time Streaming Applications with Go and Kafka

Now that we have a solid understanding of Kafka and the benefits of using Go, let’s dive into building real-time streaming applications.

Step 1: Set Up Kafka

Before we start building the application, we need to set up Kafka on our system. Follow these steps to get Kafka up and running:

1. Download and unzip the Kafka distribution from the Apache Kafka website.

2. Start the ZooKeeper server (required for Kafka) by running the following command:

bin/zookeeper-server-start.sh config/zookeeper.properties

3. Start the Kafka broker by running the following command:

bin/kafka-server-start.sh config/server.properties

Step 2: Create a Kafka Topic

Next, let’s create a Kafka topic that our application will use to publish and consume records. Run the following command:

bin/kafka-topics.sh --create --topic my-topic --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

Step 3: Writing the Go Application

Now that we have Kafka set up and a topic created, let’s start building our Go application. In this example, we’ll create a simple producer-consumer application.

3.1. Install Kafka Go Library

Go offers a variety of libraries for interacting with Kafka. One popular choice is the “sarama” library. Install it using the following command:

go get github.com/Shopify/sarama

3.2. Create the Producer

Create a file named producer.go and add the following code to create a Kafka producer:

package main

import (

func main() {
    // Configure the Kafka producer
    config := sarama.NewConfig()
    config.Producer.Return.Successes = true

    // Create a new Kafka producer
    producer, err := sarama.NewSyncProducer([]string{"localhost:9092"}, config)
    if err != nil {
        log.Fatalln("Failed to start producer:", err)
    defer producer.Close()

    // Publish a message to the Kafka topic
    msg := &sarama.ProducerMessage{
        Topic: "my-topic",
        Value: sarama.StringEncoder("Hello, Kafka!"),

    partition, offset, err := producer.SendMessage(msg)
    if err != nil {
        log.Println("Failed to send message:", err)
    } else {
        fmt.Printf("Message sent to partition %d at offset %d\n", partition, offset)

This code sets up a Kafka producer, sends a message to the “my-topic” topic, and prints the partition and offset information.

3.3. Create the Consumer

Create a file named consumer.go and add the following code to create a Kafka consumer:

package main

import (

func main() {
    // Configure the Kafka consumer
    config := sarama.NewConfig()
    config.Consumer.Return.Errors = true

    // Create a new Kafka consumer
    consumer, err := sarama.NewConsumer([]string{"localhost:9092"}, config)
    if err != nil {
        log.Fatalln("Failed to start consumer:", err)
    defer consumer.Close()

    // Subscribe to the Kafka topic
    topic := "my-topic"
    partitionConsumer, err := consumer.ConsumePartition(topic, 0, sarama.OffsetOldest)
    if err != nil {
        log.Fatalln("Failed to start partition consumer:", err)
    defer partitionConsumer.Close()

    // Handle messages
    signals := make(chan os.Signal, 1)
    signal.Notify(signals, os.Interrupt)

    for {
        select {
        case msg := <-partitionConsumer.Messages():
            fmt.Printf("Received message: %s\n", msg.Value)
        case err := <-partitionConsumer.Errors():
            log.Println("Error:", err)
        case <-signals:

This code sets up a Kafka consumer, subscribes to the “my-topic” topic, and continuously reads and processes messages.

Step 4: Running the Application

With the producer and consumer code in place, let’s run the application:

1. Open two terminal windows.

2. In the first terminal, navigate to the directory containing producer.go and run the producer:

go run producer.go

3. In the second terminal, navigate to the directory containing consumer.go and run the consumer:

go run consumer.go

You should see the producer sending a message and the consumer receiving and displaying the message in the respective terminal windows.


Building real-time streaming applications with Go and Kafka opens up a world of possibilities for developers to create efficient, scalable, and fault-tolerant systems. Kafka’s powerful event streaming capabilities combined with Go’s concurrency features make this combination a formidable choice for real-time data processing.

In this article, we’ve covered the basics of Apache Kafka, the benefits of using Go for real-time streaming, and a step-by-step guide to building a simple producer-consumer application. As you delve deeper into real-time streaming, you’ll find endless opportunities to architect and develop robust applications that harness the power of data as it flows in real time. So go ahead and explore the world of real-time streaming with Go and Kafka – the possibilities are limitless.

Previously at
Flag Argentina
time icon
Over 5 years of experience in Golang. Led the design and implementation of a distributed system and platform for building conversational chatbots.