Kafka Producers (Messages, Keys, Batching, Compression, Retries and Idempotence)

Every Kafka-based system begins with one critical component - the Producer. Producers are responsible for publishing messages into Kafka topics, making them the entry point for all data flowing through the platform.

Questions such as how Kafka chooses a partition, how batching improves throughput, what happens during broker failures, how retries work, and how Kafka prevents duplicate messages are frequently discussed in production environments and technical interviews.

In this article, we will take a deep dive into Kafka producers and explore message publishing, partitioning using keys, batching, compression, retries, acknowledgements, and idempotence.

Role of a Kafka Producer

A Producer is a client application that publishes records into Kafka topics. Consider an e-commerce application where a customer places an order. The application may publish an event such as:

{
  "orderId": 1001,
  "customerId": 500,
  "amount": 2500
}

The producer sends this event to Kafka. The overall flow looks like: Application → Kafka Producer → Kafka Topic. Consumers can later read and process these events independently.

The producer is responsible for determining which broker receives the message, which partition stores the message, and how delivery guarantees are enforced.

Creating a Kafka Producer

A producer requires a set of configuration properties and a KafkaProducer instance. A simple producer configuration looks like:

Properties props = new Properties();

props.put(
    ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,
    "localhost:9092"
);

props.put(
    ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
    StringSerializer.class.getName()
);

props.put(
    ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
    StringSerializer.class.getName()
);

KafkaProducer<String, String> producer =
    new KafkaProducer<>(props);

The bootstrap.servers property identifies Kafka brokers used during initial connection. The serializers convert Java objects into bytes before sending them across the network.

Publishing Messages

Publishing a message is straightforward.

ProducerRecord<String, String> record =
    new ProducerRecord<>(
        "orders",
        "customer-101",
        "Order Created"
    );

producer.send(record);

The producer does not immediately send every message to the broker. Instead, Kafka uses internal buffers and batching mechanisms to optimize network utilization and throughput.

This behavior is one reason Kafka can handle millions of messages per second.

Understanding Producer Records

A producer sends data using a ProducerRecord. A record contains:

Topic
Key
Value
Partition (optional)
Timestamp
Headers (optional)

Example:

ProducerRecord<String, String> record =
    new ProducerRecord<>(
        "orders",
        "customer-101",
        "Order Created"
    );

Here:

Topic = orders
Key = customer-101
Value = Order Created

The most important field from an architectural perspective is the Key. Kafka uses message keys to determine partition assignment.

1. When a key is present, Kafka calculates:

hash(key) % numberOfPartitions

Suppose we have four partitions:

Partition-0
Partition-1
Partition-2
Partition-3

And we send:

new ProducerRecord<>(
    "orders",
    "customer-123",
    "Order Created"
);

Kafka computes a hash of "customer-123" and consistently routes all events with the same key to the same partition. Example:

customer-123 -> Partition-1
customer-123 -> Partition-1
customer-123 -> Partition-1

This guarantees ordering for all events related to that customer. Without keys, Kafka cannot guarantee that related events remain together.

2. When No Key Is Provided: If no key exists:

new ProducerRecord<>(
    "orders",
    "Order Created"
);

Kafka uses its default partitioning strategy. Modern Kafka producers use a sticky partitioner. The producer sends multiple records to the same partition temporarily and then switches to another partition.

Example:

Message-1 -> Partition-0
Message-2 -> Partition-0
Message-3 -> Partition-0

Message-4 -> Partition-1
Message-5 -> Partition-1

This improves batching efficiency and throughput. However, ordering between messages is no longer guaranteed.

Producer Acknowledgements

One of the most important producer settings is acks. Acknowledgements determine when Kafka considers a message successfully written. Three values are available.

1. acks=0: The producer does not wait for broker confirmation. The producer immediately assumes success. This provides maximum throughput but risks message loss.
2. acks=1: The producer waits for the partition leader. This balances performance and reliability.
3. acks=all: The producer waits until all in-sync replicas acknowledge the message. This provides the strongest durability guarantees.

Most production systems use: acks=all

Producer Batching

Sending one network request per message is inefficient. Kafka therefore groups multiple records into batches.

Batching significantly improves throughput and reduces network overhead. Kafka producers batch records per partition.

1. The batch.size setting determines the maximum size of a batch. Example:

props.put(
    ProducerConfig.BATCH_SIZE_CONFIG,
    32768
);

This allows Kafka to accumulate up to 32 KB of records before sending them. Larger batches usually increase throughput but may increase latency slightly.

2. The linger.ms Configuration. Sometimes the batch is not full. Kafka can wait briefly for additional records.

props.put(
    ProducerConfig.LINGER_MS_CONFIG,
    10
);

This tells Kafka, "Wait up to 10 milliseconds before sending."

Benefits include, Higher throughput, larger batches, and fewer network requests. Many production workloads use values between: 5 ms, 10 ms and 20 ms depending on latency requirements.

Compression in Kafka Producers

Compression reduces network traffic and storage consumption. Kafka supports multiple compression algorithms, i.e. gzip, snappy, lz4, zstd, etc.

props.put(
    ProducerConfig.COMPRESSION_TYPE_CONFIG,
    "snappy"
);

Compression occurs at the batch level rather than the individual message level. This approach delivers much better compression ratios.

- gzip provides excellent compression but higher CPU usage.
- snappy offers faster compression and decompression.
- lz4 balances speed and compression ratio.
- zstd often provides the best overall compression efficiency in modern deployments.

For most production systems today "zstd" is becoming the preferred choice.

Producer Retries

Distributed systems experience failures.
- Network interruptions occur.
- Brokers restart.
- Leaders change.

Kafka producers automatically retry failed requests.

Example:

props.put(
    ProducerConfig.RETRIES_CONFIG,
    Integer.MAX_VALUE
);

Retries improve reliability significantly. However, they introduce another challenge: duplicate messages. Consider this scenario.

1. Producer sends a message.
2. Broker writes the message successfully.
3. Acknowledgement is lost due to a network issue.
4. Producer assumes failure and retries.
5. The broker now receives the same message again.

Duplicate records have entered Kafka. For payment systems, order processing systems, and financial applications, this can be disastrous.Kafka introduced idempotent producers to solve this problem.

Idempotence guarantees that a message is written exactly once to a partition even if retries occur.

Enable it using:

props.put(
    ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG,
    true
);

When enabled, Kafka assigns a unique producer identifier and sequence numbers. Example:

Message 1 -> Sequence 1
Message 2 -> Sequence 2
Message 3 -> Sequence 3

If Message 2 is retried:

Message 2 -> Sequence 2

Kafka recognizes it as a duplicate and ignores the second write. This prevents duplicate records while still allowing retries. When idempotence is enabled, Kafka automatically enforces certain settings.

acks=all
retries > 0
max.in.flight.requests.per.connection <= 5

These settings ensure correct ordering and duplicate prevention.

The max.in.flight.requests.per.connection setting dictates how many unacknowledged requests the client can send on a single TCP connection. In Apache Kafka, the default is 5. A higher value improves throughput but risks message reordering during retries, while setting it to 1 enforces strict sequential ordering.

Synchronous vs Asynchronous Sends

Kafka producers can send messages asynchronously.

producer.send(record);

This provides maximum throughput. The producer continues execution without waiting.

For synchronous sending:

producer.send(record).get();

The producer waits until Kafka confirms the write. Asynchronous publishing is preferred in most high-throughput systems. Synchronous publishing is generally reserved for critical operations requiring immediate confirmation.

Recommended Configuration

For most enterprise workloads, a reliable producer configuration resembles:

props.put(
    ProducerConfig.ACKS_CONFIG,
    "all"
);

props.put(
    ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG,
    true
);

props.put(
    ProducerConfig.COMPRESSION_TYPE_CONFIG,
    "zstd"
);

props.put(
    ProducerConfig.LINGER_MS_CONFIG,
    10
);

props.put(
    ProducerConfig.BATCH_SIZE_CONFIG,
    32768
);

props.put(
    ProducerConfig.RETRIES_CONFIG,
    Integer.MAX_VALUE
);

This configuration prioritizes durability, throughput, and operational safety.

Common Kafka Producer Interview Questions

How Kafka chooses a partition?
- If a key exists, Kafka hashes the key and maps it to a partition. Without a key, Kafka uses its default partitioning strategy.

Why batching improves throughput?
- Batching reduces network calls and allows compression to operate on larger groups of records.

Difference between acks=1 and acks=all.
- The first waits only for the leader, while the second waits for all in-sync replicas and therefore provides stronger durability guarantees.

Retries vs Idempotence
- Retries improve reliability but may introduce duplicates. Idempotence eliminates duplicates while preserving retry behavior.

When to use message keys?
- Whenever ordering of related events must be preserved.

Conclusion

Kafka producers are far more sophisticated than simple message publishers. Behind every published record, Kafka performs partition selection, batching, compression, retries, acknowledgements, and duplicate prevention.

Understanding these mechanisms is essential for building reliable and scalable event-driven systems.