Digital Developer Conference: Hybrid Cloud. On Sep 22 & 24, start your journey to OpenShift certification. Free registration

IBM Developer Blog

Follow the latest happenings with IBM Developer and stay in the know.

See what's going on in the Kafka community this month

This is the 40th edition of the Kafka Monthly Digest. In this edition, I’ll cover what happened in the Apache Kafka community in May 2021.

For last month’s digest, see Kafka Monthly Digest: April 2020.



I released Kafka 2.7.1 on May 10. This bugfix version addresses 45 JIRAs including a handful of blockers. For more details, see the release notes or the release plan.


Kafka 3.0.0 is the next major release. As mentioned last month, the next phase is KIP freeze and it is targeted for early June. See the release plan in the wiki for all the details.


Last month, the community submitted 6 KIPs (KIP-738 to KIP-744, KIP-742 was skipped), and these are the ones that caught my eye.

  • KIP-738: Removal of Connect’s internal converter properties. Connect’s internal converter properties (internal.key.converter and internal.value.converter) have been deprecated since Kafka 2.0.0. This KIP is proposing to remove these settings because users often confuse them with the actual data converters and setting these properties incorrectly can lead to worker failures.

  • KIP-739: Block Less on KafkaProducer#send. KafkaProducer.send()) returns a Future but in some cases (when the buffer is full or when a metadata refresh is required) it can also block. This can cause issues in fully asynchronous frameworks so this KIP aims at introducing a send() method that never blocks.

With 3.0.0 approaching, Kafka Streams is taking the opportunity to do some cleanup with the following KIPs:

  • KIP-741: Change default serde to be null. At the moment, by default, Streams uses a byte array Serdes. However, in practice, it’s often preferred for users to explicitly specify Serdes in order to immediately catch serialization issues. Hence this KIP proposes removing the default Serdes.

  • KIP-743: Remove config value 0.10.0-2.4 of Streams built-in metrics version config. KIP-444 in Kafka 2.4 improved Streams operational experience, which included adding labels to many metrics, and to not break compatibility, a configuration was introduced to keep the old behavior. This KIP proposes removing that configuration to simplify the maintenance of logic handling metrics.

Community Releases

In this section, I will cover releases of some community projects. This list only includes projects that are open source.

  • Librdkafka 1.7.0. Librdkafka is a Kafka client in C/C++. This new version improves the reliability of idempotent and transactional producers and adds support for the OpenSSL Engine. The Confluent clients in (Golang, Python and .Net) based on librdkafka have also been updated to use this new release.
  • strimzi-kafka-operator 0.23: Strimzi is a Kubernetes Operator for running Kafka. Kafka 2.6.2 and 2.8.0 are now supported. As always, this release includes loads of incremental improvements such as support for configuring connection count and creation rate quotas and configuring a custom Authorizer.
  • Sarama 1.29.0. Sarama is a pure Golang Kafka client. This new release adds support for Kafka 2.8.0. It’s also now possible to alter SCRAM credentials via DeleteUserScramCredentials and UpsertUserScramCredentials and the handling of zstd batches has been improved.


Get started with Kafka

IBM Event Streams for Cloud is Apache Kafka-as-a-Service for IBM Cloud. Get started with IBM Event Streams today.