Apache Kafka® 3.3 is released! With over two years of development, KIP-833 marks KRaft as production ready for new AK 3.3 clusters only. On behalf of the Kafka community, Danica Fine (Senior Developer Advocate, Confluent) shares highlights of this release, with KIPs from Kafka Core, Kafka Streams, and Kafka Connect.
To reduce request overhead and simplify client-side code, KIP-709 extends the OffsetFetch API requests to accept multiple consumer group IDs. This update has three changes, including extending the wire protocol, response handling changes, and enhancing the AdminClient to use the new protocol.
Log recovery is an important process that is triggered whenever a broker starts up after an unclean shutdown. And since there is no way to know the log recovery progress other than checking if the broker log is busy, KIP-831 adds metrics for the log recovery progress with `RemainingLogsToRecover` and `RemainingSegmentsToRecover`for each recovery thread. These metrics allow the admin to monitor the progress of the log recovery.
Additionally, updates on Kafka Core also include KIP-841: Fenced replicas should not be allowed to join the ISR in KRaft. KIP-835: Monitor KRaft Controller Quorum Health. KIP-859: Add metadata log processing error-related metrics.
KIP-834 for Kafka Streams added the ability to pause and resume topologies. This feature lets you reduce rescue usage when processing is not required or modifying the logic of Kafka Streams applications, or when responding to operational issues. While KIP-820 extends the KStream process with a new processor API.
Previously, KIP-98 added support for exactly-once delivery guarantees with Kafka and its Java clients. In the AK 3.3 release, KIP-618 offers the Exactly-Once Semantics support to Confluent’s source connectors. To accomplish this, a number of new connectors and worker-based configurations have been introduced, including `exactly.once.source.support`, `transaction.boundary`, and more.
Image attribution: Apache ZooKeeper™: https://zookeeper.apache.org/ and Raft logo: https://raft.github.io/
EPISODE LINKS
Data-Driven Digitalization with Apache Kafka in the Food Industry at BAADER
Chaos Engineering with Apache Kafka and Gremlin
Boosting Security for Apache Kafka with Confluent Cloud Private Link ft. Dan LaMotte
Confluent Platform 6.2 | What’s New in This Release + Updates
Adopting OpenTelemetry in Confluent and Beyond ft. Xavier Léauté
Running Apache Kafka Efficiently on the Cloud ft. Adithya Chandra
Engaging Database Partials with Apache Kafka for Distributed System Consistency ft. Pat Helland
The Truth About ZooKeeper Removal and the KIP-500 Release in Apache Kafka ft. Jason Gustafson and Colin McCabe
Resilient Edge Infrastructure for IoT Using Apache Kafka ft. Kai Waehner
Data Management and Digital Transformation with Apache Kafka at Van Oord
Powering Microservices Using Apache Kafka on Node.js with KafkaJS at Klarna ft. Tommy Brunn
Apache Kafka 2.8 - ZooKeeper Removal Update (KIP-500) and Overview of Latest Features
Connecting Azure Cosmos DB with Apache Kafka - Better Together ft. Ryan CrawCour
Automated Cluster Operations in the Cloud ft. Rashmi Prabhu
Resurrecting In-Sync Replicas with Automatic Observer Promotion ft. Anna McDonald
Building Real-Time Data Pipelines with Microsoft Azure, Databricks, and Confluent
Smooth Scaling and Uninterrupted Processing with Apache Kafka ft. Sophie Blee-Goldman
Event-Driven Architecture - Common Mistakes and Valuable Lessons ft. Simon Aubury
The Human Side of Apache Kafka and Microservices ft. SPOUD
Gamified Fitness at Synthesis Software Technologies Using Apache Kafka and IoT
Create your
podcast in
minutes
It is Free
Insight Story: Tech Trends Unpacked
Zero-Shot
Fast Forward by Tomorrow Unlocked: Tech past, tech future
The Unbelivable Truth - Series 1 - 26 including specials and pilot
Lex Fridman Podcast