Security is a primary consideration for any system design, and Apache Kafka® is no exception. Out of the box, Kafka has relatively little security enabled. Rajini Sivaram (Principal Engineer, Confluent, and co-author of “Kafka: The Definitive Guide” ) discusses how Kafka has gone from a system that included no security to providing an extensible and flexible platform for any business to build a secure messaging system. She shares considerations, important best practices, and features Kafka provides to help you design a secure modern data streaming system.
In order to build a secure Kafka installation, you need to securely authenticate your users. Whether you are using Kerberos (SASL/GSSAPI), SASL/PLAIN, SCRAM, or OAUTH. Verifying your users can authenticate, and non-users can’t, is a primary requirement for any connected system.
But authentication is only one part of the security story. We also need to address other areas. Kafka added support for fine-grained access control using ACLs with a pluggable authorizer several years ago. Over time, this was extended to support prefixed ACLs to make ACLs more manageable in large organizations. Now on its second generation authorizer, Kafka is easily extendable to support other forms of authorization, like integrating with a corporate LDAP server to provide group or role-based access control.
Even if you’ve set up your system to use secure authentication and each user is authorized using a series of ACLs if the data is viewable by anyone listening, how secure is your system? That’s where encryption comes in. Using TLS Kafka can encrypt your data-in-transit.
Security has gone from a nice-to-have to being a requirement of any modern-day system. Kafka has followed a similar path from zero security to having a flexible and extensible system that helps companies of any size pick the right security path for them.
Be sure to also check out the newest Apache Kafka Security course on Confluent Developer for an in-depth explanation along with other recommendations.
EPISODE LINKS
Apache Kafka 3.5 - Kafka Core, Connect, Streams, & Client Updates
A Special Announcement from Streaming Audio
How to use Data Contracts for Long-Term Schema Management
How to use Python with Apache Kafka
Next-Gen Data Modeling, Integrity, and Governance with YODA
Migrate Your Kafka Cluster with Minimal Downtime
Real-Time Data Transformation and Analytics with dbt Labs
What is the Future of Streaming Data?
What can Apache Kafka Developers learn from Online Gaming?
Apache Kafka 3.4 - New Features & Improvements
How to use OpenTelemetry to Trace and Monitor Apache Kafka Systems
What is Data Democratization and Why is it Important?
Git for Data: Managing Data like Code with lakeFS
Using Kafka-Leader-Election to Improve Scalability and Performance
Real-Time Machine Learning and Smarter AI with Data Streaming
The Present and Future of Stream Processing
Top 6 Worst Apache Kafka JIRA Bugs
Learn How Stream-Processing Works The Simplest Way Possible
Building and Designing Events and Event Streams with Apache Kafka
Rethinking Apache Kafka Security and Account Management
Create your
podcast in
minutes
It is Free
Insight Story: Tech Trends Unpacked
Zero-Shot
Fast Forward by Tomorrow Unlocked: Tech past, tech future
The Unbelivable Truth - Series 1 - 26 including specials and pilot
Lex Fridman Podcast