If you’ve heard the term “clusters,” then you might know it refers to Confluent components and features that we run in all three major cloud providers today, including an event streaming platform based on Apache Kafka®, ksqlDB, Kafka Connect, the Kafka API, databalancers, and Kafka API services. Rashmi Prabhu, a software engineer on the Control Plane team at Confluent, has the opportunity to help govern the data plane that comprises all these clusters and enables API-driven operations on these clusters.
But running operations on the cloud in a scaling organization can be time consuming, error prone, and tedious. This episode addresses manual upgrades and rolling restarts of Confluent Cloud clusters during releases, fixes, experiments, and the like, and more importantly, the progress that’s been made to switch from manual operations to an almost fully automated process. You’ll get a sneak peek into what upcoming plans to make cluster operations a fully automated process using the Cluster Upgrader, a new microservice in Java built with Vertx. This service runs as part of the control plane and exposes an API to the user to submit their workflows and target a set of clusters. It performs statement management on the workflow in the backend using Postgres.
So what’s next? Looking forward, there will be the selection phase will be improved to support policy-based deployment strategies that enable you to plan ahead and choose how you want to phase your deployments (e.g., first Azure followed by part of Amazon Web Services and then Google Cloud, or maybe Confluent internal clusters on all cloud providers followed by customer clusters on Google Cloud, Azure, and finally AWS)—the possibilities are endless!
The process will become more flexible, more configurable, and more error tolerant so that you can take measured risks and experience a standardized way of operating Cloud. In addition, expanding operation automations to internal application deployments and other kinds of fleet management operations that fit the “Select/Apply/Monitor” paradigm are in the works.
EPISODE LINKS
Ask Confluent #18: The Toughest Questions ft. Anna McDonald
Joining Forces with Spring Boot, Apache Kafka, and Kotlin ft. Josh Long
Building an Apache Kafka Center of Excellence Within Your Organization ft. Neil Buesing
Creating Your Own Kafka Improvement Proposal (KIP) as a Confluent Intern ft. Leah Thomas
Confluent Platform 6.0 | What's New in This Release + Updates
Using Event Modeling to Architect Event-Driven Information Systems ft. Bobby Calderwood
Using Apache Kafka as the Event-Driven System for 1,500 Microservices at Wix ft. Natan Silnitsky
Top 6 Things to Know About Apache Kafka ft. Gwen Shapira
5 Years of Event Streaming and Counting ft. Gwen Shapira, Ben Stopford, and Michael Noll
Championing Serverless Eventing at Google Cloud ft. Jay Smith
Disaster Recovery with Multi-Region Clusters in Confluent Platform ft. Anna McDonald and Mitch Henderson
Developer Advocacy (and Kafka Summit) in the Pandemic Era
Apache Kafka 2.6 - Overview of Latest Features, Updates, and KIPs
Testing ksqlDB Applications ft. Viktor Gamov
How to Measure the Business Value of Confluent Cloud ft. Lyndon Hedderly
Modernizing Inventory Management Technology ft. Sina Sojoodi and Rohit Kelapure
Fault Tolerance and High Availability in Kafka Streams and ksqlDB ft. Matthias J. Sax
Benchmarking Apache Kafka Latency at the 99th Percentile ft. Anna Povzner
Open Source Workflow Automation with Apache Kafka ft. Bernd Ruecker
Growing the Event Streaming Community During COVID-19 ft. Ale Murray
Create your
podcast in
minutes
It is Free
Insight Story: Tech Trends Unpacked
Zero-Shot
Fast Forward by Tomorrow Unlocked: Tech past, tech future
The Unbelivable Truth - Series 1 - 26 including specials and pilot
Lex Fridman Podcast