In principle, data mesh architecture should liberate teams to build their systems and gather data in a distributed way, without having to explicitly coordinate. Data is the thing that can and should decouple teams, but proper implementation has its challenges.
In this episode, Kris talks to Florian Albrecht (Solution Architect, Hermes Germany) about Galapagos, an open-source DevOps software tool for Apache Kafka® that Albrecht created with his team at Hermes, a German parcel delivery company.
After Hermes chose Kafka to implement company-wide event-driven architecture, Albrecht’s team created rules and guidelines on how to use and really make the most out of Kafka. But the hands-off approach wasn’t leading to greater independence, so Albrecht’s team tried something different to documentation— they encoded the rules as software.
This method pushed the teams to stop thinking in terms of data and to start thinking in terms of events. Previously, applications copied data from one point to another, with slight changes each time. In the end, teams with conflicting data were left asking when the data changed and why, with a real impact on customers who might be left wondering when their parcel was redirected and how. Every application would then have to be checked to find out when exactly the data was changed. Event architecture terminates this cycle.
Events are immutable and changes are registered as new domain-specific events. Packaged together as event envelopes, they can be safely copied to other applications, and can provide significant insights. No need to check each application to find out when manually entered or imported data was changed—the complete history exists in the event envelope. More importantly, no more time-consuming collaborations where teams help each other to interpret the data.
Using Galapagos helped the teams at Hermes to switch their thought process from raw data to event-driven. Galapagos also empowers business teams to take charge of their own data needs by providing a protective buffer. When specific teams, providers of data or events, want to change something, Galapagos enforces a method which will not kill the production applications already reading the data. Teams can add new fields which existing applications can ignore, but a previously required field that an application could be relying on won’t be changeable.
Business partners using Galapagos found they were better prepared to give answers to their developer colleagues, allowing different parts of the business to communicate in ways they hadn’t before. Through Galapagos, Hermes saw better success decoupling teams.
EPISODE LINKS
Data-Driven Digitalization with Apache Kafka in the Food Industry at BAADER
Chaos Engineering with Apache Kafka and Gremlin
Boosting Security for Apache Kafka with Confluent Cloud Private Link ft. Dan LaMotte
Confluent Platform 6.2 | What’s New in This Release + Updates
Adopting OpenTelemetry in Confluent and Beyond ft. Xavier Léauté
Running Apache Kafka Efficiently on the Cloud ft. Adithya Chandra
Engaging Database Partials with Apache Kafka for Distributed System Consistency ft. Pat Helland
The Truth About ZooKeeper Removal and the KIP-500 Release in Apache Kafka ft. Jason Gustafson and Colin McCabe
Resilient Edge Infrastructure for IoT Using Apache Kafka ft. Kai Waehner
Data Management and Digital Transformation with Apache Kafka at Van Oord
Powering Microservices Using Apache Kafka on Node.js with KafkaJS at Klarna ft. Tommy Brunn
Apache Kafka 2.8 - ZooKeeper Removal Update (KIP-500) and Overview of Latest Features
Connecting Azure Cosmos DB with Apache Kafka - Better Together ft. Ryan CrawCour
Automated Cluster Operations in the Cloud ft. Rashmi Prabhu
Resurrecting In-Sync Replicas with Automatic Observer Promotion ft. Anna McDonald
Building Real-Time Data Pipelines with Microsoft Azure, Databricks, and Confluent
Smooth Scaling and Uninterrupted Processing with Apache Kafka ft. Sophie Blee-Goldman
Event-Driven Architecture - Common Mistakes and Valuable Lessons ft. Simon Aubury
The Human Side of Apache Kafka and Microservices ft. SPOUD
Gamified Fitness at Synthesis Software Technologies Using Apache Kafka and IoT
Create your
podcast in
minutes
It is Free
Insight Story: Tech Trends Unpacked
Zero-Shot
Fast Forward by Tomorrow Unlocked: Tech past, tech future
The Unbelivable Truth - Series 1 - 26 including specials and pilot
Acquired