Is it possible to build a real-time data platform without using stateful stream processing? Forecasty.ai is an artificial intelligence platform for forecasting commodity prices, imparting insights into the future valuations of raw materials for users. Nearly all AI models are batch-trained once, but precious commodities are linked to ever-fluctuating global financial markets, which require real-time insights. In this episode, Ralph Debusmann (CTO, Forecasty.ai) shares their journey of migrating from a batch machine learning platform to a real-time event streaming system with Apache Kafka® and delves into their approach to making the transition frictionless.
Ralph explains that Forecasty.ai was initially built on top of batch processing, however, updating the models with batch-data syncs was costly and environmentally taxing. There was also the question of scalability—progressing from 60 commodities on offer to their eventual plan of over 200 commodities. Ralph observed that most real-time systems are non-batch, streaming-based real-time data platforms with stateful stream processing, using Kafka Streams, Apache Flink®, or even Apache Samza. However, stateful stream processing involves resources, such as teams of stream processing specialists to solve the task.
With the existing team, Ralph decided to build a real-time data platform without using any sort of stateful stream processing. They strictly keep to the out-of-the-box components, such as Kafka topics, Kafka Producer API, Kafka Consumer API, and other Kafka connectors, along with a real-time database to process data streams and implement the necessary joins inside the database.
Additionally, Ralph shares the tool he built to handle historical data, kash.py—a Kafka shell based on Python; discusses issues the platform needed to overcome for success, and how they can make the migration from batch processing to stream processing painless for the data science team.
EPISODE LINKS
Becoming Data Driven with Apache Kafka and Stream Processing ft. Daniel Jagielski
Integrating Spring Boot with Apache Kafka ft. Viktor Gamov
Confluent Platform 6.1 | What’s New in This Release + Updates
Building a Microservices Architecture with Apache Kafka at Nationwide Building Society ft. Rob Jackson
Examining Apache Kafka Performance Metrics ft. Alok Nikhil
Distributed Systems Engineering with Apache Kafka ft. Guozhang Wang
Scaling Developer Productivity with Apache Kafka ft. Mohinish Shaikh
Change Data Capture and Kafka Connect on Microsoft Azure ft. Abhishek Gupta
Event Streaming Trends and Predictions for 2021 ft. Gwen Shapira, Ben Stopford, and Michael Noll
How to Become a Certified Apache Kafka Expert ft. Niamh O’Byrne and Barry Ballard
Mastering DevOps with Apache Kafka, Kubernetes, and Confluent Cloud ft. Rick Spurgeon and Allison Walther
Apache Kafka 2.7 - Overview of Latest Features, Updates, and KIPs
Choreographing the Saga Pattern in Microservices ft. Chris Richardson
Apache Kafka and Porsche: Fast Cars and Fast Data ft. Sridhar Mamella
Tales from the Frontline of Apache Kafka DevOps ft. Jason Bell
Multi-Tenancy in Apache Kafka ft. Anna Pozvner
Distributed Systems Engineering with Apache Kafka ft. Roger Hoover
Why Kafka Streams Does Not Use Watermarks ft. Matthias J. Sax
Distributed Systems Engineering with Apache Kafka ft. Apurva Mehta
Most Terrifying Apache Kafka JIRAs of 2020 ft. Anna McDonald
Create your
podcast in
minutes
It is Free
Insight Story: Tech Trends Unpacked
Zero-Shot
Fast Forward by Tomorrow Unlocked: Tech past, tech future
Black Wolf Feed (Chapo Premium Feed Bootleg)
Bannon`s War Room