Serverless Stream Processing with Apache Kafka ft. Bill Bejeck

What is serverless?

Having worked as a software engineer for over 15 years and as a regular contributor to Kafka Streams, Bill Bejeck (Integration Architect, Confluent) is an Apache Kafka® committer and author of “Kafka Streams in Action.” In today’s episode, he explains what serverless and the architectural concepts behind it are. 

To clarify, serverless doesn’t mean you can run an application without a server—there are still servers in the architecture, but they are abstracted away from your application development. In other words, you can focus on building and running applications and services without any concerns over infrastructure management. 

Using a cloud provider such as Amazon Web Services (AWS) enables you to allocate machine resources on demand while handling provisioning, maintenance, and scaling of the server infrastructure. 

There are a few important terms to know when implementing serverless functions with event stream processors: 

• Functions as a service (FaaS)
• Stateless stream processing
• Stateful stream processing

Serverless commonly falls into the FaaS cloud computing service category—for example, AWS Lambda is the classic definition of a FaaS offering. You have a greater degree of control to run a discrete chunk of code in response to certain events, and it lets you write code to solve a specific issue or use case. 

Stateless processing is simpler in comparison to stateful processing, which is more complex as it involves keeping the state of an event stream and needs a key-value store. ksqlDB allows you to perform both stateless and stateful processing, but its strength lies in stateful processing to answer complex questions while AWS Lambda is better suited for stateless processing tasks. 

By integrating ksqlDB with AWS Lambda together, they deliver serverless event streaming and analytics at scale.

EPISODE LINKS

• Serverless Stream Processing with Apache Kafka, AWS Lambda, and ksqlDB
• Stateful Serverless Architectures with ksqlDB and AWS Lambda 
• Serverless GitHub repository
• Kafka Streams in Action
• Watch the video version of this podcast
• Join the Confluent Community
• Learn more with Kafka tutorials, resources, and guides at Confluent Developer
• Live demo: Intro to Event-Driven Microservices with Confluent
• Use PODCAST100 to get an additional $100 of free Confluent Cloud usage (details)