Building Observable Systems with eBPF and Linux (with Mohammed Aboullaite)

How do you monitor distributed systems that span dozens of microservices, multiple languages, and different databases? The old approach of gathering logs from different machines and recompiling apps with profiling flags doesn't scale when you're running thousands of servers. You need a unified strategy that works everywhere, on every component, in every language—and that means tackling the problem from the kernel level up.

Mohammed Aboullaite is a backend engineer at Spotify, and he joins us to explore the latest in continuous profiling and observability using eBPF. We dive into how eBPF lets you programmatically peek into the Linux kernel without recompiling it, why companies like Google and Meta run profiling across their entire infrastructure, and how to manage the massive data volumes that continuous profiling generates. Mohammed walks through specific tools like Pyroscope, Pixie, and Parca, explains the security model of loading code into the kernel, and shares practical advice on overhead thresholds, storage strategies, and getting organizational buy-in for continuous profiling.

Whether you're debugging performance issues, optimizing for scale, or just want to see what your code is really doing in production, this episode covers everything from packet filters to cultural changes in service of getting a clear view of your software when it hits production.

---

Support Developer Voices on Patreon: https://patreon.com/DeveloperVoices

Support Developer Voices on YouTube: https://www.youtube.com/@DeveloperVoices/join

eBPF: https://ebpf.io/

Google-Wide Profiling Paper (2010): https://research.google.com/pubs/archive/36575.pdf

Google pprof: https://github.com/google/pprof

Continuous Profiling Tools:

Pyroscope (Grafana): https://grafana.com/oss/pyroscope/

Pixie (CNCF): https://px.dev/

Parca: https://www.parca.dev/

Datadog Continuous Profiler: https://www.datadoghq.com/product/code-profiling/

Supporting Technologies:

OpenTelemetry: https://opentelemetry.io/

Grafana: https://grafana.com/

New Relic: https://newrelic.com/

Envoy Proxy: https://www.envoyproxy.io/

Spring Cloud Sleuth: https://spring.io/projects/spring-cloud-sleuth

Mohammed Aboullaite:

LinkedIn: https://www.linkedin.com/in/aboullaite/

GitHub: https://github.com/aboullaite

Website: http://aboullaite.me

Twitter/X: https://twitter.com/laytoun

Kris on Bluesky: https://bsky.app/profile/krisajenkins.bsky.social

Kris on Mastodon: http://mastodon.social/@krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/