Streaming real-time data at scale and processing it efficiently is critical to cybersecurity organizations like SecurityScorecard. Jared Smith, Senior Director of Threat Intelligence, and Brandon Brown, Senior Staff Software Engineer, Data Platform at SecurityScorecard, discuss their journey from using RabbitMQ to open-source Apache Kafka® for stream processing. As well as why turning to fully-managed Kafka on Confluent Cloud is the right choice for building real-time data pipelines at scale.
SecurityScorecard mines data from dozens of digital sources to discover security risks and flaws with the potential to expose their client’ data. This includes scanning and ingesting data from a large number of ports to identify suspicious IP addresses, exposed servers, out-of-date endpoints, malware-infected devices, and other potential cyber threats for more than 12 million companies worldwide.
To allow real-time stream processing for the organization, the team moved away from using RabbitMQ to open-source Kafka for processing a massive amount of data in a matter of milliseconds, instead of weeks or months. This makes the detection of a website’s security posture risk happen quickly for constantly evolving security threats. The team relied on batch pipelines to push data to and from Amazon S3 as well as expensive REST API based communication carrying data between systems. They also spent significant time and resources on open-source Kafka upgrades on Amazon MSK.
Self-maintaining the Kafka infrastructure increased operational overhead with escalating costs. In order to scale faster, govern data better, and ultimately lower the total cost of ownership (TOC), Brandon, lead of the organization’s Pipeline team, pivoted towards a fully-managed, cloud-native approach for more scalable streaming data pipelines, and for the development of a new Automatic Vendor Detection (AVD) product.
Jared and Brandon continue to leverage the Cloud for use cases including using PostgreSQL and pushing data to downstream systems using CSC connectors, increasing data governance and security for streaming scalability, and more.
EPISODE LINKS
From Monolith to Microservices with Sam Newman
Exploring Event Streaming Use Cases with µKanren ft. Tim Baldridge
Introducing JSON and Protobuf Support ft. David Araujo and Tushar Thole
Scaling Apache Kafka in Retail with Microservices ft. Matt Simpson from Boden
Connecting Snowflake and Apache Kafka ft. Isaac Kunen
AMA with Tim Berglund | Streaming Audio Special
Kubernetes Meets Apache Kafka ft. Kelsey Hightower
Apache Kafka Fundamentals: The Concept of Streams and Tables ft. Michael Noll
IoT Integration and Real-Time Data Correlation with Kafka Connect and Kafka Streams ft. Kai Waehner
Confluent Platform 5.5 | What's New in This Release + Updates
Making Abstract Algebra Count in the World of Event Streaming ft. Sam Ritchie
Apache Kafka 2.5 – Overview of Latest Features, Updates, and KIPs
Streaming Data Integration – Where Development Meets Deployment ft. James Urquhart
How to Run Kafka Streams on Kubernetes ft. Viktor Gamov
Cloud Marketplace Considerations with Dan Rosanova
Explore, Expand, and Extract with 3X Thinking ft. Kent Beck
Ask Confluent #17: The “What is Apache Kafka?” Episode ft. Tim Berglund
Domain-Driven Design and Apache Kafka with Paul Rayner
Machine Learning with TensorFlow and Apache Kafka ft. Chris Mattmann
Distributed Systems Engineering with Apache Kafka ft. Gwen Shapira
Create your
podcast in
minutes
It is Free
Insight Story: Tech Trends Unpacked
Zero-Shot
Fast Forward by Tomorrow Unlocked: Tech past, tech future
The Unbelivable Truth - Series 1 - 26 including specials and pilot
Well There‘s Your Problem