S10E1 - Engineering the Super Bowl ft. Catherine Johnson, VP of Global Solutions Engg. at Hydrolix

2026-02-03

Download

Real-time analytics at a petabyte scale isn't just a technical challenge; it's a business survival requirement. Catherine Johnson, VP of Global Solutions Engineering at Hydrolix, joins the show to deconstruct the "impossible" architecture required to power the 2025 Super Bowl broadcast for Fox Sports. From managing 1.4 petabytes of daily log data to the brutal reality of why traditional auto-scaling fails during mission-critical events, Catherine reveals the strategic framework behind being a "Truth Teller" in the high-stakes world of Solutions Engineering.

Key Takeaways

1. Data Architecture as a Competitive Moat

- Normalization is Non-Negotiable: At a petabyte scale, you cannot afford "dirty" data. Success requires normalizing disparate CDN logs—matching units (ms vs. s) and handling recursive URL encoding—into a single, queryable schema.

- Indexing vs. Regex: Computational intensity kills performance. Strategic indexing for exact matches must replace regular expressions for high-frequency queries to avoid massive, costly table scans.

- Schema Flexibility: Implementing multiple schemas on a single table allows for both granular technical deep-dives and high-level executive overviews without duplicating storage.

2. Scaling Strategies for "High-Intensity" Events

- The Limits of Auto-scaling: For predictable surges like the Super Bowl, relying on auto-scaling is a risk. Pre-scaling to 3x expected peak ensures availability when AWS regional compute limits are hit.

- Multi-Region Redundancy: True global scale often exceeds the capacity of a single cloud region. Architecting for multi-region deployment is a requirement, not an option, for Tier-1 broadcast events.

- Segregated Query Pools: Prevent "compute competition" by isolating resources. Executive dashboards, SRE monitoring, and ad-hoc troubleshooting should never fight for the same compute cycles.

3. Solutions Engineering as "Truth Telling"

- The Trust-Based Framework: A Solution Engineer’s (SE) primary role isn't selling—it's building trust through accurate empathy. If the product isn't a fit, say it. Protecting your professional reputation outlasts any single sales cycle.

- Root Cause Inquiry: When a customer asks for a feature or query optimization, pause. Don't answer the technical question until you've uncovered the business outcome they are trying to achieve.

- Business Mapping: Every technical requirement must map directly to a business requirement. If it doesn't, it’s just unnecessary complexity.

4. The "Break-Fast" Learning Philosophy

- Fearless Experimentation: The learning curve is shortened by breaking things in dedicated environments. If you only follow the "happy path" of a tutorial, you haven't actually learned the system.

- Bridging Data Realities: There is often a gap between how data is stored for performance and how it looks in the real world. Success in SE requires the ability to bridge these two perspectives for the customer.

Chapters:

00:10 - Introduction: Meet Catherine Johnson
00:50 - The Origins of Hydrolix: Solving the CDN Log Crisis
06:10 - Deep Dive: Behind the Scenes of the 2025 Super Bowl
10:14 - When the Path Changes: Adjusting Architecture Mid-Season
14:25 - Multi-Region Deployment & AWS Compute Limits
16:51 - Half-Second Query Times: How to Segregate Compute
25:49 - The Non-Obvious Skills of Top-Tier SEs
31:32 - The "Farming" Lesson: Understanding How Businesses Make Money
37:04 - Lightning Round