Today we’re joined by Sherry Yang, senior research scientist at Google DeepMind and a PhD student at UC Berkeley. In this interview, we discuss her new paper, "Video as the New Language for Real-World Decision Making,” which explores how generative video models can play a role similar to language models as a way to solve tasks in the real world. Sherry draws the analogy between natural language as a unified representation of information and text prediction as a common task interface and demonstrates how video as a medium and generative video as a task exhibit similar properties. This formulation enables video generation models to play a variety of real-world roles as planners, agents, compute engines, and environment simulators. Finally, We explore UniSim, an interactive demo of Sherry's work and a preview of her vision for interacting with AI-generated environments.
The complete show notes for this episode can be found at twimlai.com/go/676.
Constraint Active Search for Human-in-the-Loop Optimization with Gustavo Malkomes - #505
Fairness and Robustness in Federated Learning with Virginia Smith -#504
Scaling AI at H&M Group with Errol Koolmeister - #503
Evolving AI Systems Gracefully with Stefano Soatto - #502
ML Innovation in Healthcare with Suchi Saria - #501
Cross-Device AI Acceleration, Compilation & Execution with Jeff Gehlhaar - #500
The Future of Human-Machine Interaction with Dan Bohus and Siddhartha Sen - #499
Vector Quantization for NN Compression with Julieta Martinez - #498
Deep Unsupervised Learning for Climate Informatics with Claire Monteleoni - #497
Skip-Convolutions for Efficient Video Processing with Amir Habibian - #496
Advancing NLP with Project Debater w/ Noam Slonim - #495
Bringing AI Up to Speed with Autonomous Racing w/ Madhur Behl - #494
AI and Society: Past, Present and Future with Eric Horvitz - #493
Agile Applied AI Research with Parvez Ahammad - #492
Haptic Intelligence with Katherine J. Kuchenbecker - #491
Data Science on AWS with Chris Fregly and Antje Barth - #490
Accelerating Distributed AI Applications at Qualcomm with Ziad Asghar - #489
Buy AND Build for Production Machine Learning with Nir Bar-Lev - #488
Applied AI Research at AWS with Alex Smola - #487
Causal Models in Practice at Lyft with Sean Taylor - #486
Create your
podcast in
minutes
It is Free
20/20
The Dropout
Ten Percent Happier with Dan Harris
World News Tonight with David Muir
NEJM This Week