Today we’re joined by Sherry Yang, senior research scientist at Google DeepMind and a PhD student at UC Berkeley. In this interview, we discuss her new paper, "Video as the New Language for Real-World Decision Making,” which explores how generative video models can play a role similar to language models as a way to solve tasks in the real world. Sherry draws the analogy between natural language as a unified representation of information and text prediction as a common task interface and demonstrates how video as a medium and generative video as a task exhibit similar properties. This formulation enables video generation models to play a variety of real-world roles as planners, agents, compute engines, and environment simulators. Finally, We explore UniSim, an interactive demo of Sherry's work and a preview of her vision for interacting with AI-generated environments.
The complete show notes for this episode can be found at twimlai.com/go/676.
Off-Line, Off-Policy RL for Real-World Decision Making at Facebook - #448
A Future of Work for the Invisible Workers in A.I. with Saiph Savage - #447
Trends in Graph Machine Learning with Michael Bronstein - #446
Trends in Natural Language Processing with Sameer Singh - #445
Trends in Computer Vision with Pavan Turaga - #444
Trends in Reinforcement Learning with Pablo Samuel Castro - #443
MOReL: Model-Based Offline Reinforcement Learning with Aravind Rajeswaran - #442
Machine Learning as a Software Engineering Enterprise with Charles Isbell - #441
Natural Graph Networks with Taco Cohen - #440
Productionizing Time-Series Workloads at Siemens Energy with Edgar Bahilo Rodriguez - #439
ML Feature Store at Intuit with Srivathsan Canchi - #438
re:Invent Roundup 2020 with Swami Sivasubramanian - #437
Predictive Disease Risk Modeling at 23andMe with Subarna Sinha - #436
Scaling Video AI at RTL with Daan Odijk - #435
Benchmarking ML with MLCommons w/ Peter Mattson - #434
Deep Learning for NLP: From the Trenches with Charlene Chambliss - #433
Feature Stores for Accelerating AI Development - #432
An Exploration of Coded Bias with Shalini Kantayya, Deb Raji and Meredith Broussard - #431
Common Sense as an Algorithmic Framework with Dileep George - #430
Scaling Enterprise ML in 2020: Still Hard! with Sushil Thomas - #429
Create your
podcast in
minutes
It is Free
20/20
The Dropout
Ten Percent Happier with Dan Harris
World News Tonight with David Muir
NEJM This Week