188: World Models

Intro topic: Running

News/Links:

• Flow matching versus diffusion models
  • https://youtu.be/firXjwZ_6KI?is=QMq8DcCsXTTktOuE
• OpenCV 5
  • https://opencv.org/opencv-5/
• Claude fable beats pokemon with no harness
  • https://youtu.be/Ty_50J84fMY?si=EJ1KjZCZegipfCsV
• Can the stockmarket swallow Anthropic, SpaceX and OpenAI?
  • https://archive.ph/nKEVw
    
    

Book of the Show

• Patrick
  • Strength of the Few - James Islington
    • https://amzn.to/4pmr10T
• Jason
  • Descender - Jeff Lemire
    • https://amzn.to/3QKhp3l

Patreon Plug https://www.patreon.com/programmingthrowdown?ty=h

Tool of the Show

• Patrick
  • No Man’s Sky
• Jason
  • Paperlib https://paperlib.app/en/
    
    

Topic: World Models

• Making decisions with AI
  • Action-Value (called a Q model): What is the long-term value of making a decision at a position
  • Policy: What action should I take (must be a distribution)
  • Value (called a V model): What is the value of a position (depends on policy)
  • Advantage/Disadvantage: difference in value given two policies
  • When advantage is +, do that more.
• Model-Free
  • Look at the current situation and suggest an action
  • Run that action in the real world and measure the effect
  • Use that measurement to suggest better actions next time
• Model-Based
  • Observe rollouts (sequences of situations) and learn the dynamics
  • Choose an action, use your dynamics model to measure the consequence
  • Potentially do MPC (try many actions and choose the best)
• World Models
  • Observe many many rollouts and learn a full forward model (how to create the input in the future)
  • Train a policy & value inside the world model
  • Deploy the policy and fine-tune based on the real world