Today we’re joined by Sanmi Koyejo, assistant professor at Stanford University, to continue our NeurIPS 2024 series. In our conversation, Sanmi discusses his two recent award-winning papers. First, we dive into his paper, “Are Emergent Abilities of Large Language Models a Mirage?” We discuss the different ways LLMs are evaluated and the excitement surrounding their“emergent abilities” such as the ability to perform arithmetic Sanmi describes how evaluating model performance using nonlinear metrics can lead to the illusion that the model is rapidly gaining new capabilities, whereas linear metrics show smooth improvement as expected, casting doubt on the significance of emergence. We continue on to his next paper, “DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models,” discussing the methodology it describes for evaluating concerns such as the toxicity, privacy, fairness, and robustness of LLMs.
The complete show notes for this episode can be found at twimlai.com/go/671.
Stable Diffusion & Generative AI with Emad Mostaque - #604
Exploring Large Language Models with ChatGPT - #603
Accelerating Intelligence with AI-Generating Algorithms with Jeff Clune - #602
Programmatic Labeling and Data Scaling for Autonomous Commercial Aviation with Cedric Cocaud - #601
Engineering Production NLP Systems at T-Mobile with Heather Nolis - #600
Sim2Real and Optimus, the Humanoid Robot with Ken Goldberg - #599
The Evolution of the NLP Landscape with Oren Etzioni - #598
Live from TWIMLcon! The Great MLOps Debate: End-to-End ML Platforms vs Specialized Tools - #597
Live from TWIMLcon! You're not Facebook. Architecting MLOps for B2B Use Cases with Jacopo Tagliabue - #596
Building Foundational ML Platforms with Kubernetes and Kubeflow with Ali Rodell - #595
AI-Powered Peer Programming with Vasi Philomin - #594
The Top 10 Reasons to Register for TWIMLcon: AI Platforms 2022!
Applied AI/ML Research at PayPal with Vidyut Naware - #593
Assessing Data Quality at Shopify with Wendy Foster - #592
Transformers for Tabular Data at Capital One with Bayan Bruss - #591
Understanding Collective Insect Communication with ML, w/ Orit Peleg - #590
Multimodal, Multi-Lingual NLP at Hugging Face with John Bohannon and Douwe Kiela - #589
Synthetic Data Generation for Robotics with Bill Vass - #588
Multi-Device, Multi-Use-Case Optimization with Jeff Gehlhaar - #587
Causal Conceptions of Fairness and their Consequences with Sharad Goel - #586
Create your
podcast in
minutes
It is Free
20/20
The Dropout
Ten Percent Happier with Dan Harris
World News Tonight with David Muir
NEJM This Week