Today we’re joined by Sanmi Koyejo, assistant professor at Stanford University, to continue our NeurIPS 2024 series. In our conversation, Sanmi discusses his two recent award-winning papers. First, we dive into his paper, “Are Emergent Abilities of Large Language Models a Mirage?” We discuss the different ways LLMs are evaluated and the excitement surrounding their“emergent abilities” such as the ability to perform arithmetic Sanmi describes how evaluating model performance using nonlinear metrics can lead to the illusion that the model is rapidly gaining new capabilities, whereas linear metrics show smooth improvement as expected, casting doubt on the significance of emergence. We continue on to his next paper, “DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models,” discussing the methodology it describes for evaluating concerns such as the toxicity, privacy, fairness, and robustness of LLMs.
The complete show notes for this episode can be found at twimlai.com/go/671.
ML Platforms for Global Scale at Prosus with Paul van der Boor - #468 [TWIMLcon Sponsor Series]
Can Language Models Be Too Big? 🦜 with Emily Bender and Margaret Mitchell - #467
Applying RL to Real-World Robotics with Abhishek Gupta - #466
Accelerating Innovation with AI at Scale with David Carmona - #465
Complexity and Intelligence with Melanie Mitchell - #464
Robust Visual Reasoning with Adriana Kovashka - #463
Architectural and Organizational Patterns in Machine Learning with Nishan Subedi - #462
Common Sense Reasoning in NLP with Vered Shwartz - #461
How to Be Human in the Age of AI with Ayanna Howard - #460
How to Be Human in the Age of AI with Ayanna Howard - #460
Evolution and Intelligence with Penousal Machado - #459
Innovating Neural Machine Translation with Arul Menezes - #458
Building the Product Knowledge Graph at Amazon with Luna Dong - #457
Towards a Systems-Level Approach to Fair ML with Sarah M. Brown - #456
AI for Digital Health Innovation with Andrew Trister - #455
System Design for Autonomous Vehicles with Drago Anguelov - #454
Building, Adopting, and Maturing LinkedIn's Machine Learning Platform with Ya Xu - #453
Expressive Deep Learning with Magenta DDSP w/ Jesse Engel - #452
Semantic Folding for Natural Language Understanding with Francisco Weber - #451
The Future of Autonomous Systems with Gurdeep Pall - #450
Create your
podcast in
minutes
It is Free
20/20
The Dropout
Ten Percent Happier with Dan Harris
World News Tonight with David Muir
NEJM This Week