Today we’re joined by Sanmi Koyejo, assistant professor at Stanford University, to continue our NeurIPS 2024 series. In our conversation, Sanmi discusses his two recent award-winning papers. First, we dive into his paper, “Are Emergent Abilities of Large Language Models a Mirage?” We discuss the different ways LLMs are evaluated and the excitement surrounding their“emergent abilities” such as the ability to perform arithmetic Sanmi describes how evaluating model performance using nonlinear metrics can lead to the illusion that the model is rapidly gaining new capabilities, whereas linear metrics show smooth improvement as expected, casting doubt on the significance of emergence. We continue on to his next paper, “DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models,” discussing the methodology it describes for evaluating concerns such as the toxicity, privacy, fairness, and robustness of LLMs.
The complete show notes for this episode can be found at twimlai.com/go/671.
Brain-Inspired Hardware and Algorithm Co-Design with Melika Payvand - #585
Equivariant Priors for Compressed Sensing with Arash Behboodi - #584
Managing Data Labeling Ops for Success with Audrey Smith - #583
Engineering an ML-Powered Developer-First Search Engine with Richard Socher - #582
On The Path Towards Robot Vision with Aljosa Osep - #581
More Language, Less Labeling with Kate Saenko - #580
Optical Flow Estimation, Panoptic Segmentation, and Vision Transformers with Fatih Porikli - #579
Data Governance for Data Science with Adam Wood - #578
Feature Platforms for Data-Centric AI with Mike Del Balso - #577
The Fallacy of "Ground Truth" with Shayan Mohanty - #576
Principle-centric AI with Adrien Gaidon - #575
Data Debt in Machine Learning with D. Sculley - #574
AI for Enterprise Decisioning at Scale with Rob Walker - #573
Data Rights, Quantification and Governance for Ethical AI with Margaret Mitchell - #572
Studying Machine Intelligence with Been Kim - #571
Advances in Neural Compression with Auke Wiggers - #570
Mixture-of-Experts and Trends in Large-Scale Language Modeling with Irwan Bello - #569
Daring to DAIR: Distributed AI Research with Timnit Gebru - #568
Hierarchical and Continual RL with Doina Precup - #567
Open-Source Drug Discovery with DeepChem with Bharath Ramsundar - #566
Create your
podcast in
minutes
It is Free
20/20
The Dropout
Ten Percent Happier with Dan Harris
World News Tonight with David Muir
NEJM This Week