arxiv Preprint - In-Context Pretraining: Language Modeling Beyond Document Boundaries
AI Breakdown

arxiv Preprint - In-Context Pretraining: Language Modeling Beyond Document Boundaries

2023-10-18
In this episode we discuss In-Context Pretraining: Language Modeling Beyond Document Boundaries by Weijia Shi, Sewon Min, Maria Lomeli, Chunting Zhou, Margaret Li, Victoria Lin, Noah A. Smith, Luke Zettlemoyer, Scott Yih, Mike Lewis. This paper introduces a new approach called IN-CONTEXT PRETRAINING for training large language models. It addresses the limitation of current LM training pipelines that concatenate random sets of short documents without providing signal for predicting the next...
View more
Comments (3)

More Episodes

All Episodes>>

Get this podcast on your phone, Free

Create Your Podcast In Minutes

  • Full-featured podcast site
  • Unlimited storage and bandwidth
  • Comprehensive podcast stats
  • Distribute to Apple Podcasts, Spotify, and more
  • Make money with your podcast
Get Started
It is Free