Podbean logo
  • Discover
  • Podcast Features

    Your all-in-one podcasting solution.

    Podcast App

    The best podcast player & podcast app.

  • Livestream

    High-performing audio live, without limits.

    Podcast Studio

    Easy-to-use audio recorder app.

  • PodAds

    Dynamic Ad Insertion for podcasts.

  • Premium

    Convert listeners into buyers anywhere, anytime
    with the convenience of Podbean Premium.

    Patron

    The seamless way for fans to support you directly
    from your podcast.

  • Ads Marketplace

    Join Ads Marketplace to earn money
    through sponsorship on your podcast.

  •  
  • All Arts Business Comedy Education
  • Fiction Government Health & Fitness History Kids & Family
  • Leisure Music News Religion & Spirituality Science
  • Society & Culture Sports Technology True Crime TV & Film
  • Live
  • Log in
  • Start your podcast for free
  • Podcasting
    • Podcast Features
    • Live Stream
    • PodAds
    • Podcast App
    • Podcast Studio
  • Monetization
    • Premium
    • Patron
    • Ads Marketplace
  • Enterprise
  • Pricing
  • Discover
  • Log in
    Sign up free
Papers Read on AI

Papers Read on AI

News:Tech News

OPT: Open Pre-trained Transformer Language Models

OPT: Open Pre-trained Transformer Language Models

2022-05-09
Download
Large language models, which are often trained for hundreds of thousands of compute days, have shown remarkable capabilities for zero- and few-shot learning. Given their computational cost, these models are difficult to replicate without significant capital. For the few that are available through APIs, no access is granted to the full model weights, making them difficult to study. We present Open Pre-trained Transformers (OPT), a suite of decoder-only pre-trained transformers ranging from 125M to 175B parameters, which we aim to fully and responsibly share with interested researchers. We show that OPT-175B is comparable to GPT-3, 1 while requiring only 1/7th the carbon footprint to develop. We are also releasing our logbook detailing the infrastructure challenges we faced, along with code for experimenting with all of the released models. 2022: Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, Luke Zettlemoyer Ranked #1 on Stereotypical Bias Analysis on CrowS-Pairs https://arxiv.org/pdf/2205.01068v3.pdf
view more

More Episodes

DeepNet: Scaling Transformers to 1, 000 Layers
2022-05-20 16
Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning
2022-05-19 26
CLIPort: What and Where Pathways for Robotic Manipulation
2022-05-15 21
Gradio: Hassle-Free Sharing and Testing of ML Models in the Wild
2022-05-14 17
Aesthetic Text Logo Synthesis via Content-aware Layout Inferring
2022-05-13 22
Language Models Can See: Plugging Visual Controls in Text Generation
2022-05-12 23
FederatedScope-GNN: Towards a Unified, Comprehensive and Efficient Package for Federated Graph Learning
2022-05-11 22
Better plain ViT baselines for ImageNet-1k
2022-05-09 28
Quantile Encoder: Tackling High Cardinality Categorical Features in Regression Problems
2022-05-06 26
LM-Debugger: An Interactive Tool for Inspection and Intervention in Transformer-Based Language Models
2022-05-05 30
Patches Are All You Need?
2022-05-03 20
OneFlow: Redesign the Distributed Deep Learning Framework from Scratch
2022-05-02 19
Understanding The Robustness in Vision Transformers
2022-04-30 24
PP-Matting: High-Accuracy Natural Image Matting
2022-04-29 16
LinkBERT: Pretraining Language Models with Document Links
2022-04-12 35
Deep Learning Methods for Improved Decoding of Linear Codes
2022-04-10 25
Instant Neural Graphics Primitives with a Multiresolution Hash Encoding
2022-04-07 28
SimSwap: An Efficient Framework For High Fidelity Face Swapping
2022-04-06 22
Sparse Instance Activation for Real-Time Instance Segmentation
2022-04-05 25
  • ←
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • →
01234567910111213141516171819

Get this podcast on your
phone, FREE

Download Podbean app on App Store Download Podbean app on Google Play

Create your
podcast in
minutes

  • Full-featured podcast site
  • Unlimited storage and bandwidth
  • Comprehensive podcast stats
  • Integrate with iTunes and Google
    store
  • Make money with your podcast
Get started

It is Free

  • Podcast Services

    • Podcast Features
    • Pricing
    • Enterprise Solution
    • Private Podcast
    • The Podcast App
    • Live Stream
    • Audio Recorder
    • Remote Recording
  •  
    • Create a Podcast
    • Video Podcast
    • Start Podcasting
    • Start Radio Talk Show
    • Education Podcast
    • Switch to Podbean
    • Submit Your Podcast
    • Podbean Plugins
  •  
    • Church Podcast
    • Nonprofit Podcast
    • Get Sermons Online
    • Free Audiobooks
    • How to Start a Podcast
    • How to Start a Live Podcast
    • How to Monetize a podcast
    • How to Promote Your Podcast
  • MONETIZATION

    • Premium Podcast
    • Podcast Advertising
    • Patron Program
  • Support

    • Contact Us
    • Support Center
    • Developers
    • Resources
    • Free Webinars
    • Podcast Events
    • Podbean Academy
    • Podcasting Smarter
  • Podbean

    • About Us
    • Careers We are hiring !
    • Affiliate Program
    • Badges
    • Terms of Use
    • Privacy Policy
    • Podbean Blog

Copyright © 2006-2022 Podbean.com