Computer Vision - Scaling Laws for Native Multimodal Models Scaling Laws for Native Multimodal Models

Hey PaperLedge crew, Ernis here, ready to dive into another fascinating piece of research! This time, we're tackling the quest to build AI models that can truly see, hear, and understand the world around them, just like we do. Think of it as giving computers common sense, but through their "senses".

For a while now, the go-to method has been like building with LEGOs. You've got your "vision LEGO" (trained to understand images), your "language LEGO" (trained to understand text), and then you try to snap them together and hope they play nice. This is called a late-fusion architecture. The big language model is only seeing the image after it’s already been processed by something else.

But is that really the best way? Is there something inherently better about this approach?

That's exactly what the researchers behind this paper asked. They wanted to know if building these "Frankenstein" models was the only path to success, or if there was a better, more unified approach. They focused on what they call native multimodal models (NMMs). Think of it like baking a cake from scratch (NMM), versus assembling a pre-made cake from separate components (late-fusion).

They basically went on a model-training spree! They trained hundreds of different models with different architectures, to see which one performed better. Their investigation looked at the scaling laws of multimodal models. Think of "scaling laws" as studying how the model's performance changes as you make it bigger and feed it more data.

"Our investigation reveals no inherent advantage to late-fusion architectures over early-fusion ones... On the contrary, early-fusion exhibits stronger performance at lower parameter counts, is more efficient to train, and is easier to deploy."

And guess what? The results were surprising. They found that the "cake from scratch" approach – what's called early-fusion – actually held its own, and in some ways even beat the LEGO method, especially when the models were smaller.

So, what exactly is early-fusion? Instead of pre-training a vision encoder and then plugging it into a language model, early-fusion means feeding the model both the image data and the text data right from the start. The model learns to process them together, from the ground up. This "holistic" approach can actually be more efficient and easier to manage.

Think about it like this: imagine learning to ride a bike. You could learn to balance first, then learn to pedal, then try to put it all together. Or, you could just hop on the bike and learn everything at once. The second approach, the holistic approach, might be a little wobbly at first, but you might actually get the hang of it faster!

But here’s where it gets really cool. The researchers didn’t stop there. They took their best "cake from scratch" model and gave it a secret ingredient: Mixture of Experts (MoEs). Imagine having a team of specialists, each focusing on a different aspect of the problem (like vision or language), and the model learns to delegate tasks to the right expert. This boosted the model's performance even further!

So, why does all this matter? Well, for a few reasons:

• For researchers, it challenges the assumption that late-fusion is the only way forward and opens up new avenues for exploration.
• For developers, it suggests that early-fusion architectures could be a more efficient and practical choice for building multimodal AI systems.
• For everyone, it means we're getting closer to AI that can truly understand the world around us, leading to more helpful and intuitive technologies.

This opens up some interesting questions, doesn't it?

• If early-fusion is so promising, why has late-fusion been the dominant approach for so long? Was it simply a matter of computational resources or a lack of understanding of how to train these models effectively?
• As models continue to scale, will the benefits of early-fusion diminish, or will they become even more pronounced?
• Could we combine the best of both worlds – early-fusion's efficiency and late-fusion's modularity – to create even more powerful multimodal AI systems?

That's all for this episode, folks! I hope you enjoyed this deep dive into the world of multimodal models. Until next time, keep exploring and keep questioning!

Credit to Paper authors: Mustafa Shukor, Enrico Fini, Victor Guilherme Turrisi da Costa, Matthieu Cord, Joshua Susskind, Alaaeldin El-Nouby