Artificial Intelligence - VideoGameBench Can Vision-Language Models complete popular video games?

Hey PaperLedge learning crew, Ernis here! Get ready to level up your knowledge because today we're diving into some seriously cool research about how well AI understands the world through sight and language, just like we do. But instead of textbooks, we're using... video games!

That's right, researchers have created a new challenge called VideoGameBench. Think of it as an obstacle course for AI, using classic 90s video games like Super Mario World, The Legend of Zelda: A Link to the Past, Kirby Super Star, and more. The goal? To see if cutting-edge vision-language models (VLMs) – that's AI that can "see" images and "understand" text – can actually play these games from start to finish.

Now, these VLMs are already pretty amazing. They can solve complex math problems and even write code! But the researchers noticed something: these AIs are really good at tasks that are hard for humans, but still struggle with things that come naturally to us, like figuring out where we are, remembering things, and understanding what we see. It's like they're brilliant at calculus but can't find their way out of a paper bag!

So, why video games? Well, video games are designed to be intuitive for humans. They rely on our natural ability to learn and understand patterns. Plus, they're a fun way to test if an AI can actually perceive, navigate, and remember, all at the same time. This is a big deal!

"Real video games are crafted to be intuitive for humans to learn and master by leveraging innate inductive biases, making them an ideal testbed for evaluating such capabilities in VLMs."

The cool part is, the AI only gets to see the game screen, just like we do. It also gets a simple description of the game's goals and controls. No extra hints or special training! It's a pure test of its ability to understand and interact with the world.

To make things even more interesting, the researchers kept three of the games a secret! This forces the AI to learn general skills instead of memorizing specific solutions. It's like teaching someone to ride a bike instead of just memorizing how to ride one specific bike on one specific path. This is a test of generalization.

So, how did the AIs do? Well... not great. Even the most advanced VLMs struggled to get past the very beginning of the games. Why? It turns out that a major problem is inference latency. That's a fancy way of saying that the AI takes too long to process what it sees and decide what to do next. Imagine trying to play a fast-paced game when you have to pause every second to think about your next move – that's what these AIs are dealing with.

To address this, the researchers created VideoGameBench Lite. In this version, the game pauses while the AI is thinking. Even with this advantage, the best AI, Gemini 2.5 Pro, only completed a tiny fraction of the games (less than 2%).

The researchers hope that this new benchmark will inspire more research into how to make AI better at understanding and interacting with the real world. It's not just about winning video games, it's about building AI that can assist us in all sorts of ways, from helping us navigate complex environments to understanding our needs and preferences.

Now, here are a few things that really got me thinking:

• Why is it so much harder for AI to learn these "human" skills compared to complex calculations? Is it a matter of data, algorithms, or something more fundamental?
• Could improving AI's "reaction time" in these games translate to real-world benefits in fields like robotics or self-driving cars?
• If AI can't even beat Super Mario World, are we overestimating its ability to truly understand and interact with the world around us?

What do you think, learning crew? Let me know your thoughts in the comments. Until next time, keep exploring!

Credit to Paper authors: Alex L. Zhang, Thomas L. Griffiths, Karthik R. Narasimhan, Ofir Press