Artificial Intelligence - Visual serial processing deficits explain divergences in human and VLM reasoning

Hey PaperLedge crew, Ernis here, ready to dive into some seriously cool research! Today, we're tackling a question that's been bugging AI researchers: Why are those fancy Vision Language Models, or VLMs – you know, the ones that can describe pictures and answer questions about them – sometimes, well, kinda…dumb?

I mean, these things ace standardized tests, but then you show them something a kid could figure out and…BAM! Total fail. It's like they're book smart but lack common sense. So, what's the deal?

This paper we're looking at today suggests it might be because VLMs struggle with something called visually-grounded serial processing. Sounds complicated, right? Let's break it down.

Think about it like this: imagine you're trying to find your keys. You don't just magically know where they are. You serially process information. You look on the table, then maybe in your coat pocket, then perhaps under the couch cushions. Each step depends on the last. That's serial processing.

Now, visually-grounded means doing that with your eyes – solving a visual puzzle, counting objects, or mentally rotating something.

The researchers hypothesized that VLMs struggle with these tasks because they aren't very good at breaking down visual problems into a series of smaller, manageable steps. It's like trying to eat a whole pizza in one bite – messy and probably impossible! Instead of taking things one step at a time, VLMs try to process everything all at once, and that can be overwhelming.

To test this, the researchers designed a series of tasks in three areas:

• Geometric Reasoning: Think of this as shape puzzles. The more complex the puzzle, the more steps you need to figure it out.
• Perceptual Enumeration: Just counting things. But they made it harder by crowding the objects together, forcing you to carefully count each one individually.
• Mental Rotation: Like imagining turning a shape in your head. The harder the turn, the more mental steps required.

They compared how humans and VLMs performed on these tasks. Crucially, they also measured how long it took humans to complete each task. The longer it took a human, the more serial processing was likely involved.

And guess what? Across all the tasks, there was a clear trend: the more serial processing a task required (meaning, the longer it took humans), the worse the VLMs performed compared to humans! The VLMs' accuracy tanked as the human reaction time increased.

As tasks required composing geometric concepts, enumerating cluttered items, or performing complex mental transformations, the gap between VLM and human performance grew significantly.

"Limitations in serial, visually grounded reasoning represent a fundamental bottleneck that distinguishes current VLMs from humans."

In other words, VLMs struggle with tasks that require breaking down a visual problem into a series of steps, and this is a major reason why they sometimes fail at seemingly simple things.

Why does this matter?

• AI Researchers: This gives us a clue about where to focus our efforts to improve VLMs. We need to find ways to make them better at serial processing.
• AI Developers: This highlights the limitations of current VLMs. We need to be aware of these limitations when designing applications.
• Everyone Else: It's a reminder that even the most advanced AI systems aren't quite as smart as we think. Human intelligence is still unique and valuable!

So, here are a couple of questions that popped into my head while reading this paper:

• If VLMs are struggling with serial processing, how can we train them to get better at it? Can we design new architectures or training methods that encourage step-by-step reasoning?
• Could this limitation explain why VLMs sometimes struggle with tasks that require common sense? Is common sense, at least in part, about being able to break down complex situations into a series of smaller, more manageable steps?

That's all for this episode, learning crew! I'm Ernis, and I look forward to discussing this with you all on our next episode!

Credit to Paper authors: Nicholas Budny, Kia Ghods, Declan Campbell, Raja Marjieh, Amogh Joshi, Sreejan Kumar, Jonathan D. Cohen, Taylor W. Webb, Thomas L. Griffiths