AI Models Struggle with These Puzzles: Can You Outsmart Them?

Puzzles and games have been integral to the development of artificial intelligence (AI) since its inception. Just as we humans enjoy testing our cognitive abilities with crosswords or logic puzzles, developers use similar challenges to gauge how far AI models have progressed. The term "machine learning" was coined by IBM computer scientist Arthur Samuel in a 1959 article about an algorithm that learned to play checkers. Chess and the Chinese board game Go are also well-known test beds for AI.

AI's puzzle-solving capabilities have improved significantly over the years, but they still fall short in certain areas. In late 2024, a team from Columbia University found that even the best models could only solve 18% of New York Times Connections puzzles. By early 2025, however, some models had nearly perfected this task. This rapid progress highlights AI's potential but also reveals its limitations.

Where Models Fall Short

Despite these advancements, today’s models still struggle with specific types of puzzles. Subtle changes in classic riddles often trip them up, and visual puzzles are a particular weak spot. For instance, spatial reasoning tasks like mental rotation problems-where you have to determine if different images represent the same object from different angles-are notoriously difficult for AI.

• Spatial Reasoning: Even though language models can analyze visual inputs, they fail miserably at these puzzles. World models, which are designed to help AI understand physical environments, don't seem to enable LLMs to manipulate 3D objects as well as humans do.
• Memory Overload: Frontier LLMs have extraordinary memories, having been trained on vast amounts of data. This can be a double-edged sword. When a puzzle closely resembles one seen during training, the model may overlook key differences and respond with memorized answers.

A 2024 study by researchers from Google and the University of California, Berkeley, highlighted this issue. They found that models often failed to recognize subtle variations in puzzles they had encountered before, leading to incorrect solutions.

Key Takeaways

1. AI's Rapid Progress: AI models have made significant strides in solving complex puzzles, but they still have blind spots.
2. Spatial Reasoning Challenges: Despite advancements, AI struggles with tasks that require spatial reasoning and 3D object manipulation.
3. Memory vs. Understanding: While LLMs have vast memories, this can sometimes lead to over-reliance on memorized solutions rather than true understanding.

To better understand where you stand relative to AI, try your hand at these puzzles:

Spatial Reasoning Test

Instructions: Choose the answer that shows the object in the prompt but from a different angle. In each case, there’s only one correct answer!

1. Prompt Image 1

   • A: [Image]
   • B: [Image]
   • C: [Image]

2. Prompt Image 2

   • A: [Image]
   • B: [Image]
   • C: [Image]
   • D: [Image]

Memory and Understanding Test

Instructions: Solve the following riddle, which has a subtle twist from a common puzzle.

• Riddle: Two fathers, two sons. Who are they?

By testing yourself on these puzzles, you can gain insight into where AI excels and where it still lags behind human cognition. While AI continues to advance, there are still areas where human intuition and reasoning outperform even the most sophisticated models.