LLMs Show Significant Progress in Geolocation Tasks, But Challenges Remain

June 6, 2025

In a recent study by Bellingcat, the geolocation capabilities of Large Language Models (LLMs) from major AI developers such as OpenAI, Google, Anthropic, Mistral, and xAI were put to the test. This follow-up to their July 2023 analysis reveals significant improvements in LLMs' ability to accurately identify locations from images, but also highlights ongoing challenges.

The Test Setup

To assess the current state of geolocation performance, Bellingcat conducted a series of 500 tests using 20 different models. Each model was tasked with identifying the location of 25 unique travel photos, which were chosen to represent a range of outdoor scenes from various continents, including both rural and urban areas.

• Image Selection: The 25 images included scenes like city streets, fields, and parked vehicles, with varying levels of difficulty in terms of geolocation. Importantly, none of these images had been published online before the test to ensure they were truly novel for the models.
• Prompting: Each LLM received a photo without any metadata and was asked, “Where was this photo taken?” If an LLM requested additional information, it was told, “There is no supporting information. Use this photo alone.”

Models Tested

The study included both older and newer versions of the models to track their evolution over time. Here's a breakdown of the models tested:

• Anthropic

  • Claude Haiku 3.5: "fastest model for daily tasks"
  • Claude Sonnet 3.7: "our most intelligent model yet"
  • Claude Sonnet 3.7 (extended thinking): "enhanced reasoning capabilities for complex tasks"
  • Claude Sonnet 4.0: "smart, efficient model for everyday use"

• Google

  • Google Lens (for comparison)

• OpenAI

  • GPT-4
  • GPT-3.5

• Mistral

  • Mistral-12B

• xAI

  • xAI-7B

Key Findings

Improved Accuracy

Compared to the 2023 tests, all models showed significant improvements in geolocation accuracy. This is a testament to the rapid advancements in LLMs over the past two years.

• Claude Sonnet 3.7 (extended thinking): Demonstrated the highest accuracy among the tested models, correctly identifying locations with a high degree of confidence.
• GPT-4: Showed notable improvements over GPT-3.5, particularly in handling complex scenes without clear landmarks.
• Google Lens: While still effective for reverse image search, it did not outperform the best LLMs in this specific task.

Ongoing Challenges

Despite the progress, several challenges remain:

• Hallucinations: Some models still struggled with hallucinations, providing incorrect or overly confident answers when faced with ambiguous images.
• Contextual Understanding: While many models could identify landmarks, they sometimes failed to place them within the correct geographical context. For example, identifying a famous building but placing it in the wrong city.
• Data Diversity: The performance of LLMs was generally better for locations and scenes that are well-represented in their training data. This highlights the need for more diverse training datasets.

Implications for Practitioners

For researchers and practitioners working with geolocation tasks, these findings offer several insights:

• Model Selection: When choosing a model for geolocation tasks, consider models like Claude Sonnet 3.7 (extended thinking) or GPT-4, which have demonstrated strong performance.
• Data Preprocessing: Ensure that the images used for testing are as close to real-world conditions as possible. Avoid using images with metadata or those that have been previously published online.
• Contextual Information: While LLMs can provide valuable insights, they should be used in conjunction with other tools and human oversight, especially in critical applications.

Conclusion

The rapid evolution of LLMs has brought significant improvements to their