GeoLLM-QA explained: the benchmark for tool-augmented AI agents
This article explains what the GeoLLM-QA benchmark tests, how it is built from 1,000 tasks and 117 tools, why the success rates come out so low, where AI agents fail and what the method means for production AI agents in companies.
GeoLLM-QA is an academic benchmark that tests large language models as tool-augmented agents. Instead of a single question, the model receives an instruction in natural language and has to solve it on a real remote sensing platform through a chain of verbal, visual and click-based actions. The test comprises 1,000 tasks and 117 tools and comes from the paper Evaluating Tool-Augmented Agents in Remote Sensing Platforms by Singh, Fore and Stamoulis (ICLR 2024). The result is sobering: GPT-4 Turbo reaches at best 34.99 percent success, GPT-3.5 Turbo 32.95 percent. No prompting method lifts the rate above a third. The most common error is the missed function, in more than half of the failure cases the agent does not even call a required tool. On the broader, less UI-heavy base GeoLLM-Engine with over 521,000 tasks, GPT-4 by contrast reaches 84.31 percent correctness, which shows how much real UI and multi-step tasks stress the models. For companies that build or buy AI agents, the method behind the benchmark is what counts: a task-specific evaluation with clear ground truth instead of a convincing demo.
What GeoLLM-QA measures and why it matters
GeoLLM-QA is a benchmark that tests how well an AI agent handles real analysis tasks on a remote sensing platform. Instead of a single question, the model receives an instruction in natural language and has to solve it through a chain of tool calls, map clicks and image analysis. That is what makes it more meaningful than classic question-and-answer tests.
The benchmark comes from the paper Evaluating Tool-Augmented Agents in Remote Sensing Platforms by Simranjit Singh, Michael Fore and Dimitrios Stamoulis, presented at the ICLR 2024 ML4RS workshop. Remote sensing means the analysis of satellite and aerial imagery, such as object detection, land cover classification or damage assessment after disasters.
The core point: real analyst work is never a single question but a multi-step process whose context sits in the system state, in what the map shows or what a previous step has already set. A test that captures this measures something different from a model that phrases a neat answer to an isolated question.
How the benchmark is built
GeoLLM-QA comprises 1,000 tasks on a platform with 117 tools. The tasks call for verbal, visual and click-based actions, that is language, image analysis and direct interaction with a map interface. That brings the test close to an analyst's daily work.
- The 117 tools range from Plotly Mapbox map APIs to LangChain routines for FAISS vector stores.
- The tasks are built in three steps: 25 reference templates as patterns, then an LLM-guided generation of 1,000 tasks at a 9-to-1 ratio of solvable to unsolvable, then a human-validated ground truth.
- Deliberately seeding unsolvable tasks tests whether an agent recognises when an instruction cannot be carried out at all, instead of inventing a plausible but wrong answer.
Why GPT-4 manages only a third
On GeoLLM-QA, GPT-4 Turbo reaches a best success rate of 34.99 percent, GPT-3.5 Turbo comes to 32.95 percent. For tasks that a person with platform experience solves routinely, that is a clear signal that tool use stays hard in practice.
| Model and test set | Method | Best success rate |
|---|---|---|
| GPT-4 Turbo on GeoLLM-QA | Chain-of-Thought zero-shot | 34.99% |
| GPT-3.5 Turbo on GeoLLM-QA | ReAct few-shot | 32.95% |
| GPT-4 Turbo on GeoLLM-Engine | ReAct few-shot (correctness) | 84.31% |
- GPT-4's best result comes from Chain-of-Thought zero-shot, GPT-3.5's from ReAct few-shot. No prompting trick lifts the rate above the one-third mark.
- Per task, GPT-4 uses between 8,700 and 13,100 tokens, GPT-3.5 between 7,400 and 12,900. The cost rises with the length of the tool chain.
- The small gap between GPT-4 and GPT-3.5 shows that both models fail on the same complex chains. A larger model alone does not solve the problem.
Where agents fail
The dominant error class is the missed function. In more than 50 percent of the failure cases the agent does not even call a required tool. Wrong arguments and the wrong order add to this, but the plain omission is the biggest lever.
The more tool calls a task needs, the more the success rate drops. Single-tool tasks sit far above multi-step tasks. For a production agent that means: it is not the number of available functions that decides, but whether it picks the right one at the right time.
- Context comes from the system state, not from the prompt. Models that only look at the wording miss what the map or the previous action has already set.
- Simply growing the number of tasks does not make the test harder. The success rate stays stable from 500 to 10,000 tasks. It is the complexity of a task, not its quantity.
- Similar effects appear in other domains. The article on BankerToolBench and AI hallucinations in finance describes the same break between a clean answer and correct tool use.
From GeoLLM-Engine to multi-agent
GeoLLM-QA is the curated, UI-heavy test set. Beneath it sits GeoLLM-Engine, an environment with more than 175 tools and over half a million generated tasks. On this broader set the models are markedly better, which shows how much the realistic UI coupling adds.
- GeoLLM-Engine generated its dataset with 100 parallel GPT-4 Turbo nodes: 521,868 tasks across 1,149,612 satellite images from seven open datasets.
- On this set GPT-4 reaches 84.31 percent correctness and 81.11 percent success with ReAct few-shot, far above the roughly 35 percent on the UI-heavy GeoLLM-QA.
- The follow-up work Multi-Agent Geospatial Copilots from 2026 separates orchestration and domain task and delegates to specialised sub-agents. This pattern is meant to reduce exactly the missed-function errors that slow the single agent down.
How such a design behaves in practice is set out in the article on the architecture of autonomous AI agents and risk mitigation . More autonomy needs more control, otherwise the small omissions add up to a wrong result.
What the benchmark means for companies
GeoLLM-QA is a remote sensing benchmark, but the lesson holds for any production AI agent: a convincing demo says little about everyday reliability. Anyone running agents needs a task-specific evaluation of their own.
- Measure, don't hope: define real tasks with ground truth and measure the success rate, not only whether an answer sounds plausible.
- Keep tool chains short: fewer, well-described tools beat an overloaded menu, because omission is the most common error.
- Make the system state explicit: whatever the interface or a previous step has set belongs visibly in the agent's context, not only in the original instruction.
How quickly the gap between a model's promise and its real-world performance shows up when you measure systematically is also clear from the benchmark comparison of Chinese AI models and the reality check on Agent Skills .
Challenges and risks
Benchmarks never capture the full reality, and low numbers invite false conclusions. On balance, GeoLLM-QA is a tool for understanding, not a final verdict on what language models can do.
- Model vintage: a benchmark result from 2024 does not transfer one to one to newer models. The numbers are a starting point, not a judgement on today's state of the art.
- Transferability: domain-specific benchmarks like GeoLLM-QA cannot be transferred to other fields unchecked. What transfers is the method, not the percentage.
- Value despite a low rate: a 35 percent success rate does not mean worthless. As an assistant with human oversight, agents can still save time, as long as errors can be caught.
- Cost: the token cost per task adds up quickly over long chains. Without cost measurement a production agent gets expensive before it is reliable.
What companies should do now
Anyone building or buying AI agents should adopt the method behind GeoLLM-QA, not the numbers. Four steps take priority.
Four priority steps
-
Build your own eval set
Assemble 20 to 50 real tasks with clear ground truth, including a few unsolvable cases as a control. That makes it visible whether the agent recognises when an instruction cannot be carried out.
-
Classify the errors
Do not only count the success rate, but sort the errors: missed function, wrong argument, wrong order. That shows where to start.
-
Keep the tool catalogue lean
Every extra function raises the risk that the agent overlooks the right one. Few, clearly documented tools are more reliable than an overloaded menu.
-
Consider multi-agent patterns
When a chain regularly breaks over many tool calls, it helps to separate orchestration and domain task and to delegate to specialised sub-agents.
Further reading
Frequently asked questions
GeoLLM-QA is a benchmark from the paper Evaluating Tool-Augmented Agents in Remote Sensing Platforms (Singh, Fore, Stamoulis, ICLR 2024). It tests large language models as tool-augmented agents on 1,000 real analysis tasks on a remote sensing platform with 117 tools. Instead of a single question, the model has to solve an instruction through a chain of verbal, visual and click-based actions.
On GeoLLM-QA, GPT-4 Turbo reaches a best success rate of 34.99 percent (Chain-of-Thought zero-shot), while GPT-3.5 Turbo reaches 32.95 percent (ReAct few-shot). No prompting method lifts the rate above the one-third mark. The small gap shows that both models fail on the same complex tool chains.
The dominant error class is the missed function. In more than 50 percent of failure cases the agent does not even call a required tool, regardless of the prompting strategy. On top of that, the success rate falls the more tool calls a task requires, and much of the context sits in the system state rather than the prompt.
GeoLLM-Engine is the underlying environment with more than 175 tools and over 521,000 generated tasks across 1.1 million satellite images. GeoLLM-QA is the curated, UI-heavy test set of 1,000 tasks. On the broad Engine set GPT-4 reaches 84.31 percent correctness with ReAct few-shot, but only about 35 percent on the UI-heavy GeoLLM-QA.
The lesson reaches beyond remote sensing: a convincing demo says little about everyday reliability. Anyone running AI agents in production should build their own task-specific evaluation with clear ground truth, classify errors instead of only counting the success rate, keep the tool catalogue lean and put the system state explicitly into the agent's context.