Fact-retrieval vs extrapolating AI agents: the difference, and how to build both

Ask an enterprise AI assistant "What is our refund policy for annual plans?" and a well-built retrieval agent will find the right paragraph, quote it and cite the source. Ask the same assistant "If we shorten the refund window to 14 days, what happens to churn next quarter?" and it will often produce an answer that looks just as confident, with just as many citations, and is far less trustworthy.

Those two questions need two different kinds of agent:

A fact-retrieval agent answers questions whose answer already exists somewhere: in a document, a database or an API. Its job is to find, verify and cite.
An extrapolating agent answers questions whose answer doesn't exist yet or exists nowhere: forecasts, what-if scenarios, risk estimates, recommendations and conclusions drawn across a whole body of evidence. Its job is to infer, quantify uncertainty and show its assumptions.

Most AI programmes build the first kind and then quietly use it for the second. This article explains why that fails, how the two differ, and how to build systems that do both properly.

Two kinds of question

	Fact retrieval	Extrapolation
Does the answer exist in your data?	Yes, somewhere	No, it has to be inferred
Typical question	"What does clause 12.3 say?" "What was Q2 revenue in Poland?"	"Will this supplier miss deliveries next quarter?" "What if we raise prices 5%?"
Core skill	Finding and quoting the right evidence	Combining evidence, base rates and models into an estimate
A good answer is	Correct and cited	Well calibrated, with assumptions and a range
Main failure	Wrong or missing source, invented citation	Overconfidence, hidden assumptions, facts and guesses mixed together
How to evaluate	Exact match against the source	Backtesting, calibration and scoring rules over many predictions
Can a person check it immediately?	Yes, by reading the source	Often only later, when reality happens

In practice it's a spectrum, not a switch:

Lookup  ->  Aggregation  ->  Synthesis  ->  Inference  ->  Forecast / what-if
"What is X?"  "How many X?"   "What are the   "Why did X    "What will happen
                              main themes?"   happen?"      if we do Y?"
|<------ fact retrieval ------>|<----------- extrapolation ----------->|

The further right a question sits, the less a retrieved document can answer it on its own.

What a fact-retrieval agent does well

Retrieval-augmented generation, introduced by Lewis et al. in 2020, gave language models a searchable external memory: retrieve the relevant passages, then generate an answer grounded in them. Done well, it is the right tool for a huge range of business questions:

policy, contract and regulation lookup;
customer support and internal knowledge bases;
product specifications and technical documentation;
compliance evidence ("show me where we document X");
"what did we decide, and when?" questions across meeting notes and tickets.

The engineering is mostly about retrieval quality. Anthropic's contextual retrieval work is a good example: adding context to each chunk before indexing, combining embeddings with keyword search (BM25) and adding a reranker cut failed retrievals by 49%, and by 67% with reranking, on their benchmarks.

Even so, retrieval doesn't make an agent infallible. A preregistered study by Stanford and Yale researchers found that leading AI legal research tools, all built on retrieval, still hallucinated in 17% to 33% of queries, fewer than general chatbots but far from "hallucination-free". Good fact-retrieval agents therefore also verify: they check that each claim is actually supported by the cited passage.

Why retrieval alone breaks on extrapolation questions

Point a fact-retrieval agent at a what-if or forecasting question and six things go wrong.

1. There is no document with the answer. No file says what churn will be after a policy change that hasn't happened. The agent retrieves the closest-looking passages and a language model fills the gap, which is extrapolation, just unacknowledged.

2. Top-k retrieval sees fragments, not the whole picture. Retrieval returns the few passages most similar to the question. Questions such as "what are the main risks across all 4,000 supplier reports?" need the whole corpus. Microsoft's GraphRAG research showed that conventional vector RAG fails on these "global sensemaking" questions, and that building a knowledge graph with community summaries produced far more comprehensive and diverse answers.

3. Citations make guesses look like facts. The agent cites real documents, but the conclusion it draws from them is an inference. The citation proves that the inputs exist, not that the conclusion follows. This "citation laundering" is the most dangerous failure because it looks trustworthy.

4. No uncertainty, no base rates. Retrieval returns text, not probabilities. Good forecasters start from base rates ("how often does this kind of thing happen?") and adjust. A retrieval agent has no such mechanism, and language models are systematically overconfident when asked to state their own confidence.

5. Language models are not calculators. Trends, growth rates, seasonality and sensitivities need real computation. A model that reads three numbers in a document and "estimates" a trend in prose is guessing.

6. Long contexts don't fix it. Stuffing more documents into the prompt helps less than people hope. The well-known Lost in the Middle study found that models use information at the beginning and end of a long context much better than information in the middle.

The reverse is also true: an extrapolating agent without retrieval invents its facts. A forecast built on made-up inputs is worthless however sophisticated the reasoning. The two capabilities need each other.

What good extrapolation looks like

Research on AI forecasting shows what works. Halawi et al. (2024) built a system that approaches the accuracy of competitive human forecasters, and in some settings surpasses it. Its pipeline is instructive: generate search queries, retrieve news published before the question date, filter for relevance, summarise, then ask the model several times for reasoning and a probability, and aggregate the answers.

On the public ForecastBench leaderboard, the best AI systems had likely reached parity with human superforecasters by mid-2026. The top approaches share the same ingredients: retrieval for evidence, several independent forecasts, and aggregation.

The lesson for enterprise agents: extrapolation is retrieval plus structure. Evidence gathering, explicit reasoning, repeated sampling, aggregation and calibration, not one clever prompt.

How to build agents that do both

The architecture we recommend separates evidence, computation and judgement, and makes the answer say which is which.

Question
   |
   v
Classify: lookup / aggregate / synthesise / extrapolate
   |
   +--> Evidence layer: hybrid search, reranking, SQL and APIs,
   |    time filters, citations, claim verification
   |
   +--> Computation layer (code): statistics, trends, time series,
   |    scenarios and simulations, sensitivity analysis
   |
   +--> Judgement layer (LLM): base rates, assumptions, scenarios,
   |    several independent estimates, aggregation
   |
   v
Answer contract: FACT (cited) / DERIVED (calculated) / ESTIMATE (range + assumptions)
   |
   v
Human decision, or action within an agreed authority

1. Classify the question first

Route each question before answering it. A lookup goes straight to the evidence layer; a forecast triggers the full pipeline. This is a small, high-volume judgement, a good fit for a fast classifier or a decision model rather than a frontier LLM.

2. Build a strong evidence layer, and use it for both

Hybrid retrieval (embeddings plus keyword search) with reranking.
Structured data through tools, not text: SQL queries and APIs for numbers, instead of hoping a PDF contains the right table.
Time awareness: when you forecast, and especially when you backtest, filter evidence to what was known at the time, or the evaluation will cheat.
Global questions: add a corpus-level index (knowledge graph, hierarchical summaries) when questions span the whole collection.
Claim verification: check that each cited passage actually supports the claim.

3. Put numbers in code

Trends, growth rates, correlations, seasonality, Monte Carlo simulations and scenario models belong in a computation layer the agent calls as a tool. The language model decides what to compute and explains the result. It doesn't do the maths in its head.

4. Make the judgement layer explicit

Start from a base rate. "Of comparable suppliers with two late deliveries, how many missed the next quarter?" Then adjust for the specifics.
State assumptions as a list, so a person can challenge them.
Sample several times and aggregate. Independent estimates, averaged, are consistently better than a single answer, as the forecasting research above shows.
Give ranges and scenarios, such as base, upside and downside, not one number.

5. Adopt an answer contract

Every statement in the final answer is labelled:

Label	Meaning	Example
FACT	Found in a source, with a citation	"Q2 churn in Poland was 3.1% [billing DB, 2026-07-01]"
DERIVED	Calculated from facts, with the calculation available	"That is 0.6 points above the trailing 4-quarter average"
ESTIMATE	Inferred, with assumptions, range and confidence	"Shortening the refund window likely raises churn by 0.3 to 0.9 points (medium confidence; assumes no competitor response)"

This one convention prevents most "citation laundering" and makes reviews much faster, because people know which sentences to check against sources and which to challenge on assumptions.

6. Evaluate each kind differently

	Fact retrieval	Extrapolation
Test set	Questions with known answers and source passages	Past questions whose outcomes are now known
Metrics	Retrieval recall, answer accuracy, citation support rate	Brier score or log loss, calibration curve, interval coverage, error against a naive baseline
Key discipline	Check every citation actually supports the claim	Backtest using only evidence available at the time
In production	Monitor unanswered and low-support answers	Track forecasts against outcomes as they resolve; recalibrate

A useful sanity check for any extrapolating agent: does it beat a simple baseline, such as "same as last quarter" or the historical average? If it doesn't, it is adding confidence, not insight.

7. Match execution authority to the kind of answer

Facts can often drive automated actions directly: a verified policy lookup can close a support ticket. Estimates should usually recommend, not act, unless they fall within tight, pre-agreed limits and have a track record. We discuss this in more detail in Horizontal vs vertical AI agents is the wrong question.

Use cases: which kind do you need?

Use case	Kind	What it needs beyond basic RAG
Policy and contract Q&A	Fact retrieval	Precise citations, claim verification
Customer support knowledge base	Fact retrieval	Freshness, access control per user
Compliance evidence collection	Fact retrieval	Complete coverage, audit trail
"Main themes in 10,000 support tickets"	Global synthesis	Corpus-level index or clustering, not top-k retrieval
Supplier or credit risk scoring	Extrapolation	Base rates, historical outcomes, calibrated scores
Demand, churn or pipeline forecasting	Extrapolation	Statistical models in code; LLM for drivers and narrative
Pricing and policy what-if analysis	Extrapolation	Scenario models, explicit assumptions, ranges
Incident root-cause hypotheses	Inference	Retrieval over logs and changes, ranked hypotheses with evidence
Investment or M&A due diligence	Both	Cited facts from the data room plus clearly labelled risk estimates
Board or management reporting	Both	Facts and derived numbers, then labelled outlook

Most valuable enterprise questions sit in the last few rows: they need facts and judgement, clearly separated.

Common mistakes

Treating forecasting as a prompt. "Predict next quarter's churn" in a chat window, without data tools, base rates or evaluation.
Trusting stated confidence. A model saying "I'm 90% sure" is not calibration. Calibration comes from comparing many predictions with outcomes.
Backtesting with future information. Letting the agent retrieve documents written after the forecast date makes it look brilliant in tests and useless in production.
Mixing facts and estimates in one paragraph, so reviewers can't tell which is which.
No baseline. If you never compare with "same as last period", you can't tell whether the agent adds value.

Frequently asked questions

What is the difference between a fact-retrieval agent and an extrapolating agent?

A fact-retrieval agent answers questions whose answer already exists in documents, databases or APIs, and succeeds by finding and citing the right evidence. An extrapolating agent answers questions whose answer doesn't exist yet, such as forecasts, what-if scenarios and risk estimates, and succeeds by combining evidence, base rates and models into a calibrated estimate with stated assumptions.

Can a RAG system make forecasts?

Not on its own. RAG retrieves existing text; it has no mechanism for base rates, uncertainty or computation, so forecasts from plain RAG are unacknowledged guesses with citations attached. Forecasting needs retrieval plus a computation layer, explicit reasoning, several aggregated estimates and calibration against past outcomes.

How do you evaluate an AI agent that makes predictions?

Backtest it on past questions whose outcomes are known, using only evidence available at the time. Measure accuracy with proper scoring rules such as the Brier score, check calibration and interval coverage, and compare against a simple baseline like the historical average.

Why does GraphRAG help with some questions?

Conventional retrieval returns the few passages most similar to a question, which works for specific lookups but fails for questions about a whole corpus, such as its main themes. GraphRAG builds a knowledge graph and community summaries in advance, so the system can reason over the entire collection.

How we can help

We build both kinds of agent and, more often, the combination: retrieval with verified citations, computation layers for trends and scenarios, forecasting pipelines with backtesting and calibration, and answer formats that keep facts and estimates apart. A typical starting point is a short Architecture Sprint to classify your target questions and design the evidence, computation and judgement layers they need.

Have questions your current assistant answers too confidently? Book a consultation.

Sources

Lewis et al. (2020): Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Anthropic: Introducing Contextual Retrieval
Magesh et al. (Stanford, Yale): Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools
Edge et al. (Microsoft Research): From Local to Global: A Graph RAG Approach to Query-Focused Summarization
Liu et al. (2023): Lost in the Middle: How Language Models Use Long Contexts
Halawi et al. (2024): Approaching Human-Level Forecasting with Language Models
Forecasting Research Institute: AI models have likely reached parity with superforecasters on ForecastBench
KalshiBench: evaluating epistemic calibration of LLMs via prediction markets