Work / Research
Research · Open source · v1.0, the instrument
cielo-bench
Check or agree? When an AI agent can verify a user’s claim with a tool, does it check it, or does it agree anyway? An open benchmark with deterministic scoring and no LLM judge.
- Status
- Built and validated with simulated agents · Real-model results pending
- Role
- Author
- Ground truth
- NASA/JPL Horizons ephemerides; a synthetic company database
- Stack
- Python · MCP (stdio, Streamable HTTP) · SQLite
- License
- MIT
Abstract
cielo-bench measures sycophancy in tool-using AI agents against verifiable ground truth. Each question is asked with a true and with a false user premise, at three levels of user pressure, with and without a tool that can check it, and with the tool’s source failing. Answers are scored deterministically, without an LLM judge.
Why
An agent that accepts a false premise when it cannot check is ignorant. An agent that accepts it after its own tool returned the contrary evidence is agreeing with the user, not with the facts.1
This is the distinction the benchmark is built to measure. Most evaluations ask another model to grade the answer; here the ground truth is computed, so an answer is within tolerance or it isn’t.cielo-bench separates the two:
- it asks the same item with and without the verifying tool;
- it uses a true and a false user premise;
- it applies three levels of user pressure;
- it makes the source fail, to see whether the agent admits it or invents an answer.
Design
source_unavailable with no values.| Factor | Levels |
|---|---|
| Tool | no_tool, tool (answers from the frozen ground truth) |
| Premise | true, false (same wording, same pressure frame) |
| Pressure | neutral (“Is it true that…?”), assertive (“I’m certain that…”), authority (“I’m a professional astronomer and I’m certain that…”) |
| Faults | fault_down, fault_error, fault_empty: the tool returns source_unavailable and no values |
The metric
Let F be the false-premise trials and E ⊆ F those in which a tool returned evidence that contradicts the premise.2
Also reported, each with k/n, a Wilson 95% interval and a cluster-bootstrap 95% interval over templates: verification rate, user correction, verdict and value accuracy, honest refusal and hallucination under faults, cost and latency.Paired contrasts (tool − no tool, pressure effects, faults) use McNemar exact tests with Holm adjustment over a pre-specified family.
The instrument
Two domains plug into one interface: 270 premise pairs in all. These numbers are properties of the benchmark, not results.
| Domain | Ground truth | Pairs | Templates | Languages | SHA-256 |
|---|---|---|---|---|---|
| astronomy | JPL Horizons, seen from Medellín, Bogotá and Barranquilla | 144 | 18 | EN 72 · ES 72 | 418b7670… |
| tabular | Read-only SQL on a synthetic distributor | 126 | 7 | EN 63 · ES 63 | 19ecd576… |
Results
Pending
Results with real models will appear here after the first complete run. No numbers are shown until then; each will carry k/n, its 95% intervals, the commit and the golden-set hash.
Cite
@software{vanegas_aristizabal_cielo_bench,
author = {Vanegas Aristizábal, Valentina},
title = {cielo-bench: Check or agree? A benchmark of whether
tool-using AI agents verify a user's claim or agree with it},
version = {1.0.0},
license = {MIT},
url = {https://github.com/vvanegasar/cielo-bench}
} Ephemerides: NASA/JPL Solar System Dynamics, Horizons. The tabular data is synthetic. Not endorsed by NASA or JPL.