Skip to content
Valentina Vanegas Aristizábal

Work / Research

Research · Open source · v1.0, the instrument

cielo-bench

Check or agree? When an AI agent can verify a user’s claim with a tool, does it check it, or does it agree anyway? An open benchmark with deterministic scoring and no LLM judge.

Status
Built and validated with simulated agents · Real-model results pending
Role
Author I specify, direct and validate; implementation is agent-assisted.
Ground truth
NASA/JPL Horizons ephemerides; a synthetic company database
Stack
Python · MCP (stdio, Streamable HTTP) · SQLite
Links
Code ↗ Method ↗ Cite
License
MIT

Abstract

cielo-bench measures sycophancy in tool-using AI agents against verifiable ground truth. Each question is asked with a true and with a false user premise, at three levels of user pressure, with and without a tool that can check it, and with the tool’s source failing. Answers are scored deterministically, without an LLM judge.

Why

An agent that accepts a false premise when it cannot check is ignorant. An agent that accepts it after its own tool returned the contrary evidence is agreeing with the user, not with the facts.1

This is the distinction the benchmark is built to measure. Most evaluations ask another model to grade the answer; here the ground truth is computed, so an answer is within tolerance or it isn’t.

cielo-bench separates the two:

  • it asks the same item with and without the verifying tool;
  • it uses a true and a false user premise;
  • it applies three levels of user pressure;
  • it makes the source fail, to see whether the agent admits it or invents an answer.

Design

The flow of the design: a premise pair goes through three levels of user pressure into three conditions, no tool, tool and source fails. The tool answers from frozen ground truth; the failing source returns no values. Every agent reply is scored deterministically, with no LLM judge.Premise pairtrue / falseUser pressureneutral · assertive · authorityno tooltoolfrozen ground truthsource failsdown · error · emptyAgent replyverdict + valueDeterministic scorerno LLM judgeThe flow of the design: a premise pair goes through three levels of user pressure into three conditions, no tool, tool and source fails. The tool answers from frozen ground truth; the failing source returns no values. Every agent reply is scored deterministically, with no LLM judge.Premise pairtrue / falseUser pressureneutral · assertive · authorityno tooltoolfrozenground truthsource failsdown, error,emptyAgent replyverdict + valueDeterministic scorerno LLM judge
Fig. 1 The design. The same item runs in three conditions; the tool answers from a frozen, hashed copy of the ground truth, and the failing source returns source_unavailable with no values.
Table 1 Factors and levels.
FactorLevels
Toolno_tool, tool (answers from the frozen ground truth)
Premisetrue, false (same wording, same pressure frame)
Pressureneutral (“Is it true that…?”), assertive (“I’m certain that…”), authority (“I’m a professional astronomer and I’m certain that…”)
Faultsfault_down, fault_error, fault_empty: the tool returns source_unavailable and no values

The metric

Let F be the false-premise trials and E ⊆ F those in which a tool returned evidence that contradicts the premise.2

Also reported, each with k/n, a Wilson 95% interval and a cluster-bootstrap 95% interval over templates: verification rate, user correction, verdict and value accuracy, honest refusal and hallucination under faults, cost and latency.

Paired contrasts (tool − no tool, pressure effects, faults) use McNemar exact tests with Holm adjustment over a pre-specified family.

The instrument

Two domains plug into one interface: 270 premise pairs in all. These numbers are properties of the benchmark, not results.

Table 2 Golden sets, v1.
DomainGround truthPairsTemplatesLanguagesSHA-256
astronomyJPL Horizons, seen from Medellín, Bogotá and Barranquilla14418EN 72 · ES 72418b7670…
tabularRead-only SQL on a synthetic distributor1267EN 63 · ES 6319ecd576…

Every fact is re-derivable offline from hashed fixtures, and every report records the commit, the golden-set hash, the model versions, the design, the prices and the run date.

Results

Pending

Results with real models will appear here after the first complete run. No numbers are shown until then; each will carry k/n, its 95% intervals, the commit and the golden-set hash.

Cite

@software{vanegas_aristizabal_cielo_bench,
  author  = {Vanegas Aristizábal, Valentina},
  title   = {cielo-bench: Check or agree? A benchmark of whether
             tool-using AI agents verify a user's claim or agree with it},
  version = {1.0.0},
  license = {MIT},
  url     = {https://github.com/vvanegasar/cielo-bench}
}

Ephemerides: NASA/JPL Solar System Dynamics, Horizons. The tabular data is synthetic. Not endorsed by NASA or JPL.