Evaluation¶

ChemGraph's primary evaluation pipeline uses the deterministic structured-output judge. It asks each agent for a ResponseFormatter result, compares that result field-by-field with structured ground truth, and produces reproducible binary scores without a second judge model.

View published results on the ChemGraph Leaderboard.

The LLM-as-judge path remains available for qualitative or legacy comparisons, but it is secondary because it adds cost, nondeterminism, and judge-model bias.

No default benchmark dataset is bundled. Supply --dataset or define dataset in a selected TOML evaluation profile.

Main pipeline¶

Ground-truth structured_output
              ↓
Agent ResponseFormatter JSON
              ↓
Deterministic field comparators
              ↓
Per-query score: all expected fields pass = 1, otherwise 0
              ↓
Accuracy and per-field diagnostics

For every model, workflow, and query, the runner:

starts ChemGraph with structured output enabled;
captures the agent's tool calls, final result, and ResponseFormatter JSON;
compares each non-null expected field with the matching actual field;
records a binary score, per-field pass/fail values, and a rationale;
aggregates correct queries into structured-output accuracy.

Formatter/JSON parse failures score as incorrect and remain in the accuracy denominator. Ground truth with no non-null structured fields is also invalid and scores as a failure instead of receiving a trivial pass.

Quickstart¶

Run the structured judge explicitly. A judge model is not needed:

chemgraph eval \
  --models gpt-4o-mini \
  --dataset evaluation/questions.json \
  --workflows single_agent \
  --judge-type structured \
  --output-dir eval_results

Always include --judge-type structured unless a selected profile already sets it. The CLI's compatibility fallback is still llm when no judge type is specified.

Structured ground truth¶

The preferred dataset format is a JSON array. Each item needs a unique ID, query, and answer.structured_output object:

[
  {
    "id": "water-smiles",
    "category": "lookup",
    "query": "What is the SMILES string for water?",
    "answer": {
      "tool_calls": [
        {
          "name": "molecule_name_to_smiles",
          "args": {"name": "water"}
        }
      ],
      "result": "O",
      "structured_output": {
        "smiles": ["O"]
      }
    }
  }
]

Only non-null fields in the expected structured_output are judged, so include every result modality that the query is required to return.

Structured field	Expected shape	Comparison
`smiles`	List of SMILES strings	RDKit canonicalization; expected values matched independently of order
`scalar_answer`	`{value, property, unit}`	Value within 5% relative tolerance; normalized property category; unit match
`dipole`	`{value: [dx, dy, dz], unit}`	Components within 5% relative tolerance; normalized unit match
`vibrational_answer`	`{frequency_cm1: [...]}`	Imaginary modes removed; sorted real modes compared within 5%
`ir_spectrum`	`{frequency_cm1: [...], intensity: [...]}`	Frequencies and intensities compared within 5%
`atoms_data`	`{numbers: [...], positions: [...]}`	Atomic numbers exact; each coordinate within 0.1 Å

The default numeric tolerances come from judge_structured_output(). The CLI runner currently uses those defaults. The loader also accepts the legacy object format containing manual_workflow or llm_workflow, but new datasets should use the list format above.

Keep ground truth under version control when licensing permits, document how it was produced, and review it independently of every model being measured.

Profiles¶

Make structured judging the explicit default in a reusable profile:

[eval]
default_profile = "standard"

[eval.profiles.standard]
dataset = "evaluation/questions.json"
workflow_types = ["single_agent"]
judge_type = "structured"
structured_output = true
recursion_limit = 200
max_queries = 0

Then run:

chemgraph eval \
  --config config.toml \
  --profile standard \
  --models gpt-4o-mini

--profile requires --config. CLI values override matching profile values. If [eval] default_profile exists, providing --config without --profile selects it automatically.

The default recursion limit is 200 graph steps per query, increased from 50. A looping query can therefore run for four times as many steps before stopping, which can increase runtime and model/tool costs. Steps include middleware and tool execution; they are not a token or monetary budget, so cost does not scale by a fixed factor. Use --recursion-limit 50 to retain the previous budget, or set recursion_limit = 50 in an evaluation profile. Limits must be at least 1.

Deep Agent evaluation¶

The evaluator runs the existing deep_agent workflow with bundled skills, chemistry tool discovery, and one output directory per model/thread. File tools, shell commands, chemistry tools, and logs share real host paths under <output-dir>/logs/<model>/deep_agent/thread_<query-id>/ (model / and : become _; query IDs are URL-escaped). This directory is also the shell working directory and CHEMGRAPH_LOG_DIR; there is no /workspace alias. The agent is told to keep generated inputs, scripts, and results here. On Windows, built-in file tools use forward-slash paths without the drive prefix (for example, /Users/name/results/file.xyz); the backend supplies the workspace drive. Chemistry tools continue to use native host paths. The evaluation prompt provides both directory forms.

--output-dir is sufficient; --deepagent-workspace optionally replaces the <output-dir>/logs root. Queries require empty thread directories. Before loading models or changing checkpoints and reports, evaluation rejects any selected query whose directory is nonempty, unless a compatible checkpoint already records that query and --resume will reuse it. This also rejects artifacts from interrupted queries that have no checkpoint. Existing files are preserved. Use a new output directory for a fresh benchmark, and a new --deepagent-workspace root if that override is configured. A second check immediately before each query rejects files that appeared after preflight as an initialization error. Persistent memory and personal/project skill discovery are disabled. These directories separate artifacts; they are not operating-system sandboxes. Deep Agent evaluation requires --deepagent-auto-approve, or deepagent_auto_approve = true in the evaluation profile (deepagent_auto_approve=True in Python). Evaluation has no interactive approval handler, so missing approval is rejected before any models are initialized, including when Deep Agent is selected alongside other workflows. Use this option only in a trusted execution environment: it allows unattended tool execution, file changes, and shell commands.

chemgraph eval \
  --config config.toml \
  --models argo:gpt-4.1-mini \
  --dataset /path/to/groundtruth.json \
  --workflows deep_agent \
  --judge-type structured \
  --deepagent-auto-approve \
  --query-ids 3 5 6 7 8 9 31 \
  --output-dir eval_results/deepagent-smoke

For a proxy, add --base-url http://127.0.0.1:20219/v1 to the same command. It overrides provider URLs from config.toml for both execution and formatting (and the optional LLM judge). Profiles can also set base_url. When splitting a shell command across lines, each continuation backslash must be the last character on its line.

Download output_data.json from the ground-truth dataset at a fixed revision, and keep it outside the agent workspaces. The initial 40-query benchmark uses revision 8fe726352eb0437e4958282ca741d064fcfe6959. Omit --query-ids for the full dataset. IDs are selected in dataset order before --max-queries is applied. These options also work as profile keys: deepagent_workspace, deepagent_auto_approve, and query_ids.

After a completed run, evaluation uses the same model and provider settings, including the agent's resolved reasoning effort, to format the recorded messages into ResponseFormatter JSON, with the existing formatter prompt and one retry. This formatting call has no tools or ground truth. It does not change the Deep Agent graph. The original answer and state are preserved separately from structured_output and formatter_responses. Formatter replies are retained even if a later retry raises an exception. The deterministic judge and its tolerances are unchanged, including the orientation-sensitive dipole component comparison.

Each raw query record includes status, errors, workspace, elapsed seconds, and separate execution, formatting, and combined usage. Usage includes retries; unreported tokens remain unknown. Initialization and execution failures count as incorrect. Formatting failures, including invalid JSON and formatter model errors, count as incorrect for the structured judge. The LLM judge still scores the completed original answer and tool calls independently. Formatting failures retain their error status and diagnostics in raw records and remain included in the error counts in JSON and Markdown execution summaries.

Dataset and code checksums, dependency versions, and settings identify the run. Deep Agent resume requires a matching fingerprint, including the dataset, query_ids, max_queries, judge and execution settings, configured model/judge endpoints, recorded dependency versions, and hashed package source (.py, .md, and .template files). Provider environment variables are not fingerprinted; resume assumes the provider environment remains unchanged. Changing query-selection options invalidates checkpoints even if they select the same queries; raising --max-queries to extend a smoke test also requires a new output directory. Package source changes, including evaluator fixes, invalidate old checkpoints. There is no checkpoint migration. Use separate output directories for smoke tests and full runs; --resume reuses completed query records, including failures, when the fingerprint matches.

Reports and diagnostics¶

Select --report json, markdown, console, or all (the default). Structured results include:

structured_judge_aggregate: query count, correct count, accuracy, and parse errors for each model/workflow pair;
structured_judge_details: per-query score, field_scores, rationale, query metadata, and parse error;
raw tool calls and final agent results for debugging failures.

Use the per-field rationale to distinguish a wrong scientific value from a missing field, unit/property mismatch, geometry difference, or formatter parse failure.

Resume interrupted runs¶

The runner writes per-query checkpoints even when resume is disabled. Continue with the same dataset, model/workflow configuration, and output directory:

chemgraph eval \
  --models gpt-4o-mini \
  --dataset evaluation/questions.json \
  --judge-type structured \
  --output-dir eval_results \
  --resume

Use --max-queries for a smoke test, --recursion-limit to bound graph execution, and --tags for run metadata. Evaluation currently accepts single_agent, multi_agent, and deep_agent workflows.

Python API¶

import asyncio

from chemgraph.eval import BenchmarkConfig, ModelBenchmarkRunner


config = BenchmarkConfig(
    models=["gpt-4o-mini"],
    dataset="evaluation/questions.json",
    workflow_types=["single_agent"],
    structured_output=True,
    judge_type="structured",
    output_dir="eval_results",
)
runner = ModelBenchmarkRunner(config)
asyncio.run(runner.run_all())
runner.report(format="all")

Optional LLM judge¶

Use the LLM judge only when a deterministic structured comparison does not capture the evaluation goal or when reproducing a legacy benchmark:

chemgraph eval \
  --models gpt-4o-mini \
  --dataset evaluation/questions.json \
  --judge-type llm \
  --judge-model gpt-4o \
  --output-dir eval_results

Use --judge-type both to record deterministic and LLM-judge results side by side. Report the judge model and settings, and do not substitute the LLM score for the structured score on leaderboard-style comparisons.

Evaluation runs invoke real models and tools unless your test setup replaces them. Estimate cost, isolate credentials, and begin with a small dataset.