Evaluation

Experiments

Run versioned JSON suites and retain readable, reproducible results without mixing generated artifacts into the library.

The orchestrator on Mouse–Human

experiments/agentic_workflow runs each subagent on the OAEI anatomy track under five routing policies. Answers are shared across policies, and each policy is charged for the questions it asked.

With llama3.1 as the local tier, audits show it disagrees with the API model too often, so the orchestrator stops trusting it and routing saves little beyond the free evidence tier. Reasoning has fewer questions than one recalibration round, so the routed policy kept llama3.1's placements there. Every decision is in the showcase.

python experiments/agentic_workflow/run.py --json results/agentic_workflow/2026-10-18.json
python showcase/build.py results/agentic_workflow/2026-10-18.json

Start with the smoke benchmark

./scripts/run_experiments.sh smoke
./scripts/run_experiments.sh smoke
cat results/experiments/smoke/summary.md

This fast benchmark compares baseline relation normalization with structural refinement, then reports their precision, recall, and F1. It requires no LLM, network access, or model download. On the second run, the report marks each variant reproducible when its metric fingerprint matches the previous run for the same configuration and source revision.

Hosted tracking

uv sync --extra tracking
uv run wandb login
./scripts/run_experiments.sh smoke --tracker wandb

The Agentic Ontologist publishes a comparison table, scalar metrics, logs, manifests, and generated figures to Weights & Biases. Temporary local artifacts are deleted after a successful upload. Set WANDB_PROJECT and optional WANDB_ENTITY to choose the destination, or add --keep-local to retain both copies.

Hosted artifacts can contain ontology-derived output. Use a private W&B project and confirm your data policy before enabling remote tracking.

Compare context selection

./scripts/run_experiments.sh context

This local comparison scores the same retrieved shortlist with passthrough context and a shadow Jev policy. It reports selection precision, recall, F1, returned context size, and rejection rate without calling the Jev API. Live Jev scoring remains opt-in through AGONTO_CONTEXT_HARNESS=jev.

Fetch benchmark data

git lfs install
git lfs pull

Bundled OWL, RDF, Turtle, and model artifacts use Git LFS. Fetch them before running maintained suites; dry runs do not require the data.

Inspect before running

./scripts/run_experiments.sh quickstart --dry-run
./scripts/run_experiments.sh ablations --dry-run

Run

# Quick comparison
./scripts/run_experiments.sh quickstart

# All maintained ablations
./scripts/run_experiments.sh ablations

# One experiment
./scripts/run_experiments.sh ablations --select with-uq

The equivalent direct command is python -m experiments CONFIG.json. The shell entry point resolves named maintained suites and uses the repository environment.

Maintained ablations

Local artifacts

results/experiments/smoke/
├── suite.manifest.json
├── summary.md
├── metrics.csv
├── figures/quality.svg
├── baseline-relations.latest.md
├── refined-relations.latest.md
└── runs/<experiment>/<timestamp>-<config-hash>/
    ├── stdout.log
    ├── metrics.json
    ├── manifest.json
    └── summary.md

Run directories are immutable, so repeated experiments never overwrite one another. Start with summary.md: it presents one row per experiment with precision, recall, F1, runtime, and reproducibility. Use metrics.csv for plotting and *.latest.md only when you need details.

For suites with two or more variants, the Agentic Ontologist generates figures/quality.svg automatically. A single run stays table-only because a comparison chart would be redundant.

The manifest records the command, timestamps, duration, Git revision, dirty-worktree state, configuration fingerprint, and metrics fingerprint. Timing does not affect reproducibility comparison. Standard pipeline experiments also receive a distinct output path for each run.

The local tracker is the default. Generated results remain ignored by Git and are reproducible from configuration.