Audit, report, campaign¶

A run with six green gates has proved its number is real. It has not yet proved the number is worth anything. That judgement is recorded explicitly, with a rationale, by science-adk audit.

First, check the gates¶

science-adk score

  [pass] execution_completed: All nodes ran to completion.
  [pass] all_nodes_ran: 4 node(s) produced output.
  [pass] result_measured: evaluate reported a measurement.
  [pass] metric_is_finite: evaluate reported 0.989997.
  [pass] produced_output: 6 output value(s) recorded.
  [pass] tool_use_verified: Declared tool use matches the record.

If anything failed, stop. There is no way to audit past a failed gate, and there should not be. Fix the cause and run again.

Record the verdict¶

science-adk audit \
  --scientific-value 0.85 \
  --method-fidelity 0.90 \
  --implementation-quality 0.95 \
  --rationale "Recovered exponent 1.4555 ± 0.05 from 3565 archive planets with r² = 0.99, against the theoretical 1.5. The ~3% shortfall is consistent with the spread in host star masses, since Kepler's constant goes as (4π²/GM)^½. The fit is blind: nothing in the pipeline assumes 3/2."

All three pillars are required, each in [0, 1], and so is the rationale:

error: Missing rating for pillar(s): method_fidelity
error: An audit needs a rationale explaining the ratings.

The score is the unweighted mean of the three, rounded to four places, and zero if any gate failed.

Rating the pillars¶

Pillar	Ask	High when	Low when
`scientific_value`	Does this move the question forward?	Someone would change what they do next because of it.	It confirms something already known, or answers a question nobody asked.
`method_fidelity`	Is the method sound?	Assumptions are stated, the comparison is fair, the data supports the claim.	Leakage, an unfair baseline, an unstated approximation doing the real work.
`implementation_quality`	Is the code right?	Reproducible, readable, tools tested, seeds fixed.	Warnings suppressed, magic constants, a `--reuse` run audited as fresh.

Writing the rationale¶

The rationale is the part a reader will actually rely on. It should contain the numbers:

Good

"GNN embedding reaches RMSE 0.72 on held-out ESOL against 0.95 for Morgan fingerprints on the same seeded split (n=1128, 80/20). The gap exceeds the 0.04 spread across five seeds, so it is unlikely to be split noise."

Not good

"Works well, good result."

Every warning you chose to accept belongs here too — a suppressed numerical warning, an unfair-looking baseline you have a reason for, an approximation that matters.

Generate the report¶

science-adk report

Writes REPORT.md into the run directory from the trace and the score: the hypothesis, the DAG, each node's logs and outputs, the gates, the pillars and the rationale. It is generated, so it never disagrees with the run it describes.

cat research/001-kepler-exoplanets/runs/*/REPORT.md

Record the learning¶

science-adk learn \
  "Kepler's Third Law holds across 3565 confirmed exoplanets with r² = 0.99; the pooled exponent runs ~3% below 3/2 because of host star mass spread." \
  --run latest

LEARNINGS.md is append-only and cites its evidence. A learning is a claim you are prepared to build the next experiment on, so it should state a number and name the run that produced it — not "the GNN seemed better".

The campaign leaderboard¶

science-adk campaign
science-adk experiment list
science-adk status

CAMPAIGN.md ranks every experiment by its audited score and shows lineage, so a reader can see which question led to which and where the project actually got to.

001-kepler-exoplanets          2 run(s), best 0.90
002-mass-corrected-fit         1 run(s), awaiting audit

"awaiting audit" is the honest label for a green run nobody has judged yet.

Auditing with a coding agent¶

The auditing skill walks an agent through this: read the trace, check the gate details, look for the failure modes the gates cannot see (an unfair baseline, a leaked split, a metric that does not answer the question), then propose ratings with a rationale for you to confirm.

The agent cannot bypass anything. The gates are computed by the runtime from the recorded trace, and no rationale makes a fabricated run pass tool_use_verified.