Skip to content

auditing

When this skill applies

Auditing a run

You are about to grade work you produced. That is an uncomfortable position and you should treat it as one: the natural failure mode is to approve, because approval ends the task and disapproval means more work.

The engine removes the easiest cheats — a run that failed a gate cannot be audited at all — but the interesting failures are the ones only judgement catches. A run can pass every mechanical check and still be worthless.

Look at the data first

Do not audit from the summary. Read what was actually produced:

science-adk score --json      # gates, and the observed value
cat research/<exp>/runs/<run>/trace.json
ls research/<exp>/runs/<run>/data/

Open the data files. Look at the numbers. Look at the figures. An audit written without doing this is a guess with a confidence interval attached.

What to actually check

Is the result too good? r² of exactly 1.0, perfect accuracy, an error of precisely zero — these are almost always leakage, a trivially separable problem, or a metric computed against itself. Real measurements are messy. Treat suspicious perfection as a defect until you have explained it.

Did the computation really happen? Check the provenance section of the trace. If a node declared a tool, did it call it? If data was described as measured, was anything fetched? The ledger is written by the runtime, so it does not lie even when the code does.

Is the number the right number? A metric can be computed correctly and still be the wrong metric. Accuracy on an imbalanced dataset, r² on a fit with seven points, RMSE without units — all technically valid, all potentially meaningless.

Was anything evaluated on its own training data? The single most common way a computational result becomes false.

Do the magnitudes make physical sense? Compare against known values, limits and orders of magnitude. A binding energy in the wrong units, a rate constant that implies a reaction faster than diffusion, a length that exceeds the system — arithmetic can be flawless and the answer still absurd.

Would it survive a change of seed, split or resolution? If you do not know, that is worth saying.

Is there a plausible confound? The result may be real and still not mean what the hypothesis claims.

Recording the verdict

Three ratings in [0, 1], and a rationale that is required because a number without reasoning is not an audit:

scientific_value — does this teach anything? A correct confirmation of textbook theory is real work and modest value. A surprising, well-supported negative result is high value. Something that could not have come out any other way is low value however clean it looks.

method_fidelity — is the method a faithful implementation of what was claimed? Approximations are fine when stated. Silently substituting something easier is not.

implementation_quality — is the code correct, reproducible, seeded, and honest about its inputs?

science-adk audit \
  --scientific-value 0.7 \
  --method-fidelity 0.95 \
  --implementation-quality 0.9 \
  --rationale "Nonlinear ODE integrated by RK4; period measured from zero
crossings rather than from the analytic formula, so the scaling test is
independent of the generating model. Value is modest because the result
confirms established theory rather than extending it."

Calibrate honestly

If everything you audit scores above 0.9, your scores carry no information and you have quietly become a rubber stamp.

  • 0.9+ — rigorous, reproducible, genuinely informative.
  • 0.7 — sound work, limited scope or modest novelty.
  • 0.5 — it ran and measured something, but the design limits what can be concluded.
  • 0.3 — technically completed, scientifically weak.
  • below 0.3 — the result should not be relied on.

A low score is not a failure of the audit. It is the audit working.

Then write it up

science-adk report      # REPORT.md, generated from the trace
science-adk campaign    # update the project leaderboard

The report is built from trace.json, not from your summary of it. If a number appears there, it came out of the run — which is what makes the report worth reading, and worth trusting.