Skip to content

Integrity gates and the audit

Scoring a run happens in two stages that must not be confused:

  1. Six deterministic gates, computed by the runtime from the recorded trace. They answer is this number real? A machine decides, and any failure scores the run 0.00.
  2. Three audit pillars, recorded by a human or an auditing agent with a written rationale. They answer does this number matter?

A run that passes every gate and has no audit still scores 0.00. That is not a bug. Passing the gates is a precondition for having a result, not the result itself.

The six gates

science-adk score

  [pass] execution_completed: All nodes ran to completion.
  [pass] all_nodes_ran: 4 node(s) produced output.
  [pass] result_measured: evaluate reported a measurement.
  [pass] metric_is_finite: evaluate reported 0.989997.
  [pass] produced_output: 6 output value(s) recorded.
  [pass] tool_use_verified: Declared tool use matches the record.

execution_completed

The trace's overall state is passed. If any node failed, the gate names it:

[FAIL] execution_completed: Run state is 'failed'; failed node(s): analyze.

all_nodes_ran

Every node declared in workflow.json reached state passed or cached. A node that was skipped, or that silently never executed, is reported by id. Cached nodes count, because their outputs came from a real earlier execution of byte-identical code.

result_measured

At least one node of kind evaluation finished and produced a value output. A workflow with no evaluation node fails immediately:

[FAIL] result_measured: The workflow has no evaluation node, so nothing was
       measured.

metric_is_finite

That value is numeric and finite. Non-numeric, NaN and ±inf all fail:

[FAIL] metric_is_finite: evaluate reported nan, which is not a finite
       measurement.

A NaN is a failed computation, not a poor score. Treating it as a low number is how a broken pipeline gets recorded as a negative result.

produced_output

The run recorded at least one output value across all nodes. A pipeline that completes while producing nothing has not done anything.

tool_use_verified

The provenance gate. For every node declaring tools that completed, the ledger must contain at least one successful tool call.

[FAIL] tool_use_verified: Node(s) declared tools but made no successful tool
       call: analyze. Either the data was not really fetched, or the
       declaration is wrong.

This is the gate that catches a fabricated result. Every other gate is satisfied by a hard-coded constant; this one is not.

What a failure means

Score: 0.00 — a failed gate means this run is not a result.

There is no partial credit, no override flag, and no way to audit past a failed gate. Fix the cause and run again — the failed run stays on disk as part of the record.

The three pillars

Once the gates pass, judgement is recorded explicitly and attributably:

science-adk audit \
  --scientific-value 0.85 \
  --method-fidelity 0.90 \
  --implementation-quality 0.95 \
  --rationale "GNN embedding reaches RMSE 0.72 on held-out ESOL against 0.95 for Morgan fingerprints on the same seeded split."
Pillar Asks
scientific_value Does this move the research question forward? Would anyone act on it?
method_fidelity Is the method sound, and are its assumptions stated and respected?
implementation_quality Is the code correct, reproducible and readable?

Each rating is in [0, 1] and is clamped into range. All three are required, and so is the rationale:

ValueError: An audit needs a rationale explaining the ratings.
ValueError: Missing rating for pillar(s): method_fidelity

The formula

value = 0.0 if not gates_passed or not pillars else round(
    sum(pillars[p] for p in SCORE_PILLARS) / 3, 4
)

An unweighted mean of the three pillars, gated to zero. Equal weights are a deliberate choice: a beautiful implementation of a meaningless comparison and a scientifically important result held together with tape are both worth correcting, not averaging away.

Meeting the target

Separately from the score, the run's observed value is compared against target_value from GOAL.md:

meets_target = observed_value >= target_value

That comparison happens outside the workflow. No node ever sees the threshold, which is what keeps an evaluation agent from being written — by a person or by a model — to just clear it.

Reading it back

science-adk score --json | jq '{value, gates_passed, audited, observed_value}'
science-adk report        # REPORT.md written from the trace and the score
science-adk campaign      # the leaderboard across every experiment

score.json in the run directory holds every gate with its detail string, plus the pillars, the rationale and the audit timestamp.

Reference