experiment-design

When this skill applies

Designing an experiment

An experiment is a directed acyclic graph. Each node does one thing, declares what it consumes and produces, and is connected by typed edges. The engine validates the whole structure before a line of it executes, so mistakes here are cheap — which is exactly why it is worth being careful.

Start from the claim, not the code

Before opening an editor, answer three questions:

1. What would refute this? If nothing could, it is not a hypothesis.
2. What single number decides it? That is the target metric.
3. What is the honest comparison? A result with no baseline is unmoored. Compare against the obvious simple approach, not a straw man.

Then write the hypothesis down:

science-adk experiment new "GNN embeddings predict solubility better than Morgan fingerprints" \
  --rationale "Learned representations should capture solvation geometry that topological fingerprints discard."

The shape of a workflow

Most experiments have the same four-part spine, and it is a good default:

config → data → method → evaluation

• config — every constant in one place, so a sweep is one edit.
• data — acquire, generate or simulate. Nothing else.
• method — the thing under test. This is the node that matters.
• evaluation — measure the outcome. Must not be the node that produced it.

Add nodes when a step is separately meaningful, has a reusable output, or might fail on its own. Do not add nodes for their own sake: five honest nodes beat fifteen decorative ones.

Node kinds

kind	for	base class
config	parameters and constants	ConfigAgent
data	acquiring, generating, transforming data	DataAgent
algorithm	the method under test	AlgorithmAgent
training	fitting a model	TrainingAgent
evaluation	measuring the outcome	EvaluationAgent
visualization	figures	VisualizationAgent
composite	a nested workflow	CompositeAgent

The kind is a promise about what the node does, and the audit holds you to it.

Ports

Every input and output is named and typed. Available types:

int float bool str json list dataframe array file figure model any

Edges connect one output to one input, and the types must be compatible — identical, or involving any, or int widening to float. Reach for any only when you genuinely mean it; the point of the types is to catch a mismatch now rather than during a long run.

{
  "id": "fit",
  "name": "Fit the model",
  "kind": "training",
  "purpose": "Fit a gradient-boosted regressor on the training split.",
  "ports": {
    "inputs": [{ "name": "dataset", "type": "dataframe" }],
    "outputs": [
      { "name": "model", "type": "model" },
      { "name": "holdout", "type": "dataframe" }
    ]
  },
  "params": { "n_estimators": 400, "learning_rate": 0.05 },
  "tools": []
}

Two rules about the rest of the node:

params is the tuning surface. Anything you might sweep or that the optimizer may change belongs here, not as a literal in the code.

tools is a claim that gets checked. List the tools the node will actually call. The runtime records every call, and a node that declares a tool without calling it fails the tool_use_verified gate. Leave it empty if the node is pure computation.

Edges

{ "source": "fit", "source_port": "holdout", "target": "evaluate", "target_port": "data" }

An input port takes exactly one edge. If a node needs two things, give it two input ports.

What the validator will insist on

Run it early and often — it is instantaneous:

science-adk validate

It rejects: cycles, edges to nodes or ports that do not exist, type mismatches, two edges into one input, and required inputs left unconnected. It warns about orphaned nodes and about a workflow with no evaluation node.

Getting the structure right

One responsibility per node. If describing a node needs the word "and", it is probably two nodes.

Keep the measurement separate. The most common structural mistake is a node that computes a result and scores it in the same breath. Split them.

Let data flow, not control. No flags that switch a node between unrelated behaviours. If two things can happen, they are two nodes.

Prefer a wide graph. Independent branches are clearer and cheaper to re-run than a long chain where everything depends on everything.

Nest when a step gets big. A composite node points at another workflow.json and runs as a sub-DAG. Use it when a step deserves its own internal structure, and because a validated sub-workflow can be reused whole in another experiment.

Then

Write the agent code — load the agent-authoring skill. Every node in workflow.json needs exactly one file at agents/<node-id>.py, and its declared ports must match the DAG exactly.