optimizing¶
When this skill applies
Improving a result¶
The experiment works and the number is not good enough. There are two ways forward: make the science better, or make the measurement more flattering. Only one of them is worth doing, and the second is always easier.
First, understand why it fell short¶
Ask what is actually limiting the result:
- The method is too weak for the problem.
- The data is too small, too noisy, or missing what matters.
- The parameters are badly chosen.
- The implementation has a defect that is costing accuracy.
- The hypothesis is wrong — the effect is not there.
The last is a real answer and reaching it is progress, not failure. An experiment that establishes an effect does not exist has done its job.
Change one thing at a time¶
Every run is recorded, so the history only means something if changes are isolated:
--reuse re-executes only what changed, which makes single-variable iteration
fast enough to be worth doing properly. Changing five things and seeing an
improvement teaches you nothing about which of the five mattered.
Where to change things¶
Parameters live in workflow.json under params. Change them there rather
than in code: the trace then records exactly which configuration produced which
result, and the diff is one line.
Method changes belong in the agent. If the approach itself is wrong, say so plainly — swapping in a fundamentally different method is arguably a new hypothesis and may deserve a new experiment.
Design changes mean editing the DAG. If the experiment cannot answer the question as structured, restructure it and re-validate.
Record what you learn¶
Each attempt teaches something, and the point of writing it down is that the next person — including you next week — does not repeat it:
science-adk learn "Raising the exoplanet row cap above 5000 does not change the recovered Kepler exponent; the archive returns 3565 usable planets and the fit is already saturated." \
--experiment 001-kepler-exoplanets --run 20260820-004148
Evidence is required. An insight with no run behind it is an opinion.
What is not allowed¶
These are the ways an optimisation loop turns into a fraud, and they are tempting precisely because they work:
Tuning against the target. The threshold is invisible to agent code on purpose. Do not read it in, and do not choose parameters by how close they get to it rather than by whether they are right.
Switching to a metric that scores better. If RMSE is the honest measure, a sudden preference for r² needs a scientific reason, stated before the switch.
Dropping the data that hurts. Excluding outliers can be correct, but the rule must be principled and fixed in advance, not "these three points were inconvenient".
Loosening the test. Reducing the holdout, evaluating on training data, relaxing a tolerance — all of these improve the number by measuring something easier.
Weakening the simulation. Coarsening a grid until instability disappears does not make the instability go away; it makes it invisible.
If you find yourself reasoning about how to make the number look better rather than how to make the result more correct, stop. That instinct is the failure mode this whole system is built to catch.
Knowing when to stop¶
science.toml sets max_optimize_iterations because unbounded optimisation
converges on overfitting, not truth.
When you reach the limit, report honestly: the best result achieved, why it falls short, and what would be needed to do better. "We could not reach the target with this approach, and here is what constrains it" is a genuine scientific finding.
Then consider whether the next step is a different hypothesis rather than
another attempt at this one — load the evolving skill.