Can an agent find the leak?

A credit default model was built clean: 180 features, a temporal split, and every column that could only be known after the outcome removed before training. Then a single leaking column was planted back into some of the data, and an agent with eight read-only tools was asked to audit the model and say what was wrong with it.

The runs Run the audit yourself

The agent never sees the data. It sees a fixed set of artefacts through eight tools — a data dictionary it can search, a SHAP ranking, per-feature coverage and association figures, ablation results — and it has to reason from those to a verdict. Twelve real runs were recorded against live API calls. Every one of them is replayable here, turn by turn, including the tool arguments and the full results the agent got back.

What the agent flagged

Across the scorable runs the agent raised flags. The answer key credits of them. It is tempting to call the rest errors, and that would be wrong: the key recognises exactly one kind of problem, the columns removed for carrying post-application information. A flag it does not credit is a flag outside that list, which is not the same as a flag that is mistaken.

Flags the answer key credits

Flags the answer key does not credit

None of these claims leakage in the sense the key scores. They argue distribution shift, missingness that tracks loan vintage, drift across vintages, and whether the 2017 labels are final. Those arguments are ungraded: the scorer cannot settle them. Each card links to the turn in the replay where the agent produced it, so the evidence can be read rather than taken on trust.

The scorer's output, unchanged

Nothing above re-derives a verdict. Every figure is read from the scorer's own output at page load, and that output is reproduced here in full — precision, recall, the false negatives, the hard negatives, the derivative resolutions and the per-run canary block — so that nothing looks hidden by the framing.