I built a system that lets my AI models improve themselves. It works like this: the model reviews its own performance, proposes a lesson — a permanent instruction it will carry into future work — and the lesson gets applied. Then, crucially, the model is re-tested against the same benchmark it was scored on before, and the two scores are compared.
If the lesson helped, it stays. If it didn’t, it’s automatically reverted.
Seven lessons have gone through that loop so far. Six made the model measurably worse. One was inconclusive. None helped.
All seven were reverted. Not one survived contact with a measurement.
The one that should frighten you
Lesson six was about formatting. Something to the effect of always structure your reports consistently. Utterly reasonable. The kind of instruction any of us would write without a second thought, and would never think to test, because what’s the risk in asking for tidier output?
Here’s what it did.
The model’s numeracy score went from 10 out of 10 to 0 out of 10.
Not degraded. Not “slightly less reliable at maths”. Perfect to zero, from an instruction that had nothing whatsoever to do with arithmetic.
Lesson seven was the same shape. An instruction to count the lines in your answer and verify each one before submitting — careful, self-checking, exactly the sort of discipline you’d want. It cut the model’s constraint-following score from 6 to 1.5. The instruction telling it to follow instructions more carefully made it follow instructions less carefully.
Why this happens
The models here are small — around a billion parameters, the size you run when you need something fast and cheap on modest hardware. Small models have limited capacity for holding instructions, and that turns out to matter enormously.
Every instruction you add competes for that capacity. Tell a small model to obsess over formatting, and the attention it was spending on getting the number right goes to counting lines instead. It isn’t ignoring your instruction. It’s obeying it, completely, at the expense of everything else it was doing well.
There’s a second mechanism, subtler and worse. Several of these lessons mandated fixed text — always open with this heading, always end with this summary. But a chunk of the benchmark checks whether the model can produce an exact requested output. A model dutifully prepending its mandatory heading to a response that was supposed to be the single word “yes” now fails a test it used to pass. The lesson didn’t make the model stupider. It made it unable to comply, by forcing it to add text where none was wanted.
That failure is invisible unless you re-run the same test. The output looks better. It’s neater, more structured, more professional. It’s also wrong more often.
The part that generalises
You do not have a self-improving AI system. But if you use AI at work, you are doing this exact thing, constantly:
- Adding a line to the prompt because it got something wrong last Tuesday
- Telling it to “always include a disclaimer”
- Asking it to be more thorough, or more concise, or more careful
- Bolting on a rule after a customer complaint
Every one of those is a lesson. Every one has the same potential to fix the thing in front of you while quietly breaking something behind you. And unless you have a fixed set of examples you re-run after each change, you will never find out. You’ll see the thing you were looking at get better, conclude the change worked, and move on.
Six out of seven, in my case. If I’d been going on impressions, I’d have kept all six.
What to do instead
Keep twenty examples. Real ones, from your actual work, with the answers you’d want. This is the whole trick, and it costs an afternoon. Twenty is enough to catch a collapse like 10-to-0.
Re-run all twenty after every prompt change. Not the one you were trying to fix — all of them. The damage is never in the example you’re staring at.
Judge on the full set, not the target. A change that improves your target case and breaks two others is a bad change, even though it did exactly what you asked. My system treats a regression anywhere as outranking an improvement in the target, which is why all six were reverted automatically rather than argued about.
Be willing to throw the change away. This is harder than it sounds, because you wrote it and it sounded sensible. The measurement doesn’t care.
Distrust the sensible-sounding ones most. Nobody tests “always start with a summary”. That’s precisely why it’s dangerous — the instructions that get scrutinised aren’t the ones that cause quiet damage.
The real lesson
I built the improvement loop expecting to watch models get better. What it actually produced was a 0-for-7 record and one instruction that annihilated a model’s ability to do arithmetic by asking it to tidy its formatting.
That’s not a failure of the system. That is the system working — the failures were always happening, in every prompt tweak anyone has ever made. The only difference is that this time something was measuring, so they got caught and reverted instead of shipping.
If there’s one thing worth taking from this: “it seems better” is not a result. It’s a feeling, and on this evidence it’s wrong about six times out of seven.
Real results from my own lab, July 2026, on 1B-parameter models. Larger models are more robust to this — but “more robust” is not “immune”, and the only way to know where yours sits is to have the twenty examples. The measurement approach behind this is described in the Five Pillars whitepaper.