Why every AI-written commit needs a regression harness

An AI agent can produce in an afternoon what a developer might write in a week. That is the opportunity. The risk is that it can also break, in an afternoon, something that took a month to get right — and do it in a diff that looks perfectly reasonable.

Cheap code, expensive mistakes

Human developers carry implicit knowledge: "don't touch the tax rounding", "that endpoint is used by the mobile app". Agents don't, unless the knowledge is encoded somewhere they can't ignore. The most reliable place to encode it is a test that fails.

What our harness runs

• Unit and component tests on every change, fast enough to run inside the agent's own loop.
• Integration and contract tests against real databases and service contracts, so a renamed field can't quietly break a consumer.
• End-to-end journeys in a real browser for the flows that make money: sign-up, checkout, provisioning, billing.
• Security checks: dependency scanning, secret detection and static analysis.
• Architecture rules that fail the build when a layer reaches where it shouldn't.

Across two of our platforms that adds up to more than 5,600 automated tests. The number matters less than the habit: no change merges without them passing.

Make it fast or it won't be used

A harness that takes an hour gets skipped. We split suites so the agent gets feedback in minutes, run the slow journeys in CI, and treat a flaky test as a bug with the same priority as a production defect. A test nobody trusts is worse than no test, because it teaches everyone — human and agent — to ignore red.

Let the harness talk back to the agent

The biggest gain came when failures were fed directly back into the agent's next attempt: the failing assertion, the stack trace, the relevant snippet. Most regressions are fixed by the agent itself before a person ever sees them. Review time goes to design and risk, where people add the most.

Start small

If you have little coverage today, don't aim for completeness. Write end-to-end tests for your three most valuable journeys, add contract tests at your service boundaries, and require both to pass for every AI-assisted change. That alone catches the majority of the expensive mistakes.