Make it fail on purpose

A feature makes money. A check protects it. Those are not the same job, and the difference is why checks rot.

When you ship a feature, the market tells you fast whether it works. Someone signs up or they don't. A check has no such feedback loop. A check that has quietly stopped working looks exactly like a check that is passing, and nothing outside the build can tell you which one you have.

At corporate scale a QA team is paid to close that gap. Their whole job is the negative case: break the thing on purpose and confirm the alarm rings. At solo scale there is no such team, and the negative case is the first thing to fall off the list, because it is the work that sits furthest from revenue. Features feel like money. Checks feel like insurance. The trap is believing features are money. Features are money-making. Checks are money-protecting. Skip the second one and you don't notice for months, which is exactly the failure mode I wrote about last time.

The count I had been avoiding

In Green is not evidence I found four checks that had never done anything. An em-dash linter whose regex could not match. A security scanner that was never installed. A cost gate that was only ever red. A config file I had documented but never written. The post ended on a rule I was very pleased with: make it fail on purpose. Every gate should have a known-bad fixture, something you can point it at that must fail, because if you have never watched a check fail you do not have a check, you have a decoration that emits green.

Then I did the obvious thing I had not done. I counted how many of my own gates actually had one.

The answer was humbling. Across my repos I run several dozen distinct gates. The reusable workflow every repo calls runs nine third-party scanners: Trivy twice, Gitleaks, Checkov, Hadolint, Actionlint, Shellcheck, ZAP, and an Infracost cost gate. Not one of them has a fixture. My content pipeline runs four gates on every note: prose style, spelling, frontmatter schema, link integrity. Not one of them has a fixture. The em-dash rule, the one this whole series keeps circling back to, had none.

What did have fixtures? Only the checks I had written myself. The little scripts that scan for hardcoded hex colors, or unpinned GitHub Actions, or drift in my shared config files: every one of those has a negative test that feeds it bad input and asserts it goes red. I have a unit-testing habit, and it reaches exactly as far as the code I author. Everything I bought, and every rule I configured inside someone else's tool, I took on faith. The fixture was present wherever I already had the habit, and absent everywhere I was trusting a vendor or my own YAML.

The two places I had gone further

Twice, I had wired the negative case into CI itself, so it runs forever instead of once. One is a schema gate for my product metadata: CI points the validator at a deliberately invalid file on every run and fails the build unless the validator rejects it. The other is my design-token scanner, which re-runs its own "this hex literal must be caught" test inside every repo that calls it. Two, out of several dozen. Both, again, on code I wrote.

Has any of this ever caught a regression?

Here is the part I have to be honest about, because the whole point of the last post was honesty about what a practice actually catches. Has one of these fixtures ever caught a real regression? A gate that silently broke, and a fixture that screamed?

No. Not once.

Every real catch I can point to was something else. The em-dash breakage from last post I found by hand, poking at it, not because a fixture rang. The one genuine vulnerability my pipeline caught recently, a critical CVE in a transitive dependency that showed up the week I was writing the last post, was a live scanner doing its ordinary job, not a canary. And the closest thing I have to a fixture story runs the wrong way: my hex-color scanner once misfired, flagged a pull-request number as if it were a color, and blocked a real merge. I fixed it by adding the negative tests I should have had first. The fixture did not prevent the bug. The missing fixture allowed it, and I added the fixture afterward, as an apology.

So this is not a victory lap. I am writing down a rule I have mostly not been following.

So I built one

Then I built the first one properly, on the gate that most deserved it.

There is now a file in my repo whose only job is to be wrong. It contains a single em dash and a line telling you not to remove it. It never renders and it is not a post. And there is a step in CI that points Vale at that one file and inverts the result: if Vale reports the em dash, the step passes; if Vale comes back clean, the step fails the build and tells me the gate has gone inert. It is about fifteen lines. The em-dash rule had been silently dead once already, for the entire life of the repo, and nothing would have told me if it happened again. Now something will.

The first time I ran it, I broke the rule on purpose and watched the build go red. That is the only moment in this whole exercise that constitutes actual evidence the check works. Everything before it was faith.

A fixture is a second thing that can rot

I want to be clear-eyed about the obvious objection. A known-bad file that quietly stops being known-bad is just more green, one layer down. Adding a fixture and never watching it fail leaves you exactly where you started.

So the discipline is not "add fixtures." The discipline is: break the thing once, on purpose, watch the alarm ring with your own eyes, and then wire that watching into CI so it repeats without you. The fixture is not the point. The watching is the point. The fixture is just how you make the watching automatic, and cheap enough that you keep doing it after the day you cared.

The rule

So here is the rule, and it is going into my standards where the checklist that opens every story will ask about it. No gate ships without a fixture that proves it can fail. And the smaller, harder half, the one I keep having to relearn: a fixture you have never watched fail is just another green light.

If you run any checks at all, you almost certainly have one that has never rung. Pick it. Break the thing it is meant to catch. Watch. It takes ninety seconds, and it is the only proof you are ever going to get.

• #building-lionshead
• #ci-cd
• #testing