Green is not evidence

Someone on Hacker News asked a one-line question about the security checks I run on every PR: how often do those checks actually hold your PRs back?

I didn't know. That bothered me more than it should have, so I counted every PR check I have ever run. 2,864 of them, across nine repos.

The answer to his question turned out to be boring. What I found on the way to it did not.

The boring answer

286 of the 2,864 runs failed. About 10%.

That number is useless on its own, and it took me embarrassingly long to see why. A 10% failure rate sounds like a tax, one PR in ten held up at the door. It isn't, because the failures are not spread out. They arrive in tight three or four day clusters with months of silence between them, and every cluster sits on top of a migration. Late April, when I pulled these checks into a shared reusable workflow: eighteen cost-gate failures over four days while I shook out credential and fetch-depth bugs, then ten weeks of nothing. A second product adopting that workflow in May, with a commit two days later called fix(ci): resolve remaining PR check failures. Skeleton repos I bootstrapped in July that lit up the same way and then went quiet.

The repos with the worst failure rates in the whole org are the ones that have only ever been in setup, with no steady state to average against. The worst sits at 57%, and it is the template that the others get cloned from.

None of that is in the numbers. I only know it because I remember those weeks. Read cold, the dates say the opposite: that my oldest and busiest repo carries the most failures and is therefore the most burdened. It carries the most failures because it is where I did the extraction. The context that told the true story from its inverse lived in my head and nowhere else in the data. That is the benign version of a problem I was about to hit in a much worse form, one where the missing context was not in my head, or anywhere I could reach.

The number that didn't fit

I had the answer to the question and could have stopped. What stopped me was a small number that didn't fit.

I have a Vale rule in my content pipeline called EmDashes. It is twelve lines long. It is set to level: error, which fails the build. It exists because I don't write with em dashes, I never have, and I would rather they never reach the site. My drafts run through Claude, which likes them, so the rule is there to strip the ones I miss. That is stated plainly on the colophon; the rule is the enforcement.

The post that started all of this shipped with fifteen em dashes in it.

Vale ran on that post. Vale passed. On that same file, in that same run, Vale reported fifteen passive-voice alerts and ten weak-word alerts, so it was finding the file, parsing the MDX, and applying my custom styles. It reported zero em dashes in a document containing fifteen of them.

Here is why. Vale's existence check compiles a tokens list into a regex with word boundaries wrapped around it:

(?i)(?m)\b(?:—|–)\b

\b asserts a boundary between a word character and a non-word character. An em dash is punctuation. There is no word character on either side of the match to anchor the boundary against, so \b can never be satisfied, so the pattern matches nothing. Ever. Vale has a nonword: true option that drops the boundaries, and the rule works correctly the moment you add it.

That rule had sat in my repo, at error level, catching absolutely nothing, since the day I wrote it. And there is no observable difference between that and a rule that works. Both emit silence. Both go green.

It was worse than one bug

I fixed the regex, shipped it, felt good about myself, and then went looking for proof that the fix actually gated anything. It didn't.

Here is the CI step that ran Vale:

vale --no-exit --output=line web/content/notes/ | tee vale-report.txt
if grep -E ' error ' vale-report.txt; then
  exit 1
fi

--no-exit forces Vale to exit 0 regardless of what it finds. That was deliberate: print the report first, decide separately. The deciding is that grep. And --output=line emits path:line:col:check:message. There is no severity field anywhere in that format. The grep could never match. The two halves cancel each other perfectly: Vale is forbidden from failing the step, and the thing meant to fail the step is searching for a word the format never prints. That job had passed every build it ever ran. It could not have done anything else.

Then, while testing the fix, Vale flagged an em dash inside a fenced code block, which markdown scoping is supposed to skip. It turns out Vale has no built-in knowledge of the .mdx extension: without an explicit [formats] mapping it treats the file as plain text, applies no markdown scoping at all, and lints your code blocks and inline code as if they were prose. My config carried a comment confidently asserting that "Vale's markdown scoping handles fenced code blocks and inline code natively." It does not, because I never told Vale what it was reading. Six of the alerts on my notes tree were phantoms from that.

So: a rule that could not match, inside a runner that could not fail, parsing a format it did not know how to read. Three independent bugs stacked on one rule, every one of them silent, the whole assembly green. I only found the second and third because I went looking for evidence that my fix to the first one worked.

Once I saw it, it was everywhere

Then I looked at the rest, and found the same disease in three more checks.

Semgrep. My own security baseline mandates it. My own post names it. It is not wired into any workflow, in any repo. A Semgrep that was never installed reports zero findings. So does a Semgrep that is running clean.

The cost gate. Failing for months in two repos, red on every Terraform PR, for the excellent reason that I never provisioned the GitHub App token it needs to read my private modules. It was not gating anything. It was just red, in the corner, next to the checks that were working.

A config file that does not exist. My post says known-false-positive suppressions for the IaC scanner go in a .checkov.yaml. Zero of my nine repos contain that file. I have never written one. I documented a system I had imagined.

I audited the tools with tools that lied

It gets worse, because the tools I used to audit the tools had the same disease.

My first pass used gh run list --limit 100. It told me my secret scanner had fired twice. It has fired thirty-five times. The window silently truncated at a hundred runs, and my busiest repo has over five thousand. My second pass filtered to pull-request events, which silently discarded every push run. A sampled window told me one mature repo had a spotless record; its real number is 27 failures out of 313. Every one of those queries returned a confident, well-formatted, wrong answer. Not one of them announced what it could not see. I was auditing my instruments with instruments that shared their failure mode.

The thing I actually learned

A check that isn't running looks exactly like a check that's passing.

That is the whole post. Green means "nothing was reported." It does not mean "nothing is wrong," and it does not even mean "something looked." The absence of an alert is perfectly compatible with the absence of a scanner, and from outside the build you cannot tell those two apart.

It is the same shape as the 10% number, pushed to its limit. There, the surface reading inverted the truth, and only what I happened to remember could set it right. Here there is nothing to remember. A green check withholds the one fact you actually need, whether a check even ran, and no amount of context brings it back.

Make it fail on purpose

Make it fail on purpose. That is the entire technique.

The only reason I know my em dash rule works now is that I fed Vale a line containing an em dash and watched it report the em dash, then removed the fix and watched it report nothing. Ninety seconds, and the before-and-after is the proof. The only reason I know my one grandfathered exemption is scoped to a single file, rather than having quietly switched the rule off everywhere, is that I temporarily broke a different post and watched the gate catch it.

Every gate you own should have a known-bad fixture: something you can point it at that must fail. If you have never watched a check fail, you do not have a check. You have a decoration that emits green.

This generalizes well past linters. Any alarm you have never heard ring is in one of two states, and you do not know which one.

The punchline

I wrote most of this up as an update to the original post. In it, I said the security stack had caught two real things in four months: a CVE in a transitive dependency, and an IAM policy attached to a user instead of a role. Thin, I said, but the premium is near zero and the downside ends the product.

The pull request carrying that sentence went red.

Trivy caught CVE-2026-54466, rated CRITICAL, in websocket-driver 0.7.4, reached through @firebase/database. It had nothing to do with anything I had changed. It was disclosed against code I merged weeks ago, which is precisely the case that a PR-time scan exists to catch. Six minutes to bump it and go green.

So it is three, not two, and the third arrived while I was typing the sentence that said there were two.

I would take this trade again tomorrow. Near-zero premium against a downside that ends the product is not a hard call. But "we run X" is a claim, not a fact, until you have watched X fail.

What's next

The preview-environment pattern that ZAP runs against is already written up: Preview environments are the dev loop. Vercel plus a fresh Neon Postgres branch per PR, a real running app talking to a real database on every review, for almost nothing. This post was written before that one and sat in a drawer for a week. The next one is on making the invisible checks announce themselves, which is the other half of this problem.

• #building-lionshead
• #ci-cd
• #security