Four tests that could not fail

Quilore is live at quilore.com. It has a large test suite, and this is a post about the parts of it that were worthless while looking exactly like the parts that were not.

A guard test here is a test that stops a specific mistake coming back. Not “does this function return the right number”, but “does this file still contain the thing we decided it must never contain again”. They are cheap, they encode a decision, and they are the easiest kind of test to write badly, because a badly written one passes.

The four

Each of these was written, read back, reviewed and run. Each passed. Not one of them could have failed.

The guard	What it was actually matching
No opacity in the mark	The comment saying opacity had been removed
No em dash in shipped source	Nothing at all
No DOM access in the pure date module	The comment saying the module touches no DOM
No email collected on the waiting page	The page’s own copy, which mentions the email Spotify sends

Three of those four are one mistake made three times: the guard asserted on prose rather than on behaviour. A test that reads a source file as text will happily match the comment that explains why the forbidden thing is absent. The comment is written in the same language as the thing it describes, and greps do not know the difference.

The second row is a different fault and a better story. The pattern used a word boundary, \b, and the file was written through a shell heredoc, which turned it into a literal backspace byte. The regex then required a control character before the word, which no source file contains. It looked correct in the diff. It matched nothing, ever.

How they were found

By breaking each one on purpose.

That is the only method that works here. A guard that cannot fail passes, reads correctly, and sits in a green suite looking like coverage. Nothing distinguishes it from a working guard except deliberately introducing the fault it exists to catch and confirming it goes red.

There is a trap inside the method, which we also walked into. One mutation appeared to prove a guard worked when it had not: the edit meant to move an element below the nav had put it before the nav instead, so nothing about the page had actually changed and the failure being observed had a different cause. Print the state you think you mutated. A mutation you did not verify is a second unverified thing checking the first one.

Reading source is not running it

At 409 tests, the suite was green against a file that could not parse.

A stray literal newline inside a string broke the main module. The page did not load at all, in any browser, for anyone. Every test in the suite read source as text, and not one of them handed it to a parser, so a syntax error that stopped the entire product was invisible to all 409 of them.

It was found by opening the page, which by then was the only thing that had tried to run the file.

Every shipped module is now compiled from a data URI inside the suite, so a syntax error throws where the tests can see it.

And running it is not looking at it

The share composer draws a card to a canvas so someone can post their listening history as an image. On the square shape, and only on the songs card, the fifth row crossed the footer rule and the closing line printed directly on top of the site address.

The PNG saved perfectly. It simply said two things in the same place.

Sixty-eight tests passed against that card, because the layout was arithmetic that no assertion touched. The row height took the larger of what the contents wanted and what the space allowed, which guaranteed every row got what it asked for, which meant five rows could ask for more card than the card had. Nothing clamped the total.

The vertical maths now lives in a pure function that takes the available height as an argument, and the tests assert that the rows fit inside it across twenty cases, in both shapes, with and without an artist line. None of them need a browser. The bug did.

What replaced the four

• Strip comments before asserting on source, so a guard cannot match the explanation of itself.
• Never build a regex through a shell heredoc. Write it in the editor, or dump the bytes and look at them.
• Assert the relationship rather than a literal. One guard checked that the browser theme colour equalled a hardcoded value, so when the palette changed the page went stale and the test kept passing. It now reads the colour out of the stylesheet and requires the two to agree.
• Mutate every new guard, and confirm the mutation landed before believing the failure.

What we are not claiming

None of this is an argument against guard tests. They caught real regressions in this project, including a dead stylesheet rule that had been quietly winning specificity fights against the component that replaced it.

It is an argument against reading a green suite as proof of anything except that the tests ran.

And the honest ending is that writing the rule down did not fix us. The heredoc mistake happened again later in the same project, in a guard protecting against a theme flash, after the rule existed and after it had already been recorded twice. It was caught the same way as the others, by trying to break it and noticing that it would not break.