← back to notes

When the measurement is the broken thing

Twenty-odd checks printed green over a heading half-covered by the navigation bar. None of the checks were wrong.

  • for practitioners
  • verification
  • measurement
  • research methods
  • 2026

What green actually meant

I keep a folder of small programs whose only job is to tell me when a web page is wrong. One afternoon all of them printed the verdict I wanted. Then I looked at the page and found a heading parked under the fixed navigation bar with half of it covered, exactly as a reader would have found it.

  • what did not happenNo check failed, errored, or timed out. Each one ran and each was correct about the question I gave it.
  • what green saysNone of the things that have gone wrong before went wrong again.
  • what it does not sayAnything about the questions nobody thought to write down.

Every check I own began as a surprise: something broke, I was annoyed, and I wrote a program so that particular thing would announce itself next time. The suite is a list of the ways I have already been caught out. Thin coverage prints green and thorough coverage prints green, and an unasked question produces no output at all. It sits in the silence between the checks, and silence is what success sounds like.

Twice in one month, the instrument was the broken thing

When a number startled me and I chased it to its cause, the cause was more often the instrument than the page. Two from the same month, and neither instrument malfunctioned.

The two instruments compared: what each reported, what was actually true, how the reading felt, and what the instrument was really pointing at.
the settle routine the screenshot script
what it did waited for the page to stop moving, then measured assembled the address of the page to photograph
what it reported 22 elements out of place a clean set of images
what was true all 22 were healthy; it read them mid-animation it photographed a 404, which is close to blank
how the reading felt alarming calm
what it cost an afternoon hunting a layout that was fine a false record, unnoticed; nobody chases a clean result
Accurate, and about the wrong thing Read down the last two rows: the reading that felt alarming cost an afternoon, and the one that felt calm did the real damage.

Two rules, both cheap

  1. A surprising number is probably a broken number. Not the only hypothesis, and an uncomfortable one, since it invites explaining away every result you dislike. Re-run it under changed conditions, confirm the tool is looking at what you think it is, and go and look at the thing yourself at least once.
  2. A check that has never failed has not been shown to work. Break the subject on purpose, watch the check go red for the right reason, put it back. A program that returns true unconditionally returns true on a working page too, and I have written that program without meaning to.

A dashboard fails the same way

Swap the probe for a metric and nothing about this changes. A number carries no marker of its own validity, and it cannot tell you what it failed to measure, because that is the part it has no access to.

  • a usability score climbsThe wording of the task changed between rounds, and got easier to parse.
  • a survey distribution comes back cleanThe question means one thing to the people who wrote it and something adjacent to the people answering it.
  • engagement collapses on a TuesdayA tag stopped firing, and someone builds a quarter of work on the cliff.

The discipline transfers without much translation.

  • know what it asksAnd say out loud what it cannot ask.
  • write the limits beside the resultThey are invisible in the output and forgotten inside a week.
  • prove it can failOn the failure you would most regret missing.
  • keep one route that skips the instrumentReading twelve raw responses is worse than a dashboard at everything except noticing the dashboard is wrong.

Still figuring out

  • Doubting the instrument first costs time on every defect that turns out to be a defect. Plenty of times the measurement was fine and I spent twenty minutes proving that the problem I was told about is the problem.
  • My heuristic (measurement error is often extreme, categorical, or suspiciously tidy, while a defect is usually specific and partial) is a guess dressed as a rule, and it has been wrong.
  • The fallback I keep recommending is to look with your own eyes, and eyes are an instrument too, with their own coverage gaps and a much worse memory. I have stared straight at a broken page and seen nothing wrong with it.