Posts tagged Testing.

The same model scored 70%, then 30%, on the same injection-resistance eval. Neither number was real: the first came from a prompt shape production never serves, the second from a token cap that truncated answers before the validator could see them.

A feature passed six-reviewer diff review, two rounds of execution and bug fixes, and full CI, then shipped broken. The only test covering it mocked a state its own state machine can never produce. Every offline gate proved the change was well-formed, not that it ran. Here's the anatomy of that false green, and the fix.

A workspace delete that passed four code reviews and a green CI run still crashed production with a 500. The test suite ran on SQLite, which silently ignores a lock clause Postgres rejects outright, so the tests were honestly green about the wrong database.