Engineering
Engineering.
4 deep dives and 8 lessons learned.
- 01Lessons learnedTrust boundaries don't live in foldersA ticket that just moved a folder, same files, same format, diff untouched, quietly moved tweet-derived text into a trusted agent's read path: a prompt-injection regression that shipped clean because it passed its own acceptance criteria. The fix: key trust on provenance the server controls, never on which folder a file sits in.
- 02Lessons learnedThe model that scored 7 and 3 on the same testThe same model scored 7 out of 9, then 3 out of 9, on the same injection-resistance eval. Neither number was real: the first came from a prompt shape production never serves, the second from a token cap that truncated answers before the validator could see them.
- 03Lessons learnedFalse green: when a passing test vouches for a dead featureA feature passed six-reviewer diff review, two rounds of execution and bug fixes, and full CI, then shipped broken. The only test covering it mocked a state its own state machine can never produce. Every offline gate proved the change was well-formed, not that it ran. Here's the anatomy of that false green, and the fix.
- 04Lessons learnedThe one-line model swap that broke every multi-turn runSwapping a Gemini model alias for a concrete id looked like housekeeping. It jumped a major model version. Gemini 3 requires echoing back an opaque thought_signature field on every replayed tool call, and no test caught the gap because it only surfaces on the second turn of a run.
- 05Lessons learnedThe security rule I reverted was right all alongI reverted a security rule because a probe against the live load balancer said it failed open. Cloud Armor takes about 100 seconds to propagate a new rule to the edge; my probe fired at 10 seconds and measured a rule that did not exist yet. Verifying the right layer isn't enough, you have to wait until it has decided.
- 06Deep diveOne click, 221 re-renders: React Context can't subscribe to a fieldI memoized the context value and wrapped the hot components in React.memo, standard advice, and one click still re-rendered a list 221 times. Context notifies every consumer when the value object's identity changes, regardless of which field a consumer reads, and no memo can stop that. The fix: a selector-based Zustand store.
- 07Lessons learnedPython and JavaScript count characters differentlyI asked Claude to bold every occurrence of a word in a markdown file, and every edit landed one character off, because of a single emoji earlier in the document. Python counts that emoji as one character; JavaScript counts it as two. An integer that crosses a language boundary has units, declare them, or each side quietly picks its own.
- 08Lessons learnedThe test database that liedA workspace delete that passed four code reviews and a green CI run still crashed production with a 500. The test suite ran on SQLite, which silently ignores a lock clause Postgres rejects outright, so the tests were honestly green about the wrong database.
- 09Lessons learnedRate-limited by tokens, not requestsA 27-second stall traced back to a token quota, not a request quota: Shannon's agent loop was re-sending its entire context on every turn. The fix is prompt caching with breakpoints placed where each layer of context stops changing, plus a canary token that almost broke it again.
- 10Deep diveTranslating an SDK loop into one streamA Shannon agent run looks like one continuous stream in the browser. Underneath it's a loop of separate Anthropic API calls with tool execution wedged between them. The fix is a server-side translation layer that hides every seam behind a small, fixed vocabulary of UI events.
- 11Deep diveTwo pipelines, one timeline: rendering agent activity across live and historical viewsEvery AI agent UI is secretly two UIs: one renders events live over SSE, the other rebuilds the same timeline from the database on reload. They have to match exactly, and five separate bugs, from parsing to null handling to ordering, proved they easily don't.
- 12Deep diveHibernating an agent loopWhat happens when a cloud AI agent needs to ask the user a question mid-task? You can't just block a coroutine. The fix: treat the agent loop as a function whose entire state is a message list, serialize it to Postgres, exit the process, and reconstruct the loop from scratch when the human answers.