Python and JavaScript count characters differently

In May I asked Sonnet 4.6 to bold every occurrence of “music” in a markdown test file. It made all ten edits, and every one rendered wrong, shifted left by exactly one character: between**music**c and mathematics where between **music** and mathematics should have been. The cause was a single 🎵 emoji in a heading near the top of the file.

Python, where Shannon’s backend computes text offsets, counts that emoji as one character. JavaScript, where the frontend renders that character, counts it as two. Every offset the two sides exchanged past that emoji disagreed by one, and nothing in the stack could tell. The whole failure reduces to one invariant: an integer that crosses a language boundary has units, and if you never declare them, each side quietly picks its own.

The bug looked like a model problem

The corrupted span was five characters wide, shifted one to the left. That is exactly what you would see if the model had emitted search_text=" musi", a leading space plus the first four letters of “music”, and the find-and-replace had done its job faithfully on bad input. Sonnet 4.6 had recently shipped behavior changes around tool inputs, and I already had an eval watching what the model emits for related tool fields, so the hypothesis felt solid.

The plan I drafted was honest about the risk. The first step was a hard decision gate: query tracked_changes.original_text for the buggy run, and only proceed with a model-side fix if the stored text really was " musi". That field is set directly from the model’s search_text, so one SQL query would confirm or falsify the hypothesis in 30 seconds.

I skipped the gate. Four commits landed the model-side fix: a validator factory, an eval scenario, a diagnostic warning, and a tightened tool description. Two rounds of code review signed off, the second finding nothing to change and clearing it to merge. When I finally ran the diagnostic query that afternoon, it returned original_text == "music".

Clean, no padding. The model had been innocent the whole time, and I had shipped a four-commit fix for a bug that did not exist.

Code review never had a chance to catch this. Review checks that the code matches the plan; it cannot check that the plan matches the bug.

Removing one emoji cleared the bug

With the hypothesis dead, I went back to a clean repro. A fresh test file with plain prose about urban gardening, no emoji anywhere, did not reproduce the bug at all. Adding one heading, # 🎵 The Hidden Mathematics, made it reproduce every time. Deleting the emoji cleared it again.

The original repro file, slow_test_02.md, had contained an emoji in a heading from the start; that one character was the only difference between “bug” and “no bug”, and nothing about the symptom pointed at file content rather than tool input.

A bug that toggles on and off with one character in the document is not about what the model sends. It is about how the two sides of the stack count characters.

Python counts code points, JavaScript counts code units

len("🎵")  # => 1

"🎵".length; // => 2

Both answers are correct. A code point is Unicode’s number for one character, and Python strings index, slice, and measure by code points. JavaScript stores strings as UTF-16, where a character takes one or two 16-bit code units. Characters outside the Basic Multilingual Plane (emoji, some mathematical symbols, certain CJK ranges) take two, a pair called a surrogate pair, and JavaScript indexes by those units.

Shannon’s backend computes offsets the Python way (app/services/agent/utils.py):

idx = content.find(search_text, cursor)         # str.find returns a code-point index
results.append((idx, idx + len(search_text)))   # len counts code points

The frontend consumes them the JavaScript way (web/src/utils/changePreview.ts):

return original.slice(cursor, change.startOffset); // slice indexes UTF-16 code units

The transform that maps backend rows to frontend types copies the integers verbatim. No conversion utility existed anywhere in the codebase, because until this bug nobody knew one was needed.

Walk the failure end to end. The backend finds “music” at code-point offset 10 in that string and stores start_offset=10, end_offset=15. In the JavaScript copy of the same string, the 🎵 occupies two code units, so “music” actually starts at 11. The frontend runs slice(10, 15), grabs " musi", and the replacement lands one character early: between**music**c. The shift compounds, one extra unit of disagreement for every non-BMP character before the match, so two emojis shift every later span by two.

Both sides were internally consistent, which is what made this invisible. The backend’s own emoji unit tests passed, because str.find hands code-point offsets to code-point slicing and never leaves Python. The frontend had zero non-BMP test coverage. No test crossed the boundary where the two counting systems meet, so CI was green on both sides of a broken contract.

Convert at the boundary, and write the units down

The fix does not touch the backend’s arithmetic. Python is the source of truth for offsets and is consistent with itself, so it stays code-point native. The frontend converts at the last moment, before any slice call and before any Monaco position construction (Monaco, the editor component that renders the document, expects UTF-16 offsets in getPositionAt too). The whole utility is a loop (web/src/utils/textOffset.ts):

export function codePointToCodeUnitOffset(text: string, codePointOffset: number): number {
  let codeUnits = 0;
  let codePoints = 0;
  for (const char of text) {
    // for...of iterates by code points
    if (codePoints >= codePointOffset) break;
    codeUnits += char.length; // 1 for BMP, 2 for surrogate pair
    codePoints += 1;
  }
  return codeUnits;
}

The other half of the fix targets the upstream cause. start_offset and end_offset were declared on the data model (app/models/tables/change.py) as bare int fields with no docstring. The Python author picked code points because that is Python’s natural default; the TypeScript author picked code units because that is JavaScript’s. Both defaults are reasonable, which is exactly why the docstring now has to say it out loud: these are Python code-point units, and frontend consumers must convert.

The four mistargeted commits stay, since the validator, eval, warning, and tightened description are defensive code worth having; reverting them would be churn.

An int that crosses a language boundary has units

The type system cannot save you here. int typechecks in Python, number typechecks in TypeScript, and the units live only in each author’s head until someone writes them down. So the invariant I took away: any field whose meaning depends on the consumer’s runtime (string offsets, timestamps, durations, anything encoded) gets its semantics documented at the data model, the source of truth, not in API docs two layers downstream. The model is the contract.

The test discipline follows from it. Same-stack tests are insufficient for cross-stack claims by design; my backend emoji tests passed while the product was corrupting every emoji-adjacent edit. When data crosses an encoding boundary, at least one test has to run the full round trip with inputs that force the two sides to disagree: non-BMP characters for string offsets, DST transitions for timestamps.

And the cheapest lesson cost the most. The database row was the smoking gun from day one, and one SQL query would have saved four commits, two review rounds, and an afternoon of chasing the wrong layer. When a bug shape fits two hypotheses and a 30-second test separates them, run the test before you write the fix.