The security rule I reverted was right all along

I reverted a security rule because my own verification told me it failed open: a request with no secret header sailed past the rule and reached my backend. The verification was a real probe against the live load balancer, and it was still wrong. Cloud Armor takes about 95 to 100 seconds to push a new rule from its management API to the edge that enforces it, and my probe fired about 10 seconds after the rule was created. It measured a rule that did not exist yet, and I read the result as a design flaw.

I already knew to verify against the layer that actually decides. This episode taught me the missing half: wait until that layer has actually decided.

The 401 that launched a research program

The rule was a lockdown at the load-balancer edge: deny, with a 403, any request that doesn’t carry a correct shared-secret header, so only traffic from my own front door reaches the backend. The backend is Shannon’s application API, and by default a Cloud Run service answers the public internet on its own, so the edge rule makes the load balancer the only way in and no request can skip the front door to reach the API without the shared secret. Cloud Armor rules are written in CEL, a small expression language for matching requests, and the secret check was has(h['x']) && h['x'].matches('^S$'): confirm the header exists, then check its value. (The real header name and secret value are redacted here; h['x'] and '^S$' are stand-ins, not a literal one-character secret.) The deploy script, cloud-armor.sh, created the rule at priority 700 and then verified it the obvious way: send a request without the secret and expect a 403 from the edge.

The probe got a 401 instead. A 401 comes from the backend’s own auth, which means the request passed the edge untouched. Two explanations fit that observation exactly: the rule is live and doesn’t match absent headers, meaning it fails open, or the rule isn’t live yet. A single probe cannot tell them apart, and the more alarming reading won.

I reverted the rule, reopened the root-cause ticket, wrote up the failure blaming the has() guard, and filed a research ticket to investigate alternative CEL constructs, with a possible architecture change (moving validation into the backend) on deck if none worked.

The wrong diagnosis then propagated. It went into the root-cause reopen, the earlier writeup, a memory file my coding agents load, the research ticket and its handoff, and a six-question research plan. Three rounds of plan review didn’t catch it, because a review argues about the consequences of a premise, not the premise itself. Nothing downstream of a wrong conclusion is positioned to notice that it’s wrong.

Cloud Armor says “created” about 95 seconds before it means “enforcing”

gcloud … rules create returns when the control plane accepts the rule: the management API has validated it and stored it in the policy. Enforcement happens somewhere else, on the data plane, the serving edge that actually handles your traffic, and the config has to travel from one to the other. I measured that trip by creating a throwaway deny rule and polling it every 10 seconds: 404 through t+92s, then 403 at t+102s, monotonic, and it held. Call it 95 to 100 seconds.

My script’s verify probe landed at about t+10s. create() issues the rules create, makes about five sequential gcloud describe round-trips (roughly 10 seconds), then probes. Every run landed about 90 seconds inside the propagation window, so each got the same clean 401. The failure was deterministic, which made it look like a stable property of the rule rather than a race the probe lost every single time.

The probe was aimed at the right system. It hit the live load balancer, the exact layer that decides whether a request is denied. It still measured nothing, because that layer hadn’t heard about the rule yet.

A control probe cracked the case, not a better hypothesis

The research ticket took the fail-open premise as given and went hunting for CEL constructs that wouldn’t. The first experiment made things look worse. Test rules at priorities 9000 and 9001 never fired at all: every probe returned the backend’s 404, with waits of 5 to 30 seconds. The cause had nothing to do with CEL either.

The policy contained a throttle rule at priority 2000 with the expression 'true'. Cloud Armor evaluates rules lowest-number-first, first match wins, so every rule numbered above 2000 was unreachable. Inspecting the policy revealed it; moving the tests to 1900 and 1901 fixed reachability and still produced all-404s.

The move that broke the case open was boring: add a control. I created a deny rule that should trivially fire, a plain header match, and probed it with the matching header. It failed too, 200 instead of 403. A failing control reframes the question from “is my hypothesis wrong?” to “is my apparatus working at all?”

I ruled out Cloud CDN (off on both backends) and load-balancer routing as confounders, which left one axis nobody had tested: time. Instead of probing a fresh rule once, I polled it every 10 seconds and watched the answer change: 404 through t+92s, 403 at t+102s, and stable after that. The rules were fine. Every probe in the entire investigation had simply been early.

Post-propagation, the reverted rule was right all along

With the propagation window known, the retests took minutes. I rebuilt the original has()-guard construct faithfully (the real header, the .matches() check, the allow-path carve-out) and probed it 180 seconds after creation. Then I probed the bare == variant, the one the has() guard was added to fix:

has() guard, probed at t+180s:  absent → 403   wrong → 403   correct → 404
bare ==,     probed at t+180s:  absent → 404   wrong → 403   correct → 404

(The correct-secret probes hit a throwaway path, so passing the edge means the backend’s 404.) The first line says the reverted design denies on an absent header, exactly as intended. The second line says the fear behind the root-cause ticket was real, just misattributed: bare equality against an absent header genuinely fails open, and the has() guard is what turns absence into a denial.

The original instinct was right twice over. The fix was needed, and it worked; only the verification of it was mistimed. The eventual fix cost less than the research program it replaced: re-apply the reverted rule and add a wait.

The shipped fix is a wait that fails closed

cloud-armor.sh now runs a two-stage gate, _wait_until_live, before its positive-control verify. Stage one polls a wrong-secret probe until it returns 403: a wrong secret must always be denied, so the first 403 proves the rule is enforcing at the edge. Stage two then requires absent → 403, which is the actual threat being closed.

Each way of not getting there fails the deploy with its own exit code: 4 if the rule never went live inside the cap, 3 if it went live but still passes absent-secret requests. A --self-test round-trip in make ci guards the script itself.

The cap earned its keep on day one. The spike measured about 100 seconds, and I set PROPAGATION_TIMEOUT=240, 2.4 times that, out of plain over-waiting caution. At the real deploy on 2026-05-20, on the live global rule at priority 700, propagation took about 140 seconds before absent → 403 came back green.

A tighter cap at 120 seconds would have timed out moments before the rule went live and false-failed the deploy, recreating the exact too-early-probe bug the gate exists to prevent. A number you measured once is a floor, not the truth.

Acceptance, runtime, propagated: three different yeses

I already had a discipline for this class of bug: verify at the binding layer, because a dry-run proves acceptance, not runtime. That rule was in force here and didn’t save me, because there is a third layer. The API accepting your change is one event; the change taking effect is another.

For any system that converges asynchronously — Cloud Armor, DNS, CDN invalidation, eventually consistent stores — there is a window between those two events. A probe inside the window is worse than no probe: it returns a confident answer about a world that doesn’t exist yet. Mine returned the same confident answer on every run, and that consistency is precisely what made it believable.

So the sharpened rule I wrote down: classify every probe as testing acceptance, runtime, or propagation, and for async-converging changes, poll until live before trusting any verdict. And when a small, bounded risk starts funding a multi-day investigation, re-run the original failing probe with controls before trusting its diagnosis. The 401 was never lying. I was asking it about the wrong moment.