Postmortem: detection skipped one account for six days in August

By Jerimiah Guillot, founder of CostPulse.

From August 9 to August 14, 2026, CostPulse did not run anomaly detection on one connected AWS account. Six daily runs skipped it, so its spend for August 8 through August 13 was never checked. Every alarm we had stayed green the whole time. The account was mine. This post is what happened, why nothing told us, and what we changed so the same thing cannot stay silent again.

Impact

• One account was affected: the founder's own AWS account, connected as an ordinary tenant. No other tenant was skipped.
• No alert was missed. That account has four spend records in its entire history, none of them between August 8 and August 13, so there was nothing to detect. That is luck, not design, and the rest of this post treats it that way.
• No data was lost. Billing ingestion has no entitlement check, so it kept running throughout. Only detection stopped.

What happened

In late July we shipped a 14-day trial policy: when a trial ends without a paid plan, detection and the weekly digest pause for that tenant, and the dashboard says so. When it shipped, existing tenants were migrated. The public demo tenant moved to a permanent plan. My own tenant was given a trial like any new signup, ending August 9 at 01:30 UTC. Nobody moved it to a paid plan before that date.

From then on the entitlement gate did exactly what it was written to do. The detector ran every hour, found the account, saw an expired trial, and skipped it. What it logged was one counter, skipped_unentitled: 1, with no tenant or account attached. That line looks the same as routine billing enforcement on a lapsed trial, which is what it was. The pipeline was healthy, so every pipeline alarm was right to stay quiet.

The dashboard did show a "trial has ended, alerts are paused" banner. It was the only signal, and it sat on a page nobody had a reason to open, because a quiet alert feed is what a healthy account with no spike looks like too.

How we found it

By accident. Early on August 15 (UTC) I was chasing a different gap: the public demo feed had been producing only critical alerts since late July. That turned out to be a stale 50% threshold saved on the demo tenant, which hid every warning-sized spike. While looking at how each tenant was being evaluated, the founder tenant turned out to have last been evaluated for August 7.

Both problems had the same shape. They were data, not code. Every alarm watched whether the pipeline ran. None watched whether a particular tenant was actually evaluated, which is the thing a customer pays for. An empty alert list looks identical whether nothing spiked or nothing was watching.

The fix

At 03:11 UTC on August 15 the tenant was moved from the trial to the Team plan. A manual detector run confirmed the gate no longer skipped anything (skipped_unentitled: 0), and the next scheduled run evaluated August 14. August 8 to 13 were never evaluated at the time. In September we checked whether a replay was worth doing: with no spend records in that window, a replay would compute nothing, so none was run.

What we changed

• A skip names who and why (Aug 15). When the gate skips an account it now logs a warning with the tenant, the account and the reason (trial expired, cancelled, unknown plan), instead of a bare counter.
• The demo feed watches itself (Aug 15). The demo generator resets the demo tenant's threshold to the default on every run, and an alarm fires if the demo feed misses a day of alerts. This closes the threshold gap found during the same investigation.
• Per-account detection health (Sep 6). Each connected account now records when it was last checked and when it was last actually evaluated. A skipped account is recorded as paused, never as freshly evaluated. The dashboard shows the result as a status: healthy, may have stopped, paused, or waiting for the first check.
• An alarm on the thing customers pay for (Sep 6). If an account that should be monitored goes 36 hours without an evaluation, an alarm fires, and the log line names the tenant. It is 36 hours rather than 24 because each account is evaluated once per local day, so a 24-hour bar would trip on daylight saving and normal drift.

What is still true

A trial that ends still pauses detection. That is the policy, and it is the right one, but it means an exact repeat of August would show up as a paused status and a named warning, not as an alarm: the 36-hour alarm covers accounts that should be monitored, and an expired trial is not one. The fix for that case is telling the person whose trial it is, clearly and before it ends. Those emails are written but stay switched off until the postal address they legally need is set up. Until then, the dashboard banner is where it shows.

What to check in your own alerting

• Does anything alert on silence? For each thing you monitor, ask what you would see if the check stopped running for that one thing while the rest kept working. If the answer is "nothing", the check can die quietly.
• Are skip counters attributable? A count of skipped items with no identity is unreadable in the one moment it matters.
• Do billing or entitlement states feed monitoring? Any gate that turns monitoring off for business reasons should say so, to someone, every time it does.