CloudWatch bills custom metrics per metric per month — and a "metric" is not the name you gave it. Every unique combination of metric name and dimension values is its own billed metric. RequestLatency with a service dimension across ten services is ten metrics. Add an endpoint dimension with forty values and it's four hundred. Add anything unbounded — a container id, a customer id, a version string that changes every deploy — and the count stops being a number you chose and becomes one your traffic generates.
Why this ships silently
The dangerous edit looks like an observability improvement: someone adds a dimension to get a per-something breakdown. In code review it's one added field. On the bill it's a multiplier on every metric that carries it, and CloudWatch will happily accept millions of distinct metrics without a warning. Structured-logging metric formats make it even easier — one log field promoted to a dimension in a metric filter or EMF payload, and every unique value that field ever takes mints a new billed metric. The per-metric price is small (check AWS's current pricing; there are volume tiers) — the count is what explodes.
One metric name, three dimension choices. The billed count is the product of every dimension's distinct values — and container ids aren't a list you chose.
The signature on the bill
Cardinality explosions land with a deploy, so the spend shape is a step: CloudWatch's metric line jumps the day the dimension shipped and stays at the new level. That makes it a clean catch for day-over-day anomaly detection — CloudWatch spend up 40% against its baseline with no traffic change is exactly this — but only if something is watching per-service spend daily. On a monthly invoice it just reads as "observability got more expensive," which nobody investigates.
What to check first
aws cloudwatch list-metricsper custom namespace, and count. The namespaces with the most metrics are rarely the ones anyone expects.- Look at the dimension keys on your highest-count namespaces: anything that holds an id, a hostname, a container/task id, or a version is unbounded by design.
- In Cost Explorer, group CloudWatch by usage type — metric-months and PutMetricData API calls meter separately, and both climb with cardinality.
- For per-id visibility, keep the id in logs and query it with Logs Insights; promote only bounded, low-cardinality fields to metric dimensions.
The general rule: dimensions are for values you could list from memory. The moment a dimension's values are generated rather than chosen, you've priced your bill in someone else's units.