A SageMaker real-time endpoint is an instance — or several, behind an autoscaling policy with a minimum of at least one — that runs 24/7 from the moment you deploy a model until the moment someone deletes the endpoint. It bills by the instance-hour whether it served a million predictions or zero. That makes it one of the purest forms of idle spend in AWS: the meter measures existence, not inference.
How idle endpoints accumulate
- The demo that worked. A model gets deployed for a proof of concept, the demo succeeds, the project moves on — and the endpoint keeps its GPU instance warm for a stakeholder meeting that already happened.
- Per-experiment endpoints. Iterating on a model by deploying each candidate to its own endpoint leaves a trail of them, each holding an instance. Nothing cleans them up when the winning candidate ships.
- Autoscaling that can't reach zero. Real-time endpoints scale between a minimum and maximum instance count, and the minimum is at least one. Overnight and weekends, that minimum is pure idle — and if the instance type was sized for peak, it's an expensive one.
- Notebook-adjacent leftovers. Deploys made from a notebook during development rarely go through the infrastructure review that would ask "who turns this off?"
One month of a proof-of-concept real-time endpoint: billed instance-hours vs hours that served any prediction at all.
The alternatives exist for exactly this
Serverless inference bills per request and scales to zero — built for intermittent, latency-tolerant traffic. Async inference queues requests and can scale its instance count to zero between bursts. Batch transform spins up, processes a dataset, and terminates. A real-time endpoint is the right tool for sustained, latency-sensitive traffic — which is precisely the profile most experiments don't have. Matching the hosting mode to the traffic shape is usually worth more than any instance-type tuning.
What to check first
aws sagemaker list-endpointsin every region anyone has experimented in, then map each to an owner. Endpoints nobody claims are the finding.- For each survivor, graph CloudWatch
Invocationsagainst a week. An endpoint with single-digit invocations per day on a per-hour instance bill is a candidate for serverless or async hosting. - Check autoscaling minimums against overnight traffic, and whether the instance type still matches the model actually deployed.
- In Cost Explorer, group SageMaker spend by usage type — hosting instance-hours dominating while invocation-linked usage stays flat is the idle-endpoint signature.
Like every always-on charge, an idle endpoint never spikes — it starts, and then it persists. The catchable moment is the day it first appears in your spend, and the weekly view where a flat SageMaker line sits under near-zero invocations.