The $4,200 loop: one unscoped OpenAI key, one weekend
A retry loop, a wildcard scope and a Slack DM. How a test key became a four-figure invoice, and how faster visibility would have changed the outcome.
The key was created by someone on the data team on a Thursday to try a summariser. By Friday evening it was in three places. On Saturday at two in the morning a retry decorator with no backoff started calling it every fraction of a second. It ran for most of the weekend.
▸ MOVE YOUR CURSOR OVER THE REDACTIONS.
What happened
Nothing about this incident is exotic. That is the point. A test credential was created with the vendor’s default scope, which for most AI providers means everything the organisation can do. It was shared in a direct message because that was faster than the secrets manager. It ended up in a Lambda because the Lambda needed it. Nobody lied, nobody was careless by the standards of a normal week.
The loop itself was a @retry decorator with no backoff and no cap, wrapped around a call that was failing because the prompt exceeded the context window. Every failure was instant, so every retry was instant.
“We found out from the invoice. Not from an alert, not from a dashboard. From the invoice, three weeks later.”
What visibility would have changed
SubScope doesn’t throttle a vendor credential — nobody does that on your behalf without touching the vendor’s own console. What it does is surface the spend the moment it looks wrong, instead of three weeks later on an invoice. With hourly polling on this key, the +412% anomaly would have shown up by roughly 15:30 on Saturday, not in March. That’s still a page to the owner and a manual fix — but the damage window shrinks from a weekend to an hour.
# what SubScope would have surfaced by ~15:30 UTC Saturday
openai_prod_key_7f3a
cost: +412% vs 7d avg (visibility only — nobody throttled anything)
owner: unassigned <- flagged the moment the key was added
scope: organization.admin (wildcard — flagged, never narrowed)The three rules we took away
- No owner, no key. Ownership is a creation-time requirement, not a cleanup task.
- Default scope is the wrong scope. Vendors default to everything. Start from nothing.
- Cost is a security signal. A runaway loop and a stolen key look identical on the invoice; attribute spend to the credential and both become visible in minutes.