One: A $1.7 Billion Bill — When the Monthly Charge Was Under $5
On July 17, 2026, an AWS user named nprateem posted on Hacker News: his projected monthly AWS bill read $1,700,000,000 — $1.7 billion. His normal monthly spend? Under $5.
He thought he was seeing things. He wasn’t.
He opened an urgent support ticket and asked in his post: “Is anyone else seeing this?”
The answer: yes — a lot of people.
Screenshots flooded Reddit and X (formerly Twitter). One user whose previous month’s bill was just $0.19 received a projected charge near $2.5 billion. Another wrote on X: “I just saw $1.5 trillion on my AWS bill and my soul left my body.” The post racked up 1.3 million views and nearly 19,000 likes.
Some users, panicking, deleted every resource in their account outright — “Needless to say, I panicked and destroyed everything on that account.”
None of these figures were real charges, though. AWS confirmed quickly that the problem was with the “projected bill,” not actual billing. But even so, no one stayed calm at the sight of that number.
Two: What Actually Happened?
Within 90 minutes, AWS posted its root-cause note: “a unit pricing issue in the projected billing calculation subsystem.” Sounds convoluted. In plain terms: someone got a unit wrong.
The person who really explained it was donavanm, a former AWS engineer. In the Hacker News comments he wrote (translated):
“I dealt with this exact bug while I was at AWS. It’s a units error. We meant to charge $0.05/GB but left the unit (GB) off, so the billing system defaulted to billing per byte. $0.05 per byte of data transfer meant some customers saw million-dollar bills within hours. I got paged by support at 2 a.m., we fixed and shipped the correction between 3 and 4, then sent the apology email.”
A few key takeaways from that:
First, this has happened before. The same class of error had appeared previously — and possibly more than once.
Second, the amplification factor is staggering. 1 GB = 1,073,741,824 bytes. Forget to write “GB” and the system bills per byte, inflating the bill by a factor of one billion. A $0.05 usage charge becomes $53,687,091.20.
Third, no automated check caught it before release. It sailed through development, testing, and deployment until users saw the screenshot, panicked, and posted — only then was it noticed.
Three: How Does a Missing Unit Slip Past Every Line of Defense?
This is the most unsettling part.
donavanm went on to explain how AWS’s billing system works. Usage metering data (how much resource was consumed) and pricing data (how much per unit) are stored separately. Each SKU — each billable line item — is defined in a “pricing plan” containing the unit type, applicable region, unit price, and so on. Metering records and pricing plans are matched via fields like account ID, region, and SKU number.
If the unit type in the pricing plan is wrong — GB where it should be, or left blank, or set to byte — the conversion between metering and pricing breaks. And no automated check stops it.
One Hacker News commenter put it bluntly:
“No tests? You just get some minor detail wrong — and then hundreds of thousands of admins get a heart-attack-inducing bill?”
Another answered:
“There are tests. Test 1 verifies the service emits metering data correctly (‘we did 100 bytes of operations, confirmed billing received 100 bytes, pass’). Test 2 confirms the billing system’s calculation logic (‘feed SKU#12345 100 GB units, compute $17, pass’). But nobody runs those two test suites together — because that spans different teams, different management layers, and it’s harder to do.”
That explanation is sharp. It reveals a classic engineering-management failure: every link works in isolation, but the system breaks when you put them together. A HN user added a follow-up: “Someone at some meeting said ‘we should have tests actually walk through the billing flow,’ and someone else said ‘having tests actually generate real charges might be a legal/financial problem, maybe even a crime’ — and then nobody followed up on what the second-best alternative was.”
Four: The Real Antagonist: Cloud Billing Complexity
If you read this event as “some engineer carelessly mistyped a config,” you’ve completely missed the point.
The real villain is the complexity of cloud billing itself.
AWS has roughly 300,000 distinct SKUs (pricing units). Not 300. 300,000. Each SKU has its own pricing logic — by time, by traffic, by request count, by storage volume, by region. They stack on top of each other: a single web request can trigger compute charges, data transfer charges, storage read charges, logging charges, monitoring charges… each with its own unit and rules.
Usage data generation and pricing configuration are two entirely separate systems. They’re matched by field association, not by hardcoded validation. The design isn’t wrong per se — it gives AWS enormous flexibility to adjust pricing without touching the metering system. But the cost is: nobody can fully understand “how much will this operation actually cost me.”
In fact, AWS billing got so complicated that a whole profession emerged to interpret it — cloud cost optimization consultants. On a business doing over $100 billion in annual revenue, needing dedicated “translators” to tell customers what they actually spent says something.
Even AWS’s own engineers admitted some uncomfortable facts in the HN thread. A self-described former employee wrote: “The whole company is like a Rube Goldberg machine — few people care what happens outside their tiny slice, because they have no incentive to.”
Another ex-employee added: “I was put on a Focus plan [performance improvement] because my ‘contributions weren’t visible to leadership.’ In that environment, if I spotted a latent bug in a colleague’s code, I had zero incentive to flag and fix it — because if I did, the year-end ranking still forces a fixed percentage into the ‘low performer’ bucket, so why shouldn’t that person be the one who left the bug?”
Of course, there’s pushback. A former AWS manager countered: “Any customer impact at this scale triggers a COE (Correction of Errors) report, meaning mandatory improvement action items that eat at least one person-month. A COE is a huge headache for the whole team, which creates strong motivation to prevent problems in the first place.”
I’m not picking sides. Both can be true — cultures differ wildly across departments and management layers. But that itself shows: in an org with tens of thousands of engineers, even when everyone wants to do right, systemic holes can still slip all the way through.
Five: How Did AWS Respond?
From the AWS Health Dashboard’s public timeline:
- Jul 17, 3:52 a.m. PDT: AWS confirmed the root cause was “a unit pricing issue in the projected billing calculation subsystem” and paused bill projection calculations.
- 4:58 a.m.: Two fix paths attempted in parallel — roll back the recent change, or restore the last accurate data.
- 5:54 a.m.: Internal monitoring showed the billing subsystem could produce correct projections; further validation underway.
- 7:53 a.m.: Bad news — the rollback didn’t fix it. Both paths still in progress. Projections remain paused.
- 9:59 a.m.: Root cause located and fixed; recalculation began for all customers. Some expected to see recovery within 3 hours, all by noon Jul 18.
- 12:56 p.m.: Progress slower than expected. Full recovery pushed to early Jul 19. The statement stressed: “The displayed bill projections do not reflect actual usage and charges. Customers need take no action.”
Fixing a “forgot the unit” bug took AWS over 24 hours — and the first fix attempt failed. That, in turn, tells you: even AWS’s own engineers don’t find it easy to fix their own system’s bugs.
Six: What Does It Mean?
No actual financial loss resulted — nobody was overcharged. But the problems it exposed run far deeper than “some engineer was careless.”
First, cloud billing has become a system “nobody can fully understand.” AWS is the most extreme case, but Google Cloud and Microsoft Azure face the same problem. When a system’s complexity exceeds any single person’s comprehension, it stops being a controllable tool and becomes a black box you can only “observe, guess at, and pray doesn’t fail.”
Second, the AI era is raising the stakes. Just last week AWS announced a $1 billion investment in customer-facing AI engineering teams. A single AI training job’s monthly bill can already reach hundreds of millions of dollars. When real bills get that large, even a “projection display” error causes far more social panic and trust damage than in the past.
Third, this bug isn’t the first, and won’t be the last. Former AWS engineer donavanm lived through the identical error, paged at 2 a.m. to fix it. That means the hole has existed in AWS’s systems for at least several years, surfaced at least twice, and both times caused mass user panic. Yet it was never systematically closed.
Writing this, I’m reminded of donavanm’s description of AWS’s billing architecture: “a service’s metering data isn’t directly tied to price.” In plain language: between your usage and your bill sit at least two layers of manually configured mapping. Get any layer wrong and the result is: $5 becomes $1.7 billion.
References
- HN discussion thread: On Hacker News about inaccurate AWS projected billing data; OP nprateem posted the $1.7B screenshot, sparking 992 points and 618 comments (item?id=48945241)
- Former AWS engineer donavanm’s root-cause comment: Explains how the misconfigured pricing unit (byte vs GB) inflated bills by a billion times, and the architecture where AWS’s metering and pricing are separated
- AWS Health Dashboard notice: The official status page’s full incident timeline for “Billing Console — Inaccurate projected billing data,” from root cause to resolution
- The Next Web report: Summarizes the incident’s scope, including Reddit reactions ($0.19/month → $2.5B) and AWS’s fix schedule
- Cyber Kendra report: Cites the X post by @Bharath_uwu showing the $1.5 trillion bill screenshot, and AWS Support’s official response
- TechRadar report: Headlined “my soul left my body,” quoting multiple users’ panic, including the extreme case of someone deleting all cloud resources
