Google Cloud customer wakes up to $18,000 bill despite $7 budget

sspeckx about 3 hours ago 21 commentsRead Article on tomshardware.com

ES version is available. Content is displayed in original English for accuracy.

⚡ Community Insights

Discussion Sentiment

63% Positive

Analyzed from 673 words in the discussion.

Discussion (21 Comments)Read Original on HackerNews

ddtaylor•10 minutes ago

Sounds like a great mistake for Google to find a way to repeat. Why innovate when you can abuse users and hide behind "complexity" as "plausible deniability"?

You'll all keep using them either way.

victor106•about 2 hours ago

Why doesn’t GCP provide a way to say “shut down all my services if my cap is reached”?

brazukadev•22 minutes ago

They have a $18,000 reason for that.

caminanteblanco•7 minutes ago

Well since they waived the fees, it sounds to me like they have an $18,000 reason to stop this kind of thing from happening in the first place.

I understand that $18k is probably a drop in the bucket, but surely there's a middle ground here.

ReptileMan•about 3 hours ago

That is quite hostile to their consumers, no matter how they spin it. If you put a budget on something it should be capped.

dpoloncsak•about 2 hours ago

I am the last person to defend Google Cloud and it's awful UX.

With that said, when you go to set a budget it warns you "Setting a budget does not cap resource or API consumption. Learn more." with a hyperlink to https://docs.cloud.google.com/billing/docs/how-to/budgets?_g...

Telaneo•about 2 hours ago

The only way I can read that is 'setting a cap does nothing' but reading that tells me that it only turns on email notifications. Not any better really. It's simply not a cap. It's an alarm.

sunaookami•about 2 hours ago

Yes, there is no way to set a budget for Google Cloud. And alarms are delayed up to two hours (!)

dpoloncsak•43 minutes ago

Agreed. I don't agree with the underlying design decisions either, but you literally set a "Budget Alert" on Google Cloud. It's designed to be an alarm, not a cap. I was just trying to point that out

ReptileMan•15 minutes ago

I am fairly sure that this is antipattern on purpose. If you ask a thousand people on the street what a budget means - they will coalesce on - the money I am willing to spend on something.

The fact that google redefine what budget means and put a warning doesn't make it ok.

7bit•about 2 hours ago

Click here to let the puppy life*

* By clicking here you agree to kill it

And you're defending that?

dpoloncsak•about 1 hour ago

No, I think Google should provide easy tools to actually cap spending, instead of recommending you set quota limits on your APIs.

The article, and the comment I was replying to, make it seem like an error in the Google Budget system. I'm simply trying to say this system is working as designed and documented.

perryizgr8•about 2 hours ago

I think I read somewhere that calculating and limiting cloud usage costs is a really hard problem. But I feel that if Google were motivated to do it, they can do it. It's hard, not impossible. They just don't care to solve this particular problem.

subscribed•about 2 hours ago

If they can COUNT it and charge based on that, that means they can count it and react.

If I, not having their budget or engineers, can have pretty much instant Prometheus event reacting to metrics, surely it wouldn't be too hard for them to have triggers like this -- somehow their AI can automatically ban people based on something, can't they do something for the customers?

They can, just don't want to.

mhitza•24 minutes ago

In the article it states that this person had an account that would have been limited to $2000 in usage.

And the system automatically upgraded them to higher spending limits when they crossed the $1000 in usage costs.

They could definitely make that an opt-in feature.

AlotOfReading•about 2 hours ago

It's the same fundamental problem as view counters, something Google is famously good at solving. Eventually consistent solutions are well-understood, and wouldn't have these kinds of massive cost-overruns.

lazide•about 2 hours ago

Depends on latency. 24 hour delays on an eventually consistent counter used for billing absolutely would cause this problem.

moring•about 1 hour ago

It seems hard to believe that a one-hour delay on such a counter is impossible to achieve, and one hour would reduce the risk from "catastrophic" to "serious problem" in most cases.

Also, if implementing a cap is a desired feature that justifies trade-offs to be made, then it is psosible to translate the budget cap (in terms of money) back into service-specific caps that are easier to keep consistent. Such as "autoscale this set of VMs" and "my budget cap is $1000/hour", with the VM type being priced at $10/hour, translated to "autoscale to at most 100 instances". That would need dev work (i.e. this feature being considered important) and would not respect the budget cap in a cross-service way automatically, but still it is another piece in the puzzle.

AlotOfReading•about 1 hour ago

Yeah, there's an implicit assumption was reasonability.

But a big part of the value in large clouds like GCP is the network's interconnectedness. Plus even if there was some global event that made communications impossible only for the billing service, I'd still expect charges to top out roughly proportional to the number of partitions as they each independently exceed the threshold. GCP only has 120ish zones.

jdgoesmarching•about 2 hours ago

It’s hard on AWS as well, but I agree. There’s just no incentive for the billing experience to be better.

kjellsbells•24 minutes ago

I mean yes, look at Corey Quinn [1] for example. He has built an entire career out of the fact that cloud billing trips people up.

(Generally, tech seems to skate by on creating insanely complicated things, knowing that given enough pain, people will start blogging about their solutions, ie effectively outsourcing the cost and effort of doing something about it.)

[1] https://www.lastweekinaws.com/