ES version is available. Content is displayed in original English for accuracy.
Advertisement
Advertisement
⚡ Community Insights
Discussion Sentiment
50% Positive
Analyzed from 1227 words in the discussion.
Trending Topics
#cpu#limits#using#pods#requests#pod#scheduler#cores#don#running
Discussion (38 Comments)Read Original on HackerNews
It mentions that a cpu request is a guarantee, but how is that enforced? If I have 32 pods running on a 32 core machine, each with 1cpu requested, what stops one of those pods using an unfair share? I assume we just rely on the Linux scheduler. If I have 16 pods with 1cpu and 1 pod with 16cpu, does the Linux scheduler make sure to give the 16cpu pod more time? Or are we back to using cgroups.
> If I have 16 pods with 1cpu and 1 pod with 16cpu, does the Linux scheduler make sure to give the 16cpu pod more time?
Yes if there's CPU pressure the 16cpu will get 16x more than 1cpu.
There's a part about it: https://github.com/inevolin/k8s-cpu-limits-analyzed#3-withou...
I don't use all this fancy stuff, but wouldn't you set that up as each of the 32 pods is limited to running on a specific core? No sense letting them each run on all cores, because all of the X per core will get too big.
So this just all assumes you have a setup where all teams communicate the necessary information perfectly.. what happens in practice is workloads degrade at edge cases because there are 256 threads running for a thread pool instead of 4.
The article behind it explains it better, but a quick glance at the image above is all you need if you don't care too much about the explanation.
https://home.robusta.dev/blog/stop-using-cpu-limits
It was already the case in 2018.
Also, no mention of the scheduler overhead. And the maintenance overhead is the worst.
"For the Love of God, Stop Using CPU Limits on Kubernetes" literally same title
I’m not sure why that is but a large number of the F100s I contract with are suddenly deploying 4 times the number of containers they had before.
(The title of this was also stolen for this HN post, although the GitHub repo makes no mention of it...)
I've handled outages of CPU time available getting suddenly compressed (we were running pod priorities with staging/prod on one cluster and up to 70% spots in 2019) and then learning that some very important applications outgrew their original requests, gone unnoticed because limits were removed a year or so prior. You can fix this with monitoring/right-sizing tools, but that requires your org to not be dysfunctional, and my style of platform engineering usually has to account for the org being very dysfunctional.
Sorry for the cheeky response.
CPU Limits have a place, you don't want a bad change for 1 deployment object affect all neighbors by taking all the CPU. You need to be able to constrain the blast radius. This doc gives me strong AI vibes. Setting CPU limits isn't free. You still need to care about how the programming language that you use discovers those limits, and correctly handles them. For e.g. if you spin up a 100 Java threads, but only have 1 cpu as the limit, that's bad design.
> The CPU request typically defines a weighting. If several different containers (cgroups) want to run on a contended system, workloads with larger CPU requests are allocated more CPU time than workloads with small requests.
So I guess limits _would_ protect other pods to a degree. Though I agree that it doesn't seem worth the tradeoff of your pod getting constantly interrupted while the rest of the box is sitting idle
https://kubernetes.io/docs/concepts/configuration/manage-res...
Essentially, they avoided CFS quota throttling by assigning exclusive CPU cores via cpusets. That sacrifices some burstability and packing efficiency in exchange for stronger, more predictable CPU isolation.