Client-side load balancing at a million requests per second

Discussion (2 Comments)Read Original on HackerNews

cjbooms•about 16 hours ago

Hi all. Blog author here, happy to take any questions you might have. It's a bit of a long one, but I put a lot of effort into the story arc and readability so it's (hopefully) an easy top to bottom read, that takes you on our journey. It covers some advanced distributed computing topics, such as replicating the exact same hash-ring as our existing load-balancer in the client application (JVM), what we implemented to fade-in new pods slowly so their caches get a chance to warm, and how we attempted to migrate to Availability Zone (AZ) Aware routing to save on AWS' inter-az transfer fees. Hope you enjoy it!

charleshn•about 2 hours ago

Thanks for the article.

I have two questions/comments:

1. The N-ring fade-in is quite neat. I guess without the constraint of hash parity rendezvous hashing [0] could have been an elegant approach since it has support for weights (and generally better statistical properties than consistent hashing based on rings).

2. You mention still having the fallback of your existing load balancer. Is this a temporary thing during rollout or do you intend to keep it long-term. Asking because I generally tend to stay clear of fallbacks in distributed systems, as they introduce bimodality and metastable failures [1] [2].

[0] https://en.wikipedia.org/wiki/Rendezvous_hashing

[1] https://builder.aws.com/content/3EuS9Sakq7L3VLQIF3qzfMfke1Y/...

[2] https://brooker.co.za/blog/2021/05/24/metastable.html

Client-side load balancing at a million requests per second

⚡ Community Insights

Discussion (2 Comments)Read Original on HackerNews