We scaled PgBouncer to 4x throughput

x4m•about 2 hours ago

Just use https://github.com/yandex/odyssey :) It's a scalable PgBouncer.

sevg•about 2 hours ago

Fun (semi-related) fact, ClickHouse was originally developed by Yandex :)

saisrirampur•about 2 hours ago

We started with the most battle-tested and native option to Postgres, which is PgBouncer and tried tuning it the right way. Also now that long due kinks like support for prepared statements are solved, it’s been working really well. There are many customers scaling well with 10K+ Postgres connections. We will consider other options like odyssey, pgdog in the future!

Side note: I’m not a big fan of having 10K+ connections on Postgres, 100s are more than enough to scale Postgres well. But that’s a story for another day. ;)

nosefrog•about 3 hours ago

Interesting. We run pgbouncer via kubernetes so it was straightforward to make multiple pgbouncer processes on one machine. Also straightforward to get them running on multiple machines, which helps because we run on Azure and they like to cause rolling outages across our fleet via VM maintenance...

saisrirampur•about 2 hours ago

Ack, makes sense. I’m very curious on how this affects throughput due to a potential extra network hop from pgbouncer to Postgres. Expecting it to have a minor difference, but still curious.

JustSkyfall•about 2 hours ago

I've been using pgdog (https://github.com/pgdogdev/pgdog) and it has worked really well for my needs!

levkk•about 1 hour ago

Glad you like it. It was built to fix some of PgBouncers shortcomings that we ran into at Instacart many years ago, and to have a stronger foundation for scaling Postgres horizontally (that's sharding)!

saisrirampur•about 2 hours ago

Ack, we will consider offering pgdog in the future! More context on why we chose pgbouncer https://news.ycombinator.com/item?id=48873867

jauntywundrkind•about 1 hour ago

It's been a pong pong back and forth between PG reverse proxy submissions lately! "Why we built [Pgdog]": https://news.ycombinator.com/item?id=48819308

DylanSp•about 1 hour ago

First time I've heard of so_reuseport, which is interesting. The important parts of the setup seem to be that + peering; is peering built-in to PgBouncer and simple to set up?

__s•about 1 hour ago

Yes, https://www.pgbouncer.org/config.html#section-peers

nzeid•about 3 hours ago

Was there a disadvantage to using HAProxy + multiple PGBouncer instances?

__s•about 2 hours ago

SO_REUSEPORT[1] pretty much does all we want in kernelspace vs unnecessary userspace hop inbetween. These all run on same VM

1: https://lwn.net/Articles/542629

jauntywundrkind•about 3 hours ago

This was more for fun than real use, but I greatly enjoyed hacking something similar into rqbit bittorrent client. I wanted to run an instance of 'rqbit download' per torrent via so_reuseport. When a peer tries to connect, it gets sent to a random instance. So I built a whole rendezvous system, where instances find each other & either proxy data to each other or fd pass the socket to each other directly to get the peer socket to the instance that needs it. It uses postcard rpc to chat between instances.

Clickhouse's so_reuseport rendezvous needs are obviously for a very different, but fun to see some so_reuseport coordination like this (for a much more practical use)!

It'd be really neat to have some kind of general peering protocol that different apps could use. This whole exercise was gratuitous as heck for my application, I don't even really intend to use this, but it was a fun path to walk down. So I don't really know what the broader protocol would really be for, what we would use it for. But it seems like such a cool idea! A shared Turso database would probably be a bit more practical than the rpc system, honestly. Ha.

https://github.com/rektide/rqbit/tree/peering

odie5533•about 3 hours ago

Article should show the config:

[pgbouncer] listen_addr = 0.0.0.0 listen_port = 6432 so_reuseport = 1 peer_id = 1 unix_socket_dir = /tmp/pgbouncer1

[peers] 1 = host=/tmp/pgbouncer1 2 = host=/tmp/pgbouncer2 3 = host=/tmp/pgbouncer3 4 = host=/tmp/pgbouncer4

We scaled PgBouncer to 4x throughput

⚡ Community Insights

Discussion (15 Comments)Read Original on HackerNews