Making Postgres 300x faster for analytics: batching, operator fusion, and SIMD

ppoly2it about 5 hours ago 22 commentsRead Article on malisper.me

DE version is available. Content is displayed in original English for accuracy.

⚡ Community Insights

Discussion Sentiment

64% Positive

Analyzed from 825 words in the discussion.

Discussion (22 Comments)Read Original on HackerNews

malisper•about 1 hour ago

Author here. Let me know if you have any questions!

Let me take a shot at answering what I think will be the most common question: how can I trust this? Our #1 priority right now is correctness. Over the past two weeks, I've done a mix of formal verification and differential fuzz testing. We've been able to prove over 1000 user facing functions have the exact same logic in both pgrust and postgres (see the proofs directory if you're curious). For cases where formal verification is not easy, we've taken the c implementation of a function and the rust implementation of a function and ran millions of inputs through each of them and confirmed they gave the same results every time.

We've only covered about 15% of the surface area so far, but in the process, we've discovered ~100 bugs in pgrust and ~20 bugs in Postgres itself. My favorite postgres bug we found is this one[0]. Postgres has a quadtree implementation. Due to floating point rounding, it was possible for a point to be neither above, nor below, nor even with the center point of the quadtree.

We've also entered engagements with Antithesis[1] to do Jepsen style fault testing and Aretta[2] to do more serious formal verification.

If you want to support the project, the easiest way is to give us a star on GitHub[3]

[0] https://www.postgresql.org/message-id/19597-39c532e61d78dff6...

[1] https://antithesis.com/

[2] https://aretta.ai/

[3] https://github.com/malisper/pgrust

btown•16 minutes ago

If someone wanted to use this as a real-time WAL-tracking read-only mirror of a live production database, for analytics work, is it ready for that use case yet?

malisper•14 minutes ago

You can try it. We're happy to help you with it, but expect there to be issues to work through. You would want to do it for something non-critical

jnwatson•24 minutes ago

The floating point comparison bug is nightmare fuel. I could look at that for years and never spot the mistake.

lizimo•11 minutes ago

Is `pgrcolumnar` the default storage layout for tables? It would be cool if the same storage engine outperforms vanilla Postgres under both OLTP and OLAP workloads.

AlloyDB from Google Cloud uses columnar storage like a secondary index, while the relations are still stored in TOAST.

malisper•9 minutes ago

pgrcolumnar is not the default storage method. Right now, it's exposed as a table access method. There's lots of design space for how to do this so I want to avoid pre-committing to anything

doctorpangloss•43 minutes ago

“Show me the prompt.”

cognitiveinline•about 3 hours ago

pgrust seems to have good momentum. AGPL is an odd license for a non web project. Postgres is MIT-like, and that drove it's adoption.

Have pgrust folks reconsidered this? Else, IMO we can have an independant rust port of pgrust, which can be MIT, which will garner more attention.

malisper•about 1 hour ago

Author here. At least for databases, AGPL (or stricter) has become standard. The issue is it's so easy for megacorps (Amazon, Google, etc) to take a permissively licensed product and monetize it at the expense of the original standard.

For instance, Mongo, Cockroach, and Materialize have all gone source available. We picked AGPL because it's the best balance between open source and prevents Amazon from just repackaging it and selling it.

If AGPL is an issue for anyone, we would be happy to dual-license under a commercial license.

mey•34 minutes ago

There is nothing wrong with wanting to be compensated for your work, but for people like myself which use a cloud managed DB solution (GCP CloudSQL PostgreSQL) it means something like this would never be available.

I consider AGPL a poison pill in my work. That is not true with a suitable commercial license, although I expect a lot more commercial product (support/features/etc). As you note, your objective is to prevent commercialization of your software, but radically speeding up analytics is primarily a concern of large organizations so it seems like a mismatch in purpose.

cognitiveinline•40 minutes ago

Sure, that's your prerogative, and kudos for not talking up open source. I'm not amazon size so can't use it, and AGPL is a no go for DB, don't want to be forced to open source my app because I use this!

Will await a MIT based fork myself.

jnwatson•21 minutes ago

Why would AGPL force you to open source your app? Unless you literally compile your app with pgrust by modifying the pgrust source code, you're safe. Clients aren't bound by the AGPL because they aren't derived works.

appplication•about 2 hours ago

Oh bummer. I was really excited about pgrust but AGPL is a dealbreaker. Not for me personally, but it will never see wide adoption because it’s a banned license in most corporate environments. Lack of path to wide adoption means it’s dead in the water.

It’s weird because those who actually care about optimized pg gains are most likely large corporate customers. Why make a product targeting them and license it in such a way they’ll never use it?

This also hard blocks upstreaming any beneficial features into core Postgres.

guenthert•14 minutes ago

If it is indeed 300* faster, I'm sure more rational corporations will rethink their license policy or be left in the dust.

xyzzy_plugh•11 minutes ago

They could simply spend a few months and a few million tokens and get their own port, no?

I doubt even 30000x faster would prompt a policy change.

busterarm•about 2 hours ago

Everything around Rust is political, so the license choices are also about political statement.

johnsonjo•about 1 hour ago

Most official Rust projects are dual MIT/Apache licensed by convention [1] (and most Rust libraries from third parties I've seen that are open source MIT follow the MIT/Apache dual license), so seems like this library shouldn't just be AGPL for a typical political choice of a Rustacean?

[1]: https://rust-lang.org/policies/licenses/

busterarm•about 1 hour ago

official projects are usually run by sensible people who want to do things and aren't leading with ideology.

We're literally talking about an "X but in Rust" project already...

ognarb•about 2 hours ago

2 commits in the repo both generated by claude. This is AI slop, I wonder where you see good momentum?

Whitespace•about 2 hours ago

main indeed has two commits, but it clearly states the location of the rest of the commits, so I wouldn't be critical of main itself.

  hey claude, do a breakthrough
  You can find the actual git history at the v0.2 github tag.
  
  Co-Authored-By: Fable <noreply@anthropic.com>

Now we see https://github.com/malisper/pgrust/tree/v0.2 has almost 6000 commits in it, with the very first one on 2026-07-02. That's a lot of token momentum!

It's easy to claim AI slop nowadays, but you should still mistrust-but-verify.

esafak•about 2 hours ago

4K stargazers in a week.