ZH version is available. Content is displayed in original English for accuracy.
Advertisement
Advertisement
⚡ Community Insights
Discussion Sentiment
74% Positive
Analyzed from 4994 words in the discussion.
Trending Topics
#duckdb#aws#open#data#source#ducklabs#https#better#foundation#projects
Discussion (202 Comments)Read Original on HackerNews
Quote from the article:
"As the CWI representative on the DuckDB Foundation... When DuckLabs spun out of CWI, we created this foundation, which holds all IP of open-source DuckDB, and will continue to do so." - Peter Boncz
AWS did not acquire the DuckDB technology itself, which is MIT-licensed open source and governed by the DuckDB Foundation which holds most of the related IP [1].
What they did acquire is DuckLabs, the Amsterdam-based services and development company behind the technology which is owned by and employs the creators and major contributors.
And MotherDuck is a US-based venture-funded commercial company, whose cloud-based data platform is centered around DuckDB but has been significantly expanded recently, including Python pipelines, an agentic context layer, and a visualisation layer.
[1]: https://duckdb.foundation/
[2] https://ducklabs.com/
[3] https://motherduck.com/
Currently it says “Today, DuckLabs remains independent and fully owned by the original creators. We've deliberately chosen not to take venture capital, so we can focus on sustainable engineering, correctness, and keeping DuckDB open and MIT-licensed for everyone. “
Future will tell if that ends up being true.
Hah. Never have I ever heard a sentence that described Amazon less.
I was really surprised when I first came across DuckDB at how good at it is for its target use cases. It is a game changer for me for the "local analytics" space, and its ability to scale up to a large degree helps a lot.
It is simply awesome to be able to point DuckDB at a mess of CSV and other files and have an instant database on top of it that I can run regular SQL over, and it is fast and just works.
Out of all the big orgs, Amazon is probably the one that has the least regard for keeping technically interesting projects alive, and the certainly will bulldoze it for some dumb reason when the next re-org comes.
Of the few examples I have I my head, I'd even say that the fate of a product that has been acquired by Google is probably even better than those of Google's internally developed products. e.g. Waze is still alive and kicking 13 years after acquisition under its own brand and hasn't been completely swallowed by Google Maps. The Nest brand also stuck around for quite some time.
Amazon will drive these people to quit within a couple years over filling out MBRs and threats about how the MBR isn't good enough (MBR = monthly business report, pure bullshit theatre that drives the whole company mad 2/4 weeks every month).
Google would drive them to quit over a longer timeframe, with no threats or harrassment, just because every time they try to ship something cool, someone else has a reason not to do it and that kills you eventually.
Heck AWS is the sole reason all these projects needed to go through these license changes to prevent AWS from completely destroying their business models.
100 items max per Transaction BatchGet 100 items, 16 MB max low write limits on same key Item size 400 KB max etc.
The constraints are what let this happen. Unconstraining it might make a better generalist product but part of what you're opting into with DDB is the dumb "put an item in get an item out semantics" and the other side is knowing that it will still work if that volume increases dramatically.
I wish I could get a role to work on OLTP databases. PostgreSQL seems to be a fascinating topic so that's on my plate.
For the same price, you can run a much more capable PSQL instance with way better features, but now you're on the hook for it being up 24/7.
For example, I want to know how to calculate the charge of incremental export. One blog says it is charged by the amount of change logs (but the official doc doesn't say so), which makes sense. But how do I estimate the amount? My hunch is: Put + Write + (1~100) * TransitWrite + Update + Delete + (1~25) * BatchWrite.
I'm curious what you mean by this. I would have said the exact opposite - AWS tends to keep projects around for a very long time. They haven't acquired very many open-source projects, but the few that they have are all still running as far as I know.
I hope they just let the team carry on and don’t contaminate it with all the other craziness going on.
What big corp is not a total mess?
The things I hear from recent AWS departures are very concerning.
Maybe this will be the rare case where founders survive corporate shenanigans and keep on doing their thing. But I'm bummed, as I've heard that optimism many more times than I've seen it happen.
It's both a great time to be gold mining in tech, and also a terrible time to be a bit employee.
My guess is that Amazon wants official hosted versions and doesn’t want to go through something like the Redis fiasco with licensing. In that case, they ended up having to support their own development anyway (with Valkey), so they might as well just buy the team.
DuckDB team probably gets a nice package and pay bump, but it’s really unlikely they’re getting hundreds of millions from this.
Assuming historical returns, your money doubles roughly every 7 years, so within the rest of your lifetime, that 1 million should turn into at least 8. That's an extremely comfortable upper-middle-class lifestyle on the interest payments alone. If your children don't spend it all, your grandchildren would easily have private jet money by the time their parents retire.
You would also have to train your kids to responsibly use the money without demonstrating it, as you’d just be hoarding it. People don’t have a good track record there, either.
I think the best you could do generationally with a million would be to try to invest moderately and draw down a small percentage (2-3%) to demonstrate fully considered use of the money. This would keep you in a middle class income but give you more ability to donate charitably, vacation together, let one spouse retire earlier, solve a financial crisis for a child, etc.
Letting them in on the thinking would give them a good chance to handle a high six/low seven figure inheritance, depending on how the investing goes.
Common definition is somewhere between 1.5 to 5 million per child.
Buying a house in desirable areas is going to be in the ~ $1M range, college is going to be hundreds of thousands, etc.
MotherDuck, a semi-competitor of theirs has raised $100mil in funding. DuckLabs has reportedly taken no external funding (so all ownership is with the founders) and was profitable with 30+ employees.
On the sidelines, other companies behind beloved open source products, like Astral, Astro, Bun are getting bought left and right.
With that as a backdrop, I think they should have been able to get quite a good payout.
The DuckDB foundation owns some equity in MotherDuck, which is a data lakehouse platform based on DuckDB, and all three have been moving closely together in making DuckDB better locally as well as in the role of a query engine that really threatens a lot of amazon's role in data lakehouses. DuckDB in the hands of a good team means that you can use AWS almost only for storage, instead of using any of the managed services.
From my own experience I can say it integrates far better into your Rust app than DuckDB does.
Well over 100 monthly contributors, too.
Indexes, transactions, a first-class storage format are all things that come included with DuckDB, that you won't have with Datafusion.
DataFusion is being used as a building block for a growing number of databases and data processing engines in Rust, as it offers the necessary primitives.
I know the projects will remain open, and ostensibly still contributed to in the same direction, but surely AWS thinks of this as another piece of a product suite to compete with Databricks.
Nontheless, I think the open embedded query engine approach DuckDB is spearheading is larger than one project, and I remain excited about the broader ecosystem (especially datafusion).
Are databases not a solved problem? Why are there lots of different databases? Why is one faster than the other? What's different between them?
There's no single set of requirements and desired properties that people have for databases.
What queries does it accept? How does it persist data? How does it manage replication and partitioning across multiple servers? Are questions with many answers and the right one varies by application.
nothing prevents to build single database which would cover all such answers. Its engineering, funding and distribution problems: no-one built it yet.
Even if you did it, I'd be surprised if a single code base is optimal across the spectrum from resource constrained microcontrollers all the way up to IBM Z Series mainframes and everything in between; along with the full spectrum of persistence from in-memory MRU (or similar) caching to full two phase global consensus as well as optimistic eventual consistency.
There's just too much scope to be the best in class at everything, and even if somehow you did it, the maintenance cost of all the options would be huge.
Best you'll get is as computing continues to increase, the cost of using less than the best becomes more reasonable. There's a lot of database applications where any reasonable database works, and there's a lot of databases that are reasonable in wide application. That may well lead to fewer databases being available, but it's unlikely to converge down to a single database. Just as most engineering domains don't converge down to a single solution for all applications.
My attempt at an answer: no, databases are not solved.
Specifically, different databases are better or worse for different use-cases.
For example, Postgres is a great "all around database" - you can use it for a lot of different things. As a "relational" database, it's really good if you have a table full of users, a table full of order, and you want to see all orders made by a user with ID=123. You need to answer questions like that a lot (eg every time someone on a website loads a page) and you need the answer fast (hundreds of miliseconds at most)
However, say your use-case is more like... you've got 100 billion rows of billing data ("joe was charged $123.45 on 2026-03-07 for a shirt, blue, size 11, brand foobar") in one table. You don't care much about joe, but you want to be able to find out how much was billed, total, in 2026-03 for blue shirts (or all year for brand foobar, or all time, for size 11). Postgres would struggle with data of that volume - you'd need a really big expensive database. A "columnar" database like duckdb (or clickhouse) might be able to answer those questions better.
Anyway, different databases are better/worse for:
- Large piles of data that you need to query in seconds
- Huge (petabytes) of data that you need to query in minutes, but can query in parallel
- Many related piles (like a standard relational database)
- Cases where you're mostly getting or retrieving single items (key-value stores)
- Huge piles of data that represent a long stream of events in time (time-series datbases)
- Piles of data that look and act more like files (object stores)
- When you need strict transactions
- When your need is very write-heavy
- When your need is very read-heavy
- and probably many others - I'm not even a huge data guy :)
So it all depends on your use-case. There are still cases that are not served well by any existing database - eg "filtering billions of rows, in milliseconds, by an arbitrary portion of several dozen very-high-cardinality columns" (to use an example that came up recently for me IRL) :)
It’s not a solved problem because each iteration of technology doesn’t just fix the mistakes of the past, it’s an evolution to solve the problems of the present.
"Are cars not a solved problem? Why are there lots of different cars? Why is one faster than the other? What's different between them?"
I think the best way to approach the subject in an easy to grasp way is to ask Gemini or another frontier AI to teach you the basics, they will do a surprisingly good job and they'll be able to react to your questions with INFINITE patience.
What do you mean by solved problem? I don't own DuckDB (sqlite, postgresql, etc). If I think I can create something as good or better than DuckDB, should I give up doing so (and get filthy rich with an acquisition) because someone thinks databases are solved? Solved databases aren't mine.
1) https://duckdb.org/2026/08/17/duckdb-20-highlights
"DuckDB has been an in-process database since day one. But people have asked us – very persistently – for a client/server mode, and we have finally caved"
This was already satisfied by numerous projects and products, and feels like a "me too" attempt to capture AI-based workflows. DuckDB always felt like "SQLite for Analytic Data" but I fear these changes and now acquiring the org leading technical direction is where they deviate for good. AWS is so unnecessary for what DuckDB can (and should, IMO) be; MongoDB jumps to mind as a cautionary tale.
And DuckLabs is the engineering behind DuckDB.
If I'm correct, this seems like a signal that AWS is coming after the segment of customers wanting to host DuckDB (MotherDucks customers).
Correct me if I'm wrong though, please.
From the MD "about page":
The idea for MotherDuck came after Jordan Tigani, MotherDuck co-founder and chief duck-herder, saw DuckDB in action and thought, “Wow, this is amazing! someone should really build a serverless version.”
...
Hannes and Mark, who founded DuckLabs to focus on the core technology and build the world’s best analytics database, were looking for partners who would build a commercial cloud offering.
Otoh, I am glad those guys got paid.
Now, with Snowflake and Databricks earning big bucks in the intelligence era, time to gain market share by having DuckLabs under its ownership?
I use DuckDB extensively for local dev as well as a parquet viewer.
* Cloudflare also seems to have gotten the same idea right with R2 Data Catalog
learn about it here: https://query.farm/haybarn/
Presumably the issue is Haybarn-specific?
The CI extension build runs are still public on GitHub, but may have expired, if you bump your git sha ref it will rerun.
Current Google cache of this page states:
> "The DuckDB project is governed by the non-profit DuckDB Foundation . The Foundation and DuckLabs are not funded by external investors (e.g., venture capital)."
But "(Last updated: Aug 2026)" and I was not able to find this text anymore.
I wonder if they took vc money after all. I guess not, but still, it seem like to be difficult to live as an independent open source company. Getting a big co as a parent/sponsor is probably the next best thing.
I've been eyeballing DuckDB and LanceDB as part of AI agent memories. This gives me a vibe that AWS will use DuckDB somehow in their ai agents sometime in the near future after seeing the potential.
DuckDB to the rescue.
Also embed the crap outta DuckDB locally in local agents running on VPS, etc
So give your tools, your life, free bug fixing, priority attention to me because I am getting my paid job done. Why they need money anyway, they can leave on reputation of OSS contributors. Also not to forget I donated 5 dollars last year so now give me full certified audit of your finances of last 5 years.
"We also worried that scaling DuckLabs into a much larger sales, support, and operations organization would pull our attention away from the technical work and open-source community that made DuckDB successful in the first place."
Given AWS's services arm seems like a good play for a team. Congrats!
https://github.com/shreeve/duckdb-harbor
one feature I can't wait from the DuckDb team are real-time materialized views.
I hope the future for DuckDB is still bright
As I understand it, DuckDB is MIT-licensed and governance is provided by a separate, non-profit DuckDB foundation. This was done explicitly to prevent BSL-ification.
I'm very curious to see whether that approach holds the line. I wouldn't bet on it, but one can hope?
On the other hand, do we have to live in an oligarchy?
https://query.farm/haybarn/
Or
https://github.com/Query-farm-haybarn
I've been releasing the versions all the way since 1.5.3 and all community extensions.
What does this mean? Can I easily sign my own private extensions with my own key?
We are currently allowing unsigned extensions, loading them, then locking the duckdb config to prevent loading unsigned/untrusted extensions