Full-Text Search with DuckDB

kevincox•43 minutes ago

It seems that DuckDB by default downloads and runs extensions at runtime when you use certain features? This seems unnecessarily risky.

https://duckdb.org/docs/current/extensions/overview#autoload...

I would love to have more detail on this mechanism.

bambax•about 2 hours ago

I'm relatively new to DuckDB (coming from SQLite) and I love it so far. Some parts are magical (described in the previous article by the same author: https://peterdohertys.website/blog-posts/dab-of-duck.html)

You can point DuckDB to almost any data source and boom, you get an SQL table that you can search, sum, or join to any other data. Or you can attach existing databases from completely independent db systems, and query and join them as one, without having to first importing anything.

It feels exhilarating (if you're into that sort of thing!)

thechao•about 2 hours ago

My honeymoon with duckdb wore off pretty quickly when I need to compile it, myself, into a single-file concordance. I understand it's open source, so I'm free to be ignored. But, it's positioning itself as a drop-in replacement for SQLite; a large part of SQLite's appeal is its ergonomics — its single-fileness — letting me deliver a rational object to my users.

EDIT: "drop-in replacement like SQLite", not "for SQLite".

gcarvalho•about 1 hour ago

> it's positioning itself as a drop-in replacement for SQLite

While SQLite is often used for comparison (“SQLite for OLAP”), I’ve never seen DuckDB market itself as a “drop-in” replacement. Where did you see that?

efromvt•about 1 hour ago

Sqllite and duckdb serve pretty different niches; duckdb is less embeddable but on the OLAP side it’s by far the best today. I wouldn’t ever see them as competing for the same app, though

slotix•about 2 hours ago

We wrapped exactly this into a GUI - attach MySQL and PostgreSQL, files/ s3 as sources, query them together with DuckDB. No imports. https://streams.dbconvert.com/cross-database-sql

jiehong•about 1 hour ago

Anyone tried both duckdb and clickhouse local?

Clickhouse seems less marketed, but seems quite similar.

efromvt•about 1 hour ago

I didn’t realize they had chDB, honestly, need to give that a shot! The local CLI isn’t quite the same ergonomically

rahimnathwani•about 3 hours ago

Has anyone used DuckDB (or anything else) to create an open source way to publish a mailbox so that a regular person can browse it and search it?

I'm aware of jmail.world, but they haven't (yet?) published the source code.

I had Claude hack something together recently: https://healdsburg-youcubed-emails.vercel.app/

It works fine for this small set of emails, although the search isn't great, and there was more preprocessing that I would have liked. (I would prefer to be able to point a single binary at a pst or mbox file, and have it magically serve it like this, even if it means I need a VPS to serve it.)

joouha•about 2 hours ago

Maybe https://github.com/wesm/msgvault will do what you need?

rahimnathwani•43 minutes ago

Thanks.

msgvault seems really good. The tui is fast and FTS5 search works well.

I will definitely use it.

But it doesn't allow me to make a mailbox accessible to a wide audience, because:

- AFAICT there's no web version

- inline images don't show up

zffr•about 3 hours ago

What’s your use case for this?

bambax•about 2 hours ago

Here's one: a client of mine has a bunch of SnapLogic pipelines that are configured to send errors via email, and there is no other persistent logging system. This results in tens of thousands of emails that are insanely hard to search and parse for any useful auditing.

rahimnathwani•about 3 hours ago

Making it easy for members of the public to search and browse email sent by or to government employees.

These emails aren't published by default but email archives are often included in responses to public record requests.

Ideally anyone who receives one of these archives would be able easily inspect it themselves, and also make it available to others.

Full-Text Search with DuckDB

⚡ Community Insights

Discussion (14 Comments)Read Original on HackerNews