uv: Deduplicate all files in the wheel cache

as1297kj•40 minutes ago

What is the point of using packaging software that belongs to ClosedAI?

Astral people are already in the comments making pro AI coding statements. By using uv, you are literally supporting the people who want to make you unemployed for stock options.

Chris2048•8 minutes ago

> Astral people are already in the comments making pro AI coding statements.

I can't see any other comment on this post mentioning AI, did you just make this up?

embedding-shape•33 minutes ago

> By using uv, you are literally supporting the people who want to make you unemployed for stock options.

By using any FOSS or software produced by software developers and programmers, you are supporting the idea that computation can and should be automated, instead of having humans doing it. This is the origin of computing, and what we've been doing so far, and it continues to "eat the world" via automation, just like the past decades of it.

maeln•8 minutes ago

It's so human that many software engineers started to worry about automation and being replaced when it started to be something that might affect them.

LeBit•24 minutes ago

What is the word? "Facetious"?

TacticalCoder•about 1 hour ago

> deduplication at the file level: every file is now stored under its BLAKE3 hash

Blake3 is really a wonderfully fast cryptographic hash. I use it for my own "deduplication / integrity / berzerker" utility (which I made before LLMs were a thing).

If I've got a file named:

    DSC98731-b3-7b39197a22.JPG

then:

    - if that file doesn't checksum back to 7b39197a22 there's a file integrity problem (amazing and it already helped me troubleshoot issues)

    - if any other file has the same Blake3 7b39197a22 hash, it's a duplicate

    - if that 7b39197a22 checksum is in my database, "things can happen".

For example my DB can say "any file with a Blake3 hash of 7b39197a22 can always be deleted" or "any file with a Blake3 hash of 887463c09e, if it's got a generic filename like "dscXXXXX" can always be renamed to "20260722jackJohnAtTheBeach-b3-778463c09e.jpg" (or whatever suits you).

It's really great (and I know several here independently made similar schemes) and Blake3 is an amazing hash for those kind of use.

maeln•1 minute ago

Another cool thing about BLAKE3 is that it is a merkle tree. Not only it allows for good parallelism, but it also has a lot of cool property for data transfer. For example, you can check for partial validity, which allow for streaming error-detection and resend during the transfer. You only need to have the data and checksum in a way that you can start to reconstruct one or more subtree.

Someone•about 1 hour ago

For those wondering like me: Blake3 generates hashes of at least 224 bits, not, as a literal reading of that comment indicates, 40 bits (which would be bad for file deduplication, giving you a 50% hash collision after around a million files)

dist-epoch•about 1 hour ago

It's annoying that most file formats don't checksum their own content.

Even formats which should know better, like SQLite, delegate that to the filesystem, most of which are also not checksumed and which delegate that further to the storage.

PostgreSQL, which prides itself by it's quality and reliability, only turned on checksums by default in the last version, 18.

This is one great benefit of using .zip files as file formats, you get this for free.

OskarS•40 minutes ago

I think it's reasonable for a DB like SQLite to delegate that to the filesystem. There is an overhead for doing it on the DB level, and since SQLite is just a file on the filesystem which, presumably, is serving many other files as well, why would you trust anything else on the filesystem if you don't trust SQLite? Like, your PHP script (or nginx server executable, or whatever) that is calling SQLite, that's not going to be check-summed either. Either you trust your filesystem or you don't, and if you don't, checksum and error correct on the filesystem level.

Though fair enough, it could offer it as an opt-in thing.

dboreham•about 2 hours ago

People say uv is good because it's fast, but honestly I don't care about that. We switched to using it for distribution of our Python-based tools because it makes it very convenient to install directly from a git repository and then to subsequently update from same repository. No need to build a package.

uv: Deduplicate all files in the wheel cache

⚡ Community Insights

Discussion (9 Comments)Read Original on HackerNews