bzip3

172

ttosh about 4 hours ago 57 commentsRead Article on github.com

RU version is available. Content is displayed in original English for accuracy.

⚡ Community Insights

Discussion Sentiment

72% Positive

Analyzed from 2042 words in the discussion.

Discussion (57 Comments)Read Original on HackerNews

altairprime•about 3 hours ago

Previously:

“Hi, tool author here.” A useful explanation of Burrows-Wheelers transform as used by bzip3: https://news.ycombinator.com/item?id=42902407

“bzip3 is not yet listed on the large text compression benchmark” It is now: https://mattmahoney.net/dc/text.html

(2 years ago, 176 comments) https://news.ycombinator.com/item?id=42899713

(4 years ago, 104 comments) https://news.ycombinator.com/item?id=31324439

p-e-w•about 1 hour ago

> “Hi, tool author here.” A useful explanation of Burrows-Wheelers transform as used by bzip3

FWIW, the Burrows-Wheelers transform is also used by bzip2, so this isn’t a new feature even though that quote kinda sounds like it is.

8organicbits•about 2 hours ago

I was processing compressed .jsonl files recently (JSON lines format). I found that lzma gave a much better compression than gzip or bzip2, which helps for archival costs, but it's challenging to work with as software support is lacking. I do duckdb processing which supports gzip transparently. There's an extension for bzip2, but not for lzma or bzip3.

I ended up using gzip because it's best supported by the software I use and most likely to have support in software I adopt. But it gave the worst compression results of the options I tried. These bzip3 numbers certainly give me FOMO...

nvme0n1p1•about 1 hour ago

zstd is the go-to compression format these days. It's even supported in low-level software such as many linux filesystems.

I don't know much about duckdb but it looks like it supports zstd too: https://duckdb.org/docs/lts/data/json/loading_json

wongarsu•41 minutes ago

The thing zstd got really right is fast decompression. For write-once read-never data like backups lzma (aka xz/7zip/lzip) is great. But it takes forever to decompress. On zstd I can get good compression while decompressing the file only marginally slower than reading the uncompressed file from SSD

Writing your files directly into a compressed stream and decompressing on the fly has become almost a standard workflow for any files I'm going to read and write sequentially anyways. No need for the data to ever exist uncompressed on the file system. Previous formats never did that for me because they either had too much overhead or too little gain, often both

cb321•24 minutes ago

Agreed! Relatedly: https://news.ycombinator.com/item?id=49599953

8organicbits•5 minutes ago

How did I miss zstd?

Here are my benchmarks for 2.3 GB of jsonl, on a laptop. Compressed size, compress time, decompress time; using defaults.

    gzip  7.3%  21s  9s
    bzip2 4.6% 251s 50s
    bzip3 3.3%  82s 69s
    zstd  6.9%   2s  3s
    lzma  4.7%  51s  3s

jubilanti•about 1 hour ago

all hail zstd, the one format to rule them all

benatkin•about 1 hour ago

Not really, it's a popular dictionary-based compression format.

zinodaur•about 1 hour ago

I'm pretty new to choosing compression libraries - I started with zlib and was delighted at how much faster and smaller zstd made things.

Since we kind of need a default "Need to compress something? Use this!" setting - would you prefer zlib over zstd, or something else for that role?

esseph•about 1 hour ago

”Not really" what?

It's hard to understand what point you're trying to make. Can you clarify?

bob1029•22 minutes ago

For structured logs and json I've found a lot of success with PPM-style schemes.

If your JSON file has many of the same object, you could see ratios in the single digits.

woadwarrior01•28 minutes ago

I'd recommend trying openzl for jsonl.

amelius•about 2 hours ago

> However, the complexity of the algorithms, and, in particular, the presence of various special cases in the code which occur with very low but non-zero probability make it impossible to rule out the possibility of bugs remaining in the program.

Sounds like perhaps a nice testcase for formalization + AI?

teiferer•about 1 hour ago

It's beyond me why such foundational libraries don't have formal correctness proofs attached these days.

ot•about 2 hours ago

The benchmarks are disingenuous, to the point of looking cherry-picked. The block size for bzip3 is set to 512MB, but the window size for zstd is left to its default (8MB I believe for high levels). So in this corpus, which is made up of all versions of Perl source code concatenated, the window is too small to see all the identical files and just match them. Also corpora made out of very long repetitions are pretty much the best case scenario for BWT-based compressors.

If we match the window size of zstd to that of bzip3 we get dramatically different results:

    % gzcat *.gz | time zstd -T8 -16 | wc -c  # baseline
     2819113884
    zstd -T8 -16  2054.50s user 3.47s system 783% cpu 4:22.80 total

    % gzcat *.gz | time zstd -T8 -16 --long=29 | wc -c
     196405076
    zstd -T8 -16 --long=29  1083.06s user 2.41s system 783% cpu 2:18.55 total

Almost 15x smaller than the baseline, and more than 2x smaller than bzip3, also CPU time halves (since long matches are found earlier, so there's less work to do).

(the baseline number is slightly different because I don't have the exact Perl version set used by the author)

Also, in the benchmarks using lrzip, which would make the window size less relevant, zstd is not even compared.

nvme0n1p1•about 2 hours ago

> 8MB I believe for high levels

Yep, i found it in the source here:

- https://github.com/facebook/zstd/blob/d9c0c7e2cf8a8bf9fb98d3...

Also, zstd docs say:

> Note: If windowLog is set to larger than 27, --long=windowLog or --memory=windowSize needs to be passed to the decompressor.

That always seemed annoying to me. They couldn't allocate 5 more bits somewhere to let the decompressor autodetect longer window sizes?

dzaima•about 1 hour ago

> That always seemed annoying to me. They couldn't allocate 5 more bits somewhere to let the decompressor autodetect longer window sizes?

I believe this is just to prevent the decompressor from arbitrarily blowing up memory usage based on the input; I think if you want to accept long windows you can just always decompress with --long=63 regardless of whether the input needs it? (you will run out of RAM decompressing a long=63 file though of course)

rmunn•about 1 hour ago

2^27 is 128 megabytes. How much RAM do you want the decompressor to have to allocate for every file? Especially since you can't tell, by looking only at the file size of a compressed file, how many bytes it will decompress to. You could read the file header, but if it's a malicious "zip bomb" type of file, the header could be lying.

cb321•about 1 hour ago

I just tried a tar file of a git clone of the linux kernel sources where the most recent commit is 72c395024dac5e215136cbff793455f065603b06 (early Feb of this year). zstd -19 got a slightly smaller size (3582930348 bytes vs bz3 -b 511's 3597411687 bytes or 0.4% advantage to zstd). More significantly 4-core zstd decompression was 2.05 seconds vs a whopping 297 seconds for bzip3 -dj4 - 145x or over 2 orders of magnitude slower (about as much time to decode as to encode in the first place). bzip3 1.5.3 compiled with gcc-16.1.0. Granted, the .git objects are all compressed already and uncompressed tar-ball was only 5426667520 bytes, but even so...A lot of people care about fast(-ish) decompression. Maybe I did something wrong? Maybe `rm -rf .git` first would be a better benchmark?

pajko•33 minutes ago

This was my point too, but related to LZMA(2?) / xz, as the exact parameters were not specified while it supports setting compression level up to -9e, and the dictionary size can be controlled directly as well (in addition to quite a lot of fine-tuning knobs), increasing it up to 1536 MiB.

optionalsquid•about 2 hours ago

How does memory usage compare between your two runs?

The benchmarks report 687M for their run of zstd, and 12178M and 18301M for the two runs of BZip3. Which itself is a bit eyebrow raising

xxs•about 1 hour ago

Nice catch.

it has been a long time since: "lies, damn lies, benchmarks" failed to hold true. Sometimes I wonder why gaming benchmarks has become so common.

CJefferson•about 2 hours ago

Wow, that is widely disingenuous, I don't really think there is any excuse for that, I don't believe someone deep in compression algorithms wouldn't know they could adjust the block size, and 512GB is a huge block size for bzip3, as it needs to basically all be in memory so you can't pretend that's just 'the standard value'.

ot•about 2 hours ago

> 512GB is a huge block size for bzip3

Sorry! That was a typo, it should have been 512MB (now fixed). Still huge.

JdeBP•about 2 hours ago

An interesting unintentional benchmark is to go to https://github.com/iczelia/bzip3/releases and see to what degree bzip3 compresses its own release archives; and go to https://github.com/iczelia/bzip3/blob/master/.github/workflo... to see what options have been chosen for the other compressors here.

charcircuit•9 minutes ago

It's distasteful to use such a name when it's not created by the bzip authors. For some reason open source developers love using would be trademark infringing names instead of coming up with something unique.

CodesInChaos•about 2 hours ago

I think this should include benchmarks zstd with larger windows and long range mode. I wouldn't be surprised if the window they used is smaller than an individual version tar file, which would prevent useful compression.

altairprime•about 2 hours ago

“Additional benchmarks on the same dataset” (2025) explores various compression levels with and without long range mode: https://news.ycombinator.com/item?id=42901476

ThiraSoft•about 1 hour ago

I would be interrested in a comparison with openzl

Retro_Dev•13 minutes ago

That's like comparing apples to pears. OpenZL is not general purpose. You specify a format for data and it compresses that format. Specifying a general "could be anything" format would be interesting, but I doubt it would compress as well.

red_admiral•about 1 hour ago

Why LGPL when the original license is more permissive? (https://sourceware.org/bzip2/manual/manual.html)

atiedebee•36 minutes ago

bzip3 is not made by the same author as bzip2, that's why there is a licence disparity.

LtWorf•34 minutes ago

Please stop questioning people's choice of license.

If you don't like it, you're entitled to not using it.

sergiotapia•about 1 hour ago

"DO NOT COMPRESS ANY DATA WITH THIS PROGRAM UNLESS YOU ARE PREPARED TO ACCEPT THE POSSIBILITY, HOWEVER SMALL, THAT THE DATA WILL NOT BE RECOVERABLE."

So why would anybody, hobbyist or enterprise, use this? Or does something older like 7zip also have this caveat that I've never experienced.

fmx•about 1 hour ago

This big warning gave me pause, too. Why not have a compression option that automatically checks the data after compression (by decompressing and checking against a hash of the original data), which would reduce the probability of undetected errors to that of a hash collision? i.e. like `7z a` followed by `7z t`, but in one command.

jubilanti•43 minutes ago

Is that all that different than the standard MIT License:

> THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE....

self_awareness•about 2 hours ago

Any relation to Iczelion from the masm32 tutorial?

algorithm314•about 1 hour ago

Bro just decompile agiannis_text

roschdal•about 1 hour ago

Imagine bzip4

sedatk•about 3 hours ago

The latest release is a year ago, the last commit is two months ago, and the build is failing.

The claim “stronger than bzip2” is strange. What does it even mean?

Also, comparing parallel decompression benchmarks with bzip2 instead of pbzip2 seems unfair.

finaard•about 2 hours ago

Depends - if parallel decompression of any bzip3 archive is possible I'd consider it fair, if it requires special flags on archive creation I'd consider it unfair. I didn't see any description about that on that page.

pbzip2 only can do parallel decompression on archives created with pbzip2, otherwise it'll fall back to single thread. There nowadays seems to be lbzip2, though, which claims to be able to add SMP support for standard bzip2 archives.

I'll need to try that next time I'm working with large archives - I learned about the pbzip2 limitations the hard way last time I was shuffling around a few multi-10GB archives, and was trying to speed things up fully utilising my 32 core threadripper.

altairprime•about 2 hours ago

“for fairness, the benchmarks have been performed using single thread mode” (2025) https://news.ycombinator.com/item?id=42902241

kosolam•about 3 hours ago

Impressive compression benchmark. Four times smaller than z standard.

adrian_b•about 3 hours ago

I have not experimented with bzip3 recently, but more than a year ago I have done many tests with it.

Initially I was extremely impressed with it, because in a lot of tests it succeeded to compress hard-to-compress files, like movies, and in many cases it demonstrated a much better compromise between speed and compression ratio than zstd, i.e. depending on the command parameters I could make it either compress better than zstd at similar compression/decompression speed, or compress/decompress faster at a similar compression ratio.

Alas, the initial extremely favorable conclusion was short-lived, because trying later bzip3 on other data files gave worse results than zstd.

So the final conclusion was that the performance of bzip3 was somewhat unpredictable, being highly data dependent. For some files it provided outstanding compression ratio or speed, but for others it was inferior.

The problem was that without doing a compression there was no way to guess whether a file would be among those preferred by bzip3 or by zstd or by xz.

So now I would use it only for a file for which I want maximum compression and which I would compress once and decompress many times, so I can afford a very long compression time, during which I would test multiple compression algorithms, including bzip3 and zstd, with multiple parameter choices, and I would eventually choose the one that offers the best compromise between compression ratio and decompression time, for that particular file.

It certainly is a competitive compression algorithm, but unless it has changed since I last tested it, you cannot guess for which files it would win the compression competition.

m000•about 2 hours ago

Would a multi-stream archive format make sense at this point? I.e. store several compressed streams in the same file and use heuristics to decide where each file (or portion of file) goes.

adrian_b•about 2 hours ago

I think so.

But developing the heuristics for choosing the appropriate compression algorithm for a stream of data is likely to need a very long time for compression tests of a lot of diverse training data, similarly to the training of a specialized ML model that classifies patterns.

Such heuristics should provide not only algorithm selection, but also parameter selection, when given only some simple input, e.g. the relative importances of compression ratio, decompression speed and compression speed.

dist-epoch•about 3 hours ago

with zstd at level 16 with default params (dict size, ...). Serious compression starts at level 19 and with much higher dict sizes.

how is this an honest benchmark:

    bzip3 ... 12178M memory
    zstd  ...   687M memory

myrmidon•about 3 hours ago

There is a comparison with "zstd -19" on the Silesia corpus, showing better compression ratio for bzip3 (47.2 vs 53MB) while being ~5 times faster (and using only half the memory).

Even if the examples are highly cherry-picked, it is quite suprising to me that such pareto-dominance is possible at all.

edit: Tested it myself and found that it often also does slightly worse than zstd -19 in compression ratio but faster (it was slower in one case on "uncompressible" input).

Compression performance vs "zstd -19" seems to depends a lot on actual input data in a very unpredictable way. I'd assume the benchmarks that they show are definitely somewhat cherry-picked.

dist-epoch•about 3 hours ago

having used zstd, it has terrible defaults, optimized for speed and low-memory. You need to change it's params (not just level and dict size) to get high performance.

probably somebody should use a coding agent to do auto-research to optimize params for each compression algo, while matching one fixed goal - time, memory or size

eis•about 3 hours ago

The benchmark is very rudimentary. It does not test different levels/settings apart from its own -b 256/512 (does it affect decompression?), it doesn't measure compression time and memory usage. It does not specify parallel vs single-threaded (it mentions parallel on the one decoding number but what about the others?).

The lrzip test is interesting but it omits for example zstd and doesn't even have (de-)compression timings.

A lot more numbers are needed to present a fair and informative comparison.

I don't want this to be a swipe against bzip3, I only want to point out the presented benchmarks could be a lot better.

sylware•about 3 hours ago

Isn't that XZ?

adrian_b•about 3 hours ago

No. It is an unrelated algorithm.

vintermann•about 2 hours ago

It's a very strange algorithm completely different from most compression methods, that's what makes it interesting IMO. But it's probably not realistically competitive, since one needs to do more after the burrows-wheeler transform, and all that "more" has been ridiculously more optimized in zstd and other modern compression methods. Compressing bwt-transformed data is easier, but that doesn't mean it's easy to further than what's easy.

BoingBoomTschak•about 2 hours ago

You're probably thinking of LZMA2.