pbzip2 can fail to extract large bz2 files compressed using the traditional single threaded bzip2 tool

[VF]

Binary package hint: pbzip2

The pbzip2 manual says:

"If you have a large file that was created with bzip2 (say 1.5GB for example) you will likely not be able to decompress the file with pbzip2 since pbzip2 will try to allocate 1.5GB of memory to decompress it, and that call might fail depending on your system resources. If the same 1.5GB file had of been compressed with pbzip2, it would decompress fine with pbzip2. If you are unable to decompress a file with pbzip2 due to its size, use the regular bzip2 instead."

You can easily run into such a problem on a memory limited VPS (with multiple cores) if you want to decompress a large backup file there.

Why this problem is important:

I think it is pretty unreasonable that a decompression tool can fail because of the above reason. This kind of failure is largely unexpected by administrators and could cause serious problems in automated scripts, which could result in a data loss if left unnoticed. It also means excess memory usage while trying to decompress a file of a size more than the amount of available physical memory, which could result in severely reduced performance caused by excessive swapping.

Suggested solution:

Above a given "chunk" size pbzip2 should not try to allocate memory, just decompress in single threaded mode without trying to load the whole compressed data stream into memory. Maybe it should fall back to this legacy mode in any low-memory condition automatically, since it is still better to decompress the data in single threaded mode than failing or slowing down due to too much memory allocation.

[Jeff Gilchrist (jeff-gilchrist)]

This description is outdated and was fixed in pbzip2 1.0.5. There is now a just release 1.1.0 that should work even better:
https://bugs.launchpad.net/ubuntu/+source/pbzip2/+bug/538529

[Felix Geyer (debfx)]

Is this still an issue in lucid?

[Launchpad Janitor (janitor)]

[Expired for pbzip2 (Ubuntu) because there has been no activity for 60 days.]