clonezilla-live-20110618-natty - pbzip2 fails with "ERROR: Could not allocate memory (FileData)! Aborting..." while compressing

[Telligent]

pbzip2 version: 1.1.4
OS: live image - Ubuntu Natty repository (as of 2011/Jun/18).
Type of processor: Intel Xeon CPU E5420 @ 2.50 GHz (8 core)

I am using clonezilla-live-20110618-natty which has just been updated to use pbzip2 1.1.4. When I create an image using pbzip2 for the compression, pbzip2 fails with "ERROR: Could not allocate memory (FileData)! Aborting...". The same operation completes successfully using bzip2.

The command being run is:
partclone.ext3 -L /var/log/partclone.log -c -s /dev/sda1 --output - | pbzip2 -c -b1024 -3 | ...

[Yavor Nikolov (yavor-nikolov)]

@Telligent, thanks for reporting this!

Seems you're running out of memory.

Could you remove the -b parameter (i.e. use the default):
-b# specifies block size in 100k steps (default 9 = 900k)

pbzip2 allocates some internal buffers which depend on that block size, number of logical processors and some other parameters: -m#, -S#, -p#.

By default you may expect that enough memory should be needed to buffer 3* #CPUs number of blocks.

In addition - depending on OS&compiler, full stack size may need to be allocated up-front for each thread (default size could be too big so that's where -S param could be used for fine tuning. OS defautls could be seen/changed using ulimit command /with some params/).
---

Anyway: as a first step - just remove the -b# param (it seems too big).

[Jeff Gilchrist (jeff-gilchrist)]

How much physical RAM do you have on that system and how much is typically free when you start this operation?

With -b1024 pbzip2 is allocated 102400 KB of RAM for each block so with 8 CPUs it is likely using at least 2.4GB of RAM, possibly more depending on how fast it can flush the output buffer to disk.

Also, is there are particular reason you are using -3 to set the internal BWT block size to 300k? Have you found that this provides the best compression for the data on your partition? Generally I have found that using the default 900k (-9) block size gives better compression and better load balancing between threads but in certain cases other sizes will perform better. Is this a Clonezilla default setting or something that you found works best?

As Yavor mentioned, the best thing to try first is removing the -b#1024 parameter. Thanks again for reporting the issue.

[Telligent (techadmin)]

Thank you both for the quick response. You are indeed correct, the server was running out of memory. Since the kernel in use was 32bit, the available memory before starting pbzip2 was around 3 GB but it quickly got used up.

The settings used are the default ones set by clonezilla (which I did not create, their site is http://clonezilla.org). I will create a bug report with them with a reference to this bug report and they can adjust their settings as needed.

You may mark this bug as resolved.