pbzip2 does not see big files

Bug #363793 reported by Francesco Potortì on 2009-04-19
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
pbzip2 (Debian)
Fix Released
Unknown
pbzip2 (Ubuntu)
Wishlist
Unassigned

Bug Description

Binary package hint: pbzip2

Maybe pbzip2 is compiled without big files support?

While it can compress a 600MB file, it claims that a 9GB file does not exist.

Related branches

Tim 'Shaggy' Bielawa (tbielawa) wrote :

I'm also having the same problem with the version in Jaunty.

$ apt-cache policy pbzip2
pbzip2:
  Installed: 1.0.3-1
  Candidate: 1.0.3-1
  Version table:
 *** 1.0.3-1 0
        500 http://mirror.lcsee.wvu.edu jaunty/universe Packages
        100 /var/lib/dpkg/status

However when my office was running Hardy, pbzip2 vesrion 1.0.2-0, we had no issues with large files.

I've grabbed and compiled the 1.0.2-0 version on jaunty with debugging enabled. On a large 14 GB file the 1.0.2-0 version operates correctly.

The reason that pbzip2 isn't working is related to the off_t variable. In hardy it's correctly getting set to 8 while in hardy it's 4 after it's compiled.

Testing a 14GB file against pbzip-1.0.2-0 on Jaunty with PBZIP_DEBUG enabled:

$ pbzip2 -v -t large_file.tar.bz2
argv[1]: -v Len: 2
argv[2]: -t Len: 2
Parallel BZIP2 v1.0.2 - by: Jeff Gilchrist [http://compression.ca]
[July 25, 2007] (uses libbzip2 by Julian Seward)
off_t size: 8 uint size: 4
-------------------------------------------
      File #: 1 of 1
     Testing: large_file.tar.bz2
^C
 *Control-C or similar caught, quitting...
 Infile: (null) Outfile: (null)

Testing a 14GB file against pbzip2-1.0.3-1 on Jaunty with PBZIP_DEBUG enabled:

$ pbzip2 -v -t large_file.tar.bz2
argv[1]: -v Len: 2
argv[2]: -t Len: 2
Parallel BZIP2 v1.0.3 - by: Jeff Gilchrist [http://compression.ca]
[Oct. 31, 2008] (uses libbzip2 by Julian Seward)
off_t size: 4 uint size: 4

 *WARNING: off_t variable size only 32 bits!
 You will only able to compress files smaller than 2GB in size.

-------------------------------------------
      File #: 1 of 1
     Testing: large_file.tar.bz2
 *ERROR: Could not open input file [large_file.tar.bz2]! Skipping...
-------------------------------------------

 Start Time: 1255969853 + 12994
 Stop Time : 1255969853 + 13730

     Wall Clock: 0.000736 seconds

Tim 'Shaggy' Bielawa (tbielawa) wrote :

1.0.5-1 compiled on jaunty does not appear to fix this problem.

$ pbzip2 -v -t large-file.tar.bz2
argv[1]: -v Len: 2
argv[2]: -t Len: 2
Parallel BZIP2 v1.0.5 - by: Jeff Gilchrist [http://compression.ca]
[Jan. 08, 2009] (uses libbzip2 by Julian Seward)
off_t size: 4 uint size: 4

pbzip2: *WARNING: off_t variable size only 32 bits!
 You will only able to compress files smaller than 2GB in size.

-------------------------------------------
      File #: 1 of 1
     Testing: large-file.tar.bz2
pbzip2: *ERROR: Could not open input file [large-file.tar.bz2]! Skipping...
-------------------------------------------

 Start Time: 1255971259 + 218368
 Stop Time : 1255971259 + 219104

     Wall Clock: 0.000736 seconds

Tim 'Shaggy' Bielawa (tbielawa) wrote :

dpkg-buildpackage -us -uc -rfakeroot on 1.0.2:

# Add here commands to compile the package.
/usr/bin/make CFLAGS="-Wall -O2"
make[1]: Entering directory `/tmp/tmp.NgDOicxSBR/pbzip2-1.0.2'
g++ -O3 -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -o pbzip2 pbzip2.cpp -pthread -lpthread -lbz2

dpkg-buildpackage -us -uc -rfakeroot on 1.0.3

# Add here commands to compile the package.
/usr/bin/make CFLAGS="-Wall -O2"
make[1]: Entering directory `/tmp/tmp.ERiGlnnXBD/pbzip2-1.0.3'
g++ -Wall -O2 pbzip2.cpp -o pbzip2 -pthread -lpthread -lbz2

I had to read chapter 6.6 of the GNU Make handbook to figure it out, but the problem is happening then the rules file defines CFLAGS then the Makefile attempts to modify the variable.

Patch attached.

Tim 'Shaggy' Bielawa (tbielawa) wrote :

Linked to debian bug tracker for the same issue. Replied to the debian bug tracker with the patch attached.

Changed in pbzip2 (Debian):
status: Unknown → New
Tim 'Shaggy' Bielawa (tbielawa) wrote :

Now with debdiff!

Changed in pbzip2 (Ubuntu):
status: New → In Progress
importance: Undecided → Wishlist
assignee: nobody → Fabrice Coutadeur (fabricesp)
Tim 'Shaggy' Bielawa (tbielawa) wrote :

Not quite so much wish list as it is regression. Upstream's Makefile includes those CC args by default, when the new release was packaged it didn't account for the changes to the Makefile correctly.

Fabrice Coutadeur (fabricesp) wrote :

Uploaded for Lucid. Thanks for your work!

I had to update the maintainer in debian/control. Please remember to do it next time.

If you want to get it fixed for a previous version, please follow the SRU process (https://wiki.ubuntu.com/StableReleaseUpdates)

thanks,

Fabrice

Changed in pbzip2 (Ubuntu):
assignee: Fabrice Coutadeur (fabricesp) → nobody
status: In Progress → Fix Committed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package pbzip2 - 1.0.5-1ubuntu1

---------------
pbzip2 (1.0.5-1ubuntu1) lucid; urgency=low

  * debian/rules: added support for long files so that where pbzip2 recognize
    large files (LP: #363793)
 -- Timothy Bielawa (Shaggy) <email address hidden> Thu, 10 Dec 2009 22:50:50 -0500

Changed in pbzip2 (Ubuntu):
status: Fix Committed → Fix Released
Changed in pbzip2 (Debian):
status: New → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.