[FFe] Please sync c-blosc 1.14.0+ds1-1 from debian unstable to bionic (1.11.1 has format incompat bug!)

[Elvis Stansvik]

We discovered a bug [1] that is present in all version of Blosc from 1.11.0 to 1.13.5 (inclusive). The bug prevents data compressed with these versions using the LZ4, LZ4HC, Zlib and Zstd compressors from being decompressed using any prior versions of Blosc (it's an accidental forward-compatiblity break). So in a sense, these versions should be considered bad apples.

The bug was subsequently fixed [2] and the fix is present in the now released 1.14.0. The upstream author also blogged [3] about this incident.

I alerted the Debian package maintainer (Daniel Stender) about this [4], and he was kind enough to quickly package the bugfixed version [5].

I therefore ask you to please sync this bugfixed version to bionic, since it would be unfortunate if it was released with one of the "bad" versions.

Many thanks in advance!

[1] https://github.com/Blosc/c-blosc/issues/215
[2] https://github.com/Blosc/c-blosc/pull/216
[3] http://blosc.org/posts/new-forward-compat-policy/
[4] https://salsa.debian.org/debian/c-blosc/merge_requests/1
[5] https://packages.debian.org/source/sid/c-blosc

[Adam Conrad (adconrad)]

Approved and synced.

[Elvis Stansvik (elvstone)]

@adconrad Many thanks for being so quick with this. I've tested the package in -proposed:

root@194d7a53ffb1:/usr/share/doc/libblosc-dev/examples# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu Bionic Beaver (development branch)"
root@194d7a53ffb1:/usr/share/doc/libblosc-dev/examples# apt-cache show libblosc-dev
Package: libblosc-dev
Architecture: amd64
Version: 1.14.0+ds1-1
Priority: optional
Section: universe/libdevel
Source: c-blosc
Origin: Ubuntu
Maintainer: Ubuntu Developers <email address hidden>
Original-Maintainer: Daniel Stender <email address hidden>
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Installed-Size: 156
Depends: libblosc1 (= 1.14.0+ds1-1)
Filename: pool/universe/c/c-blosc/libblosc-dev_1.14.0+ds1-1_amd64.deb
Size: 36672
MD5sum: 9a144274a54ce28f53ba0ea3fca70d13
SHA1: 4fa0a4d01b28bf33d2dbcf070e3e423bac9e6a5f
SHA256: 759f694504c2a72d7ca5461ef9f4e4a9db635aabf3c941b981aa9aa266d852d3
Homepage: http://blosc.org/
Description: high performance meta-compressor optimized for binary data (development files)
Description-md5: d4cb8f384fede65d372e621a91fbb94e

Package: libblosc-dev
Architecture: amd64
Version: 1.11.1+ds2-3
Priority: optional
Section: universe/libdevel
Source: c-blosc
Origin: Ubuntu
Maintainer: Ubuntu Developers <email address hidden>
Original-Maintainer: Daniel Stender <email address hidden>
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Installed-Size: 148
Depends: libblosc1 (= 1.11.1+ds2-3)
Filename: pool/universe/c/c-blosc/libblosc-dev_1.11.1+ds2-3_amd64.deb
Size: 35068
MD5sum: 6f2de0ec8e1542041939e65e636403ad
SHA1: 8ef571cfcbf195d29f4ac9f7be7b372c21c00f22
SHA256: 1e1d1193fdfc7e9ca5b7a1d47907d2c36ac528ba8c6dbbd4ef99b5c1135c0a5c
Homepage: http://blosc.org/
Description: high performance meta-compressor optimized for binary data (development files)
Description-md5: d4cb8f384fede65d372e621a91fbb94e

root@194d7a53ffb1:/usr/share/doc/libblosc-dev/examples# for f in many_compressors.c multithread.c noinit.c simple.c; do gcc $f.out $f -lblosc && ./$f.out; done
Blosc version info: 1.14.0 ($Date:: 2018-02-23 #$)
Using 4 threads (previously using 1)
Using blosclz compressor
Compression: 4000000 -> 41384 (96.7x)
Succesful roundtrip!
Using lz4 compressor
Compression: 4000000 -> 51510 (77.7x)
Succesful roundtrip!
Using lz4hc compressor
Compression: 4000000 -> 38314 (104.4x)
Succesful roundtrip!
Using snappy compressor
Compression: 4000000 -> 208526 (19.2x)
Succesful roundtrip!
Using zlib compressor
Compression: 4000000 -> 22103 (181.0x)
Succesful roundtrip!
Using zstd compressor
Compression: 4000000 -> 10908 (366.7x)
Succesful roundtrip!
Blosc version info: 1.14.0 ($Date:: 2018-02-23 #$)
Using 1 threads (previously using 1)
Compression: 4000000 -> 41384 (96.7x)
Succesful roundtrip!
Using 2 threads (previously using 1)
Compression: 4000000 -> 41384 (96.7x)
Succesful roundtrip!
Using 3 threads (previously using 2)
Compression: 4000000 -> 41384 (96.7x)
Succesful roundtrip!
Using 4 threads (previously using 3)
Compression: 4000000 -> 41384 (96.7x)
Succesful roundtrip!
Blosc version info: 1.14.0 ($Date:: 2018-02-23 #$)
Compression: 40...

[Elvis Stansvik (elvstone)]

Sorry, of course meant to run `apt-cache policy`, not `apt-cache show`. Here it is:

root@f0b7772b23f5:/# apt-cache policy libblosc-dev
libblosc-dev:
  Installed: (none)
  Candidate: 1.14.0+ds1-1
  Version table:
     1.14.0+ds1-1 500
        500 http://archive.ubuntu.com/ubuntu bionic-proposed/universe amd64 Packages
     1.11.1+ds2-3 500
        500 http://archive.ubuntu.com/ubuntu bionic/universe amd64 Packages
root@f0b7772b23f5:/#

[Elvis Stansvik (elvstone)]

I've now also tested 1.14.2+ds1-1 which finally landed in -proposed (the 1.14.0+ds1-1 build failed because of an alignment issue on ARM that was subsequently fixed).

The package is looking good (transcript of testing below).

root@f0b7772b23f5:/usr/share/doc/libblosc-dev/examples# apt-cache policy libblosc*
libblosc-dev:
  Installed: 1.14.2+ds1-1
  Candidate: 1.14.2+ds1-1
  Version table:
 *** 1.14.2+ds1-1 500
        500 http://archive.ubuntu.com/ubuntu bionic-proposed/universe amd64 Packages
        100 /var/lib/dpkg/status
     1.11.1+ds2-3 500
        500 http://archive.ubuntu.com/ubuntu bionic/universe amd64 Packages
libblosc1:
  Installed: 1.14.2+ds1-1
  Candidate: 1.14.2+ds1-1
  Version table:
 *** 1.14.2+ds1-1 500
        500 http://archive.ubuntu.com/ubuntu bionic-proposed/universe amd64 Packages
        100 /var/lib/dpkg/status
     1.11.1+ds2-3 500
        500 http://archive.ubuntu.com/ubuntu bionic/universe amd64 Packages
root@f0b7772b23f5:/usr/share/doc/libblosc-dev/examples# for f in many_compressors.c multithread.c noinit.c simple.c; do gcc -o $f.out $f -lblosc && ./$f.out; done
Blosc version info: 1.14.2 ($Date:: 2018-03-16 #$)
Using 4 threads (previously using 1)
Using blosclz compressor
Compression: 4000000 -> 41384 (96.7x)
Succesful roundtrip!
Using lz4 compressor
Compression: 4000000 -> 51510 (77.7x)
Succesful roundtrip!
Using lz4hc compressor
Compression: 4000000 -> 38314 (104.4x)
Succesful roundtrip!
Using snappy compressor
Compression: 4000000 -> 208526 (19.2x)
Succesful roundtrip!
Using zlib compressor
Compression: 4000000 -> 22103 (181.0x)
Succesful roundtrip!
Using zstd compressor
Compression: 4000000 -> 10908 (366.7x)
Succesful roundtrip!
Blosc version info: 1.14.2 ($Date:: 2018-03-16 #$)
Using 1 threads (previously using 1)
Compression: 4000000 -> 41384 (96.7x)
Succesful roundtrip!
Using 2 threads (previously using 1)
Compression: 4000000 -> 41384 (96.7x)
Succesful roundtrip!
Using 3 threads (previously using 2)
Compression: 4000000 -> 41384 (96.7x)
Succesful roundtrip!
Using 4 threads (previously using 3)
Compression: 4000000 -> 41384 (96.7x)
Succesful roundtrip!
Blosc version info: 1.14.2 ($Date:: 2018-03-16 #$)
Compression: 4000000 -> 41384 (96.7x)
Decompression succesful!
Succesful roundtrip!
Blosc version info: 1.14.2 ($Date:: 2018-03-16 #$)
Compression: 4000000 -> 41384 (96.7x)
Decompression succesful!
Succesful roundtrip!

And also with regard to the original format incompatibility bug that prompted this report:

root@f0b7772b23f5:/usr/share/doc/libblosc-dev/examples# gcc -o test test.c -lblosc
root@f0b7772b23f5:/usr/share/doc/libblosc-dev/examples# ./test compress lz4 test.blosc
Blosc version info: 1.14.2 ($Date:: 2018-03-16 #$)
Compression: 4000000 -> 51420 (77.8x)
Wrote test.blosc
root@f0b7772b23f5:/usr/share/doc/libblosc-dev/examples# exit
exit
[estan@newton insight (improve-loading $%>)]$ cd
[estan@newton ~]$ docker cp f0b7772b23f5:/usr/share/doc/libblosc-dev/examples/test.c .
[estan@newton ~]$ docker cp f0b7772b23f5:/usr/share/doc/libblosc-dev/examples/test.blosc .
[estan@newton ~]$ gcc -o test test.c -lblosc
[estan@newton ~]$ ./test decompress lz4 test.blosc
Blosc version info: 1.7.0 ($Date:: 2015-...

[Graham Inggs (ginggs)]

c-blosc is not able to migrate from -proposed as the new version causes bcolz's autopkgtests to fail:
https://people.canonical.com/~ubuntu-archive/proposed-migration/update_excuses.html#c-blosc

Test failure occurs in Debian too, starting with the upload of c-blosc 1.14.0+ds1-1:
https://ci.debian.net/packages/b/bcolz/unstable/amd64/

Does bcolz need an update to work with the new libblosc?

[Elvis Stansvik (elvstone)]

@ginggs: Thanks for the heads up. I could reproduce the problem with bcolz from the Git master branch, and I've filed an upstream issue here: https://github.com/Blosc/bcolz/issues/374 (the upstream author for both packages are the same, so hopefully it's an easy fix).