Backport zstd support in old branch 20.10

[Romain Geissler]

Hi,

It seems the Docler/Moby team is about to release Docker 22.04 (they have recently updated the milestone name from 21.x). This release will bring among many other things, the support for zstd compressed images.

Zstd support is something we would like to leverage in our company, as we are using more and more podman as an alternative to docker. Podman as built, on top of regular zstd compression, some cool optimizations which are heavily inspired by the estargz format, and they named it "zstd:chunked". See more documentation about what Podman folks did here: https://www.redhat.com/sysadmin/faster-container-image-pulls

The thing is, while we use RHEL 8 + podman one side, we also have a large user base that are using Ubuntu too, and most of them still use docker, despite podman starts to be available in Ubuntu 22.04. As Docker is still the de-facto standard for many projects/users, it would be cool if at least the Docker daemon packaged by Ubuntu could support zstd compression, so we could start using zstd:chunked in our images (the speed up of partial pulls would only work with podman, but at least Docker daemons would be able to pull the images, like before).

So, would it be possible to backport the zstd support in the old Docker 20.10 branch in the Ubuntu packages ? I have submitted this backport upstream already in https://github.com/moby/moby/pull/43331 yet upstream maintainers of Docker aren't so keen on merging this, despite it's not really invasive and doesn't seem quite risky. So, would it possible to do it in the Ubuntu packaging instead ?

An alternative to backporting this would be to wait till this new Docker 22.04 branch is officially released, and move the Ubuntu packages to this release, but I am unsure you want to push that big changes to things like Ubuntu 20.04 and 22.04.

Cheers,
Romain

[Romain Geissler (rgeissler-1a)]

Note: you can test if zstd support works (or not) by trying to use this zstd:chunked image produced by the zstd:chunked developer in the podman team:

docker run --rm --pull=always gscrivano/zstd-chunked:fedora whoami

Which for now fails with:

fedora: Pulling from gscrivano/zstd-chunked
9970d86e7cb7: Extracting [==================================================>] 60.87MB/60.87MB
docker: failed to register layer: Error processing tar file(exit status 1): archive/tar: invalid tar header.
See 'docker run --help'.

[Romain Geissler (rgeissler-1a)]

Hi,

Given that docker 23 (and even 24) is out now, instead of patching, we shall rather upgrade to this released version. So closing this (and opening a new bug for the upgrade to Docker 24).

Cheers,
Romain