[needs-packaging] Docker cli-plugin docker-buildx needed since docker.io 24.0.5 if using DOCKER_BUILDKIT=1

[Heinrich Schuchardt]

Many current Docker based build recipes are based on the buildx docker-cli plugin.

Please, package the missing file.

See https://github.com/docker/buildx

Building buildx seems to require buildx itself.

For bootstrapping the easiest thing to do seems build with docker inside a docker container using

COPY --from=docker/buildx-bin:latest /buildx /usr/libexec/docker/cli-plugins/docker-buildx

to obtain initial buildx. Cf. https://docs.docker.com/buildx/working-with-buildx/

Best regards

Heinrich

[Brian Murray (brian-murray)]

*** This is an automated message ***

This bug is tagged needs-packaging which identifies it as a request for a new package in Ubuntu. As a part of the managing needs-packaging bug reports specification, https://wiki.ubuntu.com/QATeam/Specs/NeedsPackagingBugs, all needs-packaging bug reports have Wishlist importance. Subsequently, I'm setting this bug's status to Wishlist.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Tux Penguin (g944ya3s)]

The version of docker.io was recently updated to 24.0.5-0ubuntu1 in focal, jammy, etc -updates. With this version new version, if anyone is using the `DOCKER_BUILDKIT=1` flag, then they must have the docker buildx plugin present or else docker will fail.

With this change, this bug should have a higher priority because it will mean that there could be a lot of affected people.

The workaround is to use docker-ce from Docker upstream and not use docker.io

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Shih-Yuan Lee (fourdollars)]

$ rmadison docker-buildx
 docker-buildx | 0.11.2-0ubuntu2 | mantic/universe | source, amd64, arm64, armhf, ppc64el, riscv64, s390x

[willmo (willmo)]

> With this change, this bug should have a higher priority because it will mean that there could be a lot of affected people.

Right, this is a regression in Ubuntu's Docker packaging (due to an upstream change brought in with the update to 24.0.5 in bug 2030500), not a request for new functionality.

[Shih-Yuan Lee (fourdollars)]

$ rmadison docker.io
...
 docker.io | 24.0.5-0ubuntu1~20.04.1 | focal-updates/universe | amd64, arm64, armhf, ppc64el, riscv64, s390x
 docker.io | 24.0.5-0ubuntu1~22.04.1 | jammy-updates/universe | amd64, arm64, armhf, ppc64el, riscv64, s390x
 docker.io | 24.0.5-0ubuntu1~23.04.1 | lunar-updates/universe | amd64, arm64, armhf, ppc64el, riscv64, s390x
 docker.io | 24.0.5-0ubuntu1 | mantic/universe | amd64, arm64, armhf, ppc64el, riscv64, s390x

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Andreas Schildbach (schildbach)]

Yes, this regression (for Jammy, etc) has broken the building of images via Docker completely for me. I'm now back to the old docker.io package and have disabled automatic updates.

[Andreas Hasenack (ahasenack)]

So both in the old and updated version of docker, in order to experiment the new buildsystem[1], you have to pass DOCKER_BUILDKIT=1, correct?

The difference now is that with the updated version of docker, the new buildsystem requires buildx, whereas before it didn't, correct? And docker-buildx is currently only available in mantic.

Does that summarize it?

1. https://docs.docker.com/build/buildkit/

[Andreas Hasenack (ahasenack)]

With docker 20.10.21-0ubuntu3, from lunar release:

Works without warnings:

  $ docker build .

Works without warnings:

  $ DOCKER_BUILDKIT=1 docker build .

With 24.0.5-0ubuntu1~23.04.1 from lunar-updates:

Works with a warning about using a deprecated build system:

  $ docker build .
  DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
              Install the buildx component to build images with BuildKit:
              https://docs.docker.com/go/buildx/

Does not work:

  $ DOCKER_BUILDKIT=1 docker build .
  ERROR: BuildKit is enabled but the buildx component is missing or broken.
         Install the buildx component to build images with BuildKit:
         https://docs.docker.com/go/buildx/

[Andreas Schildbach (schildbach)]

On top of what Andreas said, some command-line options are taken away without buildkit, e.g. the `--output` / `-o` option.

[Andreas Hasenack (ahasenack)]

Releaseing docker-buildx and docker-compose-v2 is depending on this SRU exception update[1] being granted.

1. https://lists.ubuntu.com/archives/ubuntu-release/2023-September/005769.html

[Stephen Warren (srwarren)]

For anyone affected who needs a WAR, this is how to downgrade on 22.04 at least:

wget http://us.archive.ubuntu.com/ubuntu/pool/universe/d/docker.io/docker.io_20.10.21-0ubuntu1~22.04.3_amd64.deb

sudo dpkg -i docker.io_20.10.21-0ubuntu1~22.04.3_amd64.deb

(I don't know if that's the best version to downgrade to, but it's what was installed on one of my machines before I updated and found this issue.)

[Mark Eichin (eichin-gmail)]

If you're getting this from focal-updates, you probably still have the underlying focal-itself source around, and

    $ sudo apt install docker.io=20.10.21-0ubuntu1~20.04.2 --allow-downgrades

should work too. (I'm kind of surprised this happened - per https://wiki.ubuntu.com/Releases this is an LTS...)

(edited for confusing the codenames)

[Andreas Hasenack (ahasenack)]

> (I'm kind of surprised this happened - per https://wiki.ubuntu.com/Releases this is an LTS...)

Docker has an SRU exception, documented here:

  https://wiki.ubuntu.com/DockerUpdates

https://bugs.launchpad.net/ubuntu/+source/docker.io/+bug/1939106/comments/6 has a longer response and options for feedback on this SRU exception.

[Andreas Hasenack (ahasenack)]

I added this bug to the list of recorded regressions[1] in that policy.

1. https://wiki.ubuntu.com/DockerUpdates#Record_of_regressions

[LCID Fire (lcid-fire)]

Would really be interested in a Post-Mortem of this.
You can meanwhile switch to official Docker packages: https://docs.docker.com/engine/install/ubuntu/ to make your dev and CI machines work again.

[Andreas Hasenack (ahasenack)]

The inclusion of docker-buildx and docker-compose-v2 was approved, backports to stable releases will happen shortly.

[Andreas Hasenack (ahasenack)]

docker-buildx exists in mantic, marking that task as fix released.

[Stephen Warren (srwarren)]

The following WAR also works, without downgrading/pinning the docker.io package, since docker-buildx is (a) available in a newer distro release, and (b) statically linked it works on older distro releases than what it was built on (and furthermore doesn't seem to use any syscalls that aren't available on 22.04 either):

wget http://ftp.ubuntu.com/ubuntu/pool/universe/d/docker-buildx/docker-buildx_0.11.2-0ubuntu2_amd64.deb
sudo dpkg -i docker-buildx_0.11.2-0ubuntu2_amd64.deb

Once the real fix is available, you'll probably need to do something to make sure you pick up the package from your distro rather than this manually-installed version. Maybe just use dpkg to rm it, then apt to install the real version.

[Stephen Warren (srwarren)]

> > (I'm kind of surprised this happened - per https://wiki.ubuntu.com/Releases this is an LTS...)

Me too...

> Docker has an SRU exception, documented here:

I find it odd that the stated benefit of that SRU is to make life better for users, whereas in practice is does the exact opposite.

[Lucas Kanashiro (lucaskanashiro)]

Sorry for the inconvenience. The fix for this is just waiting on archive admins and SRU review. For docker related packages we have an exception in place which tell users what is our goal regarding this set of packages:

https://wiki.ubuntu.com/DockerUpdates

FWIW comment #23 presents a good workaround for now, until the fix lands in the main archive.

[Mark Eichin (eichin-gmail)]

Thanks for the fix, and the explanations about the process exemption.

Ironically, since the exemption exists and is clearly documented (yay!), it may lead to us being *more* likely to use the upstream packages (since the exemption means we now know that we have to put them on hold and only pull them in after we've tested them, and that's slightly easier with third-party packages.) I know it's not defined this way, but from our perspective, non-current LTS means "if something changes at all, someone thought it was pretty important". So we'd be counted in the "if we wanted a current docker, we'd be running a current ubuntu for it" camp. (I do understand that docker is specifically distinct - that some users may want a new docker to run new images on an old ubuntu - and now that I know about it, I'm not objecting to the policy.)

To be entirely fair, I will admit that it wouldn't have even come up if the upgrade hadn't broken buildkit - it would have just been "oh, that's a pleasant surprise" and we wouldn't have given it much further thought (I certainly wouldn't have found this thread.) On top of that, it looks like the missing packages are percolating through the system rapidly and we might just be able to sneak docker-buildx into our developer-deps before too many people get caught by it.

Thanks for straightening this out and being transparent about it!

[Steve Langasek (vorlon)]

Hello Heinrich, or anyone else affected,

Accepted docker-buildx into lunar-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/docker-buildx/0.11.2-0ubuntu1~23.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-lunar to verification-done-lunar. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-lunar. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello Heinrich, or anyone else affected,

Accepted docker-buildx into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/docker-buildx/0.11.2-0ubuntu1~22.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello Heinrich, or anyone else affected,

Accepted docker-buildx into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/docker-buildx/0.11.2-0ubuntu1~20.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Stephen Warren (srwarren)]

The jammy-proposed package works for me.

[Lucas Kanashiro (lucaskanashiro)]

@xypron verified the fix in all releases this other bug:

https://bugs.launchpad.net/ubuntu/+source/docker-buildx/+bug/2036350/comments/7
https://bugs.launchpad.net/ubuntu/+source/docker-buildx/+bug/2036350/comments/8
https://bugs.launchpad.net/ubuntu/+source/docker-buildx/+bug/2036350/comments/9

[murzzz (murzzz)]

I have tested yammy-proposed[1] on my use cases and it works.
When it might be available in -updates?

1: https://bugs.launchpad.net/ubuntu/+source/docker-buildx/+bug/1968035/comments/28

[Launchpad Janitor (janitor)]

This bug was fixed in the package docker-buildx - 0.11.2-0ubuntu1~23.04.1

---------------
docker-buildx (0.11.2-0ubuntu1~23.04.1) lunar; urgency=medium

  * Backport version from Mantic (LP: #1968035, #2036350).

 -- Lucas Kanashiro <email address hidden> Sat, 16 Sep 2023 06:39:49 -0300

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for docker-buildx has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package docker-buildx - 0.11.2-0ubuntu1~20.04.1

---------------
docker-buildx (0.11.2-0ubuntu1~20.04.1) focal; urgency=medium

  * Backport version from Mantic (LP: #1968035, #2036350).
    - Build with Go 1.20.
      + d/control: b-d on golang-1.20-go.
      + d/rules: add Go 1.20 to $PATH.

 -- Lucas Kanashiro <email address hidden> Sat, 16 Sep 2023 07:02:36 -0300

[Launchpad Janitor (janitor)]

This bug was fixed in the package docker-buildx - 0.11.2-0ubuntu1~22.04.1

---------------
docker-buildx (0.11.2-0ubuntu1~22.04.1) jammy; urgency=medium

  * Backport version from Mantic (LP: #1968035, #2036350).
    - Build with Go 1.20.
      + d/control: b-d on golang-1.20-go.
      + d/rules: add Go 1.20 to $PATH.

 -- Lucas Kanashiro <email address hidden> Sat, 16 Sep 2023 07:02:36 -0300