docker export is missing ownership information; chown does not work

[Artem Leshchev]

[ Impact ]
Tar archive created by docker export command is missing ownership information (all files are owned by root). If this archive is then used to recreate filesystem for unprivileged processes (like via docker import or just unpacking it and chrooting), they can fail with permission denied error or in some other way.

This bug happens when package is built with Go older than 1.19: this version of Go introduced build tag unix that is used by upstream to determine if it should add UNIX-specific attributes into archive. Older Go versions silently ignore this source code, and the result is missing UIDs and GIDs in tar archives. As Go 1.20 was backported to affected releases, patches attached use this version to fix the bug.

[ Test Plan ]
Run next commands (under user with docker group or under root):

    docker run --name lp_2029523 ubuntu:20.04 ls -l /etc/shadow
    docker export lp_2029523 | tar tv etc/shadow

First command will show that file is owned by root:shadow, while second will show that it is owned by UID/GID 0/0. Here you can downgrade or apply the fix to see that second command starts to show UID/GID 0/42 that is correct.

[ Where problems could occur ]
Upstream builds this Docker version with Go 1.19. While Go is usually pretty good at maintaining backward compatibility, there were some subtle changes in Go 1.20, like handling of TLS handshake failure and some other error handling. So there could be changes in behavior when error arrises.

[ Other Info ]
Bug introduced in 20.10.25-0ubuntu1~20.04.1, previous version 20.10.21-0ubuntu1~20.04.2 works as intended. Jammy is also affected. Newer releases are not affected as they already use Go 1.20.

The issue is caused by this change https://github.com/moby/moby/commit/721358e0cb7c10b41508f3c114baeceb2ac9ba0a#diff-12919f88ca9c04e478a6ffdf37e9a67ccdd2997afdc2e51adb1e67c53dcdbd8cL5-R5 and by packaging using Go 1.18. It can be fixed by appending this tag to DOCKER_BUILDTAGS, but building package with newer Go version is even better.

ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: docker.io 20.10.25-0ubuntu1~20.04.1
ProcVersionSignature: Ubuntu 5.15.0-72.79~20.04.1-generic 5.15.98
Uname: Linux 5.15.0-72-generic x86_64
ApportVersion: 2.20.11-0ubuntu27.27
Architecture: amd64
CasperMD5CheckResult: skip
Date: Thu Aug 3 20:27:42 2023
InstallationDate: Installed on 2023-05-19 (76 days ago)
InstallationMedia: Ubuntu 20.04.6 LTS "Focal Fossa" - Release amd64 (20230316)
SourcePackage: docker.io-app
UpgradeStatus: No upgrade log present (probably fresh install)

[Artem Leshchev (matshch)]

This is a debdiff with suggested unix build tag patch applicable to 20.10.25-0ubuntu1~20.04.1. I built this in pbuilder and it builds successfully, and I installed it, the patch works as intended.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "1-20.10.25-0ubuntu1~20.04.2.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Michael Hudson-Doyle (mwhudson)]

I think this will need to be fixed in Jammy too?

[Shengjing Zhu (zhsj)]

unix tag is added in golang 1.19. as golang 1.20 has been backported, so please build with golang-1.20 instead of explicitly setting unix tag.

[Artem Leshchev (matshch)]

Oh, I haven't noticed Go 1.20 is available in Focal now. This is a debdiff with using Go 1.20 for build, it builds successfully in pbuilder and works as intended.

[Artem Leshchev (matshch)]

You are right, jammy is also affected. Here is a debdiff for jammy with using Go 1.20 for build, it builds successfully in pbuilder and works as intended.

[Simon Quigley (tsimonq2)]

The packaging looks reasonable enough, and ready to sponsor.

Could you please update the bug description to match the template in https://wiki.ubuntu.com/StableReleaseUpdates so we can progress with this update? The uploads will surely be rejected without it.

Thank you!

[Artem Leshchev (matshch)]

I've updated the bug description, I hope this is enough.

I'm not quite sure what should I do next, like changing status, assignee, making pull requests or uploading deb packages. I could use some pointers to speed up the process if anything else is needed from my side.

[Lucas Kanashiro (lucaskanashiro)]

Hi Arthem. Could you please send PRs to the Github repo?

https://github.com/canonical/docker.io-app

[Artem Leshchev (matshch)]

Sure, here they are:
* https://github.com/canonical/docker.io-app/pull/3
* https://github.com/canonical/docker.io-app/pull/4

[Artem Leshchev (matshch)]

These patches also fix https://bugs.launchpad.net/ubuntu/+source/docker.io-app/+bug/2029893, I've updated PRs to reflect that.

[Dave Jones (waveform)]

This looks good to go (and thank you for a well done SRU template -- I do like it when the test plan simply includes the commands necessary!); sponsoring

[Lucas Kanashiro (lucaskanashiro)]

@Artem thanks for the PRs, just merged them since Dave already sponsored the uploads for you.

@Dave thanks for the uploads :)

[Andreas Hasenack (ahasenack)]

Hello Artem, or anyone else affected,

Accepted docker.io-app into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/docker.io-app/20.10.25-0ubuntu1~22.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Andreas Hasenack (ahasenack)]

Hello Artem, or anyone else affected,

Accepted docker.io-app into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/docker.io-app/20.10.25-0ubuntu1~20.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Johannes Lehtinen (johannes-lehtinen-h)]

Thank you! The proposed version fixes the permission issue for our Docker build.

See below the results from three different docker.io versions.

$ dpkg -l docker.io | tail -1
hi docker.io 20.10.21-0ubuntu1~22.04.3 amd64 Linux container runtime
...build test-image...
$ docker create --name test-container test-image
$ docker export test-container | tar tv | grep install-template-files
-rwxrwxr-x 1000/1000 1080 2023-08-04 14:16 mnt/liferay/scripts/install-template-files

$ dpkg -l docker.io | tail -1
ii docker.io 20.10.25-0ubuntu1~22.04.1 amd64 Linux container runtime
...build test-image...
$ docker create --name test-container2 test-image
$ docker export test-container2 | tar tv | grep install-template-files
-rwxrwxr-x 0/0 1080 2023-08-04 14:16 mnt/liferay/scripts/install-template-files

$ dpkg -l docker.io | tail -1
ii docker.io 20.10.25-0ubuntu1~22.04.2 amd64 Linux container runtime
$ docker create --name test-container3 test-image
$ docker export test-container3 | tar tv | grep install-template-files
-rwxrwxr-x 1000/1000 1080 2023-08-04 14:16 mnt/liferay/scripts/install-template-files

[Christian (cboitel)]

Also fixed my issue reported in https://bugs.launchpad.net/ubuntu/+source/docker.io/+bug/2029564 (duplicate of this one).

Newer output:
```
$ sudo apt list docker.io
Listing... Done
docker.io/jammy-proposed,now 20.10.25-0ubuntu1~22.04.2 amd64 [installed]

$ docker version
Client:
 Version: 20.10.25
 API version: 1.41
 Go version: go1.20.3
 Git commit: 20.10.25-0ubuntu1~22.04.2
 Built: Thu Aug 10 20:13:18 2023
 OS/Arch: linux/amd64
 Context: default
 Experimental: true

Server:
 Engine:
  Version: 20.10.25
  API version: 1.41 (minimum version 1.12)
  Go version: go1.20.3
  Git commit: 20.10.25-0ubuntu1~22.04.2
  Built: Fri Aug 4 09:20:46 2023
  OS/Arch: linux/amd64
  Experimental: false
 containerd:
  Version: 1.7.2
  GitCommit:
 runc:
  Version: 1.1.7-0ubuntu1~22.04.1
  GitCommit:
 docker-init:
  Version: 0.19.0
  GitCommit:

$ docker build -t test .
Sending build context to Docker daemon 3.584kB
Step 1/7 : FROM ubuntu:jammy
jammy: Pulling from library/ubuntu
3153aa388d02: Pull complete
Digest: sha256:0bced47fffa3361afa981854fcabcd4577cd43cebbb808cea2b1f33a3dd7f508
Status: Downloaded newer image for ubuntu:jammy
 ---> 5a81c4b8502e
Step 2/7 : COPY --chown=root:daemon data/ /data/
 ---> 1f60fedf144a
Step 3/7 : COPY --chown=nobody:nogroup nobody/ /nobody/
 ---> 1088d7e9f9aa
Step 4/7 : RUN find /data /nobody -ls
 ---> Running in 3f0f10dab086
 12107684 0 drwxr-xr-x 2 root daemon 6 Aug 11 10:10 /data
  6511548 0 drwxr-xr-x 2 nobody nogroup 6 Aug 11 10:10 /nobody
Removing intermediate container 3f0f10dab086
 ---> cf63173c347a
Step 5/7 : FROM ubuntu:jammy
 ---> 5a81c4b8502e
Step 6/7 : RUN mkdir -p /data /nobody && chown -Rhv root:daemon /data && chown -Rhv nobody:nogroup /nobody && find /data /nobody -ls
 ---> Running in 321507a49750
changed ownership of '/data' from root:root to root:daemon
changed ownership of '/nobody' from root:root to nobody:nogroup
  4597186 0 drwxr-xr-x 2 root daemon 6 Aug 11 10:10 /data
 10023529 0 drwxr-xr-x 2 nobody nogroup 6 Aug 11 10:10 /nobody
Removing intermediate container 321507a49750
 ---> f711a270c6fe
Step 7/7 : RUN find /data /nobody -ls
 ---> Running in 6fdbfa6feca8
 11373473 0 drwxr-xr-x 2 root daemon 6 Aug 11 10:10 /data
 13619412 0 drwxr-xr-x 2 nobody nogroup 6 Aug 11 10:10 /nobody
Removing intermediate container 6fdbfa6feca8
 ---> b6e280ccf13b
Successfully built b6e280ccf13b
Successfully tagged test:latest
```

[Dave Jones (waveform)]

I've verified the jammy-proposed version locally (and set the tag accordingly). Could Christian or Johannes confirm they've verified the fix on focal? (would save me setting up a focal VM for testing)

[Artem Leshchev (matshch)]

I've done testing of docker.io=20.10.25-0ubuntu1~20.04.2 from focal-proposed, the bug is fixed.

$ dpkg -l docker.io | tail -1
ii docker.io 20.10.25-0ubuntu1~20.04.1 amd64 Linux container runtime
$ docker run --name lp_2029523 ubuntu:20.04 ls -l /etc/shadow
-rw-r----- 1 root shadow 501 Aug 1 02:04 /etc/shadow
$ docker export lp_2029523 | tar tv etc/shadow
-rw-r----- 0/0 501 2023-08-01 05:04 etc/shadow
$ sudo apt install docker.io=20.10.25-0ubuntu1~20.04.2
...
$ dpkg -l docker.io | tail -1
ii docker.io 20.10.25-0ubuntu1~20.04.2 amd64 Linux container runtime
$ docker export lp_2029523 | tar tv etc/shadow
-rw-r----- 0/42 501 2023-08-01 05:04 etc/shadow

[Chris Halse Rogers (raof)]

Marking as fix released in Ubuntu, as this is fixed in Mantic

[Launchpad Janitor (janitor)]

This bug was fixed in the package docker.io-app - 20.10.25-0ubuntu1~20.04.2

---------------
docker.io-app (20.10.25-0ubuntu1~20.04.2) focal; urgency=medium

  * Fix docker export ownership info (LP: #2029523)
    - Build with Go 1.20
      + d/control: b-d on golang-1.20-go intead of golang-1.18-go
      + d/rules: add Go 1.20 to $PATH

 -- Artem Leshchev <email address hidden> Thu, 03 Aug 2023 21:03:37 +0300

[Chris Halse Rogers (raof)]

The verification of the Stable Release Update for docker.io-app has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package docker.io-app - 20.10.25-0ubuntu1~22.04.2

---------------
docker.io-app (20.10.25-0ubuntu1~22.04.2) jammy; urgency=medium

  * Fix docker export ownership info (LP: #2029523)
    - Build with Go 1.20
      + d/control: b-d on golang-1.20-go intead of golang-go
      + d/rules: add Go 1.20 to $PATH

 -- Artem Leshchev <email address hidden> Fri, 04 Aug 2023 09:20:46 +0000