Invalid free while running docker build

[Sitsofe Wheeler]

[Impact]

The bug results in an error being printed on the user screen when running docker builds.

While the error will not cause the build to fail, the bug does prevent users to login on servers with no X11 installed.

[Test Plan]

- Install docker.io and golang-docker-credential-helpers
- Run a simple docker build
- Verify that the error occurs
- Install the proposed fix
- Run a simple docker build
- Verify the error is no longer present

- alternatively, try to run "docker login" and provide a correct username and password. By the end of the login process, an error will occur and the login will not succeed.

```
apt-get update
apt-get install -y docker.io golang-docker-credential-helpers
mkdir /tmp/dummy-docker-build
cd /tmp/dummy-docker-build
cat <<EOF > Dockerfile
FROM ubuntu:focal
LABEL hello=label
EOF
docker build -t golang-docker-credential-helpers:test .
# Verify the error is triggered.
# update the package and try the build again with
# docker build -t golang-docker-credential-helpers:test
# from this directory
```

[Where problems could occur]

While the patch being applied is a simple, straightforward change, performing a rebuild of this package in bionic may pull newer versions of build dependencies which may cause unpredictable behavior in a stable package.

[Other Info]

This fix is already present upstream, in Debian, and in all following Ubuntu series.

[Original message]

Description of the problem:
Running docker build generates a warning that an invalid free was performed.

Steps to reproduce:
sudo apt-get install docker-compose
sudo docker build -f Dockerfile.build https://github.com/docker-library/hello-world.git

Expected results:
Docker container to be built without errors

Actual results:
An error is produced while the container is being built:

free(): invalid pointer
SIGABRT: abort
PC=0x7f7027c42e97 m=0 sigcode=18446744073709551610
signal arrived during cgo execution

goroutine 1 [syscall, locked to thread]:
runtime.cgocall(0x4afd50, 0xc420073cc0, 0xc420073ce8)
 /usr/lib/go-1.8/src/runtime/cgocall.go:131 +0xe2 fp=0xc420073c90 sp=0xc420073c50
github.com/docker/docker-credential-helpers/secretservice._Cfunc_free(0x12c4da0)
 github.com/docker/docker-credential-helpers/secretservice/_obj/_cgo_gotypes.go:111 +0x41 fp=0xc420073cc0 sp=0xc420073c90
github.com/docker/docker-credential-helpers/secretservice.Secretservice.List.func5(0x12c4da0)
 /build/golang-github-docker-docker-credential-helpers-cMhSy1/golang-github-docker-docker-credential-helpers-0.5.0/obj-x86_64-linux-gnu/src/github.com/docker/docker-credential-helpers/secretservice/secretservice_linux.go:96 +0x60 fp=0xc420073cf8 sp=0xc420073cc0
github.com/docker/docker-credential-helpers/secretservice.Secretservice.List(0x0, 0x756060, 0xc420014370)
 /build/golang-github-docker-docker-credential-helpers-cMhSy1/golang-github-docker-docker-credential-helpers-0.5.0/obj-x86_64-linux-gnu/src/github.com/docker/docker-credential-helpers/secretservice/secretservice_linux.go:97 +0x217 fp=0xc420073da0 sp=0xc420073cf8
github.com/docker/docker-credential-helpers/secretservice.(*Secretservice).List(0x77e548, 0xc420073e88, 0x410022, 0xc4200142d0)
 <autogenerated>:4 +0x46 fp=0xc420073de0 sp=0xc420073da0
github.com/docker/docker-credential-helpers/credentials.List(0x756ba0, 0x77e548, 0x7560e0, 0xc42000e018, 0x0, 0x10)
 /build/golang-github-docker-docker-credential-helpers-cMhSy1/golang-github-docker-docker-credential-helpers-0.5.0/obj-x86_64-linux-gnu/src/github.com/docker/docker-credential-helpers/credentials/credentials.go:145 +0x3e fp=0xc420073e68 sp=0xc420073de0
github.com/docker/docker-credential-helpers/credentials.HandleCommand(0x756ba0, 0x77e548, 0x7fff8fa9e879, 0x4, 0x7560a0, 0xc42000e010, 0x7560e0, 0xc42000e018, 0x40e398, 0x4d35c0)
 /build/golang-github-docker-docker-credential-helpers-cMhSy1/golang-github-docker-docker-credential-helpers-0.5.0/obj-x86_64-linux-gnu/src/github.com/docker/docker-credential-helpers/credentials/credentials.go:60 +0x16d fp=0xc420073ed8 sp=0xc420073e68
github.com/docker/docker-credential-helpers/credentials.Serve(0x756ba0, 0x77e548)
 /build/golang-github-docker-docker-credential-helpers-cMhSy1/golang-github-docker-docker-credential-helpers-0.5.0/obj-x86_64-linux-gnu/src/github.com/docker/docker-credential-helpers/credentials/credentials.go:41 +0x1cb fp=0xc420073f58 sp=0xc420073ed8
main.main()
 /build/golang-github-docker-docker-credential-helpers-cMhSy1/golang-github-docker-docker-credential-helpers-0.5.0/secretservice/cmd/main_linux.go:9 +0x4f fp=0xc420073f88 sp=0xc420073f58
runtime.main()
 /usr/lib/go-1.8/src/runtime/proc.go:185 +0x20a fp=0xc420073fe0 sp=0xc420073f88
runtime.goexit()
 /usr/lib/go-1.8/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420073fe8 sp=0xc420073fe0

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
 /usr/lib/go-1.8/src/runtime/asm_amd64.s:2197 +0x1

rax 0x0
rbx 0x7fff8fa9e310
rcx 0x7f7027c42e97
rdx 0x0
rdi 0x2
rsi 0x7fff8fa9e0a0
rbp 0x7fff8fa9e410
rsp 0x7fff8fa9e0a0
r8 0x0
r9 0x7fff8fa9e0a0
r10 0x8
r11 0x246
r12 0x7fff8fa9e310
r13 0x1000
r14 0x0
r15 0x30
rip 0x7f7027c42e97
rflags 0x246
cs 0x33
fs 0x0
gs 0x0

How reproducible is the problem?
Happens every time.

Version information:
Ubuntu 18.04.1 LTS
docker.io 18.06.1-0ubuntu1~18.04.1
docker-compose 1.17.1-2
golang-docker-credential-helpers 0.5.0-2

Additional information:
It looks like there's a patch which will fix this issue in the docker-credential-helpers project (see the https://github.com/docker/docker-credential-helpers/commit/73e5f5dbfea31ee3b81111ebbf189785fa69731c commit) but it only landed on master on the 19th July 2018 and the last "release" was v0.6.1. The patch looks fairly self contained though so looks like a ready candidate for backporting (and given how widespread this issue is this is the course of action I would recommend).

There are also a lot of issues elsewhere on the web describing this issue:
https://github.com/moby/moby/issues/34048 ("Error in `docker-credential-secretservice': free(): invalid pointer: 0x00000000011b3150" which reports that you need to install gnome-keyring)
https://github.com/docker/for-linux/issues/185 ("Docker-engine on Debian should have `accountsservice` and `gnome-keyring` as dependencies")
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=884596 ("Please make golang-docker-credential-helpers depend on gnome-keyring")
https://github.com/docker/docker-credential-helpers/issues/103 ("strange errors on docker build " where someone says they have both gnome-keyring and accountsservice installed but they still see a problem. https://github.com/docker/docker-credential-helpers/issues/104 and https://github.com/docker/docker-credential-helpers/issues/23 appear to be duplicates).
https://bugs.launchpad.net/ubuntu/+source/docker-compose/+bug/1792824 ("Please remove golang-docker-credential-helpers dependency from the Ubuntu docker-compose package" got the reply "Please don't ask us (Ubuntu) - go and ask Debian")
https://github.com/moby/moby/issues/37916 ("Error on build: double free or corruption (out) SIGABRT: abort PC=0x7f7464f01e97 m=0 sigcode=18446744073709551610 signal arrived during cgo execution", seems to be the same as the problem described in https://github.com/docker/docker-credential-helpers/issues/103 ).

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Anatoly Borodin (anatoly.borodin)]

A fix could be ported, or a package could be upgraded to v0.6.2. See https://github.com/docker/docker-credential-helpers/pull/29/commits/79f93e5e69abc2ead458d19f4577b64b7e35c504

[Sitsofe Wheeler (sitsofe)]

I agree with @anatoly.borodin - in fact the commit https://github.com/docker/docker-credential-helpers/commit/73e5f5dbfea31ee3b81111ebbf189785fa69731c and pull request https://github.com/docker/docker-credential-helpers/pull/29/commits/79f93e5e69abc2ead458d19f4577b64b7e35c504 are the same fix. It's a crying shame that @aidanh010 filed this with a link to a fix last year but then it was somehow forgotten about...

[Sitsofe Wheeler (sitsofe)]

diff --git a/secretservice/secretservice_linux.go b/secretservice/secretservice_linux.go
index 95a1310..383b0c2 100644
--- a/secretservice/secretservice_linux.go
+++ b/secretservice/secretservice_linux.go
@@ -93,7 +93,7 @@ func (h Secretservice) List() (map[string]string, error) {
  var listLenC C.uint
  err := C.list(credsLabelC, &pathsC, &acctsC, &listLenC)
  if err != nil {
- defer C.free(unsafe.Pointer(err))
+ defer C.g_error_free(err)
   return nil, errors.New("Error from list function in secretservice_linux.c likely due to error in secretservice library")
  }
  defer C.freeListData(&pathsC, listLenC)

[r0mulux (r-marie)]

Hello, we are also affected by this bug.

dpkg -l | grep docker
ii docker-compose 1.17.1-2 all Punctual, lightweight development environments using Docker
ii docker.io 18.09.7-0ubuntu1~18.04.3 amd64 Linux container runtime
ii golang-docker-credential-helpers 0.5.0-2 amd64 Use native stores to safeguard Docker credentials
ii python-docker 2.5.1-1 all Python wrapper to access docker.io's control socket
ii python-dockerpty 0.4.1-1 all Pseudo-tty handler for docker Python client (Python 2.x)
ii python-dockerpycreds 0.2.1-1 all Python bindings for the docker credentials store API

[Alexander M (make-simpler-software)]

What does the packag "golang-github-docker-docker-credential-helpers" do?
->It stores username and password used for `docker login` to "docker registries" in a "safer way".

When does the bug manifest/occur?
->The bug occurs for instance when running `docker build`.

How to get rid of this bug (considering in Ubuntu 18.04 this package is still not fixed month after the report is filed)?
-> The bug can be circumvented (especially for those pointless, crazy cases like `docker build` where the credentials appear to be not even necessary) by disabling this "security for your credentials feature" eithe by:
 a) removing the @&(#* package via:
   `dpkg -r --force-depends golang-docker-credential-helpers`
 b) by setting a bogus credential helper program
   `printf '{ "credsStore":"bogus" }\n' > ~/.docker/config.json`

Option b) comes with less side affects (collateral damage) than option a) wich would make `apt` complain about the "broken/missing" package.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Sitsofe Wheeler (sitsofe)]

(For anyone who comes across this one of the upstream issues (https://github.com/docker/docker-credential-helpers/issues/103#issuecomment-576854271 ) mentions the problem goes away if `pass` package is also installed and this seemed to make the issue disappear for me too)

[r0mulux (r-marie)]

It works for me.
I confirm that error is not displayed anymore if package 'pass' is installed.

[Athos Ribeiro (athos-ribeiro)]

This is a debdiff to apply the fix proposed in https://github.com/docker/docker-credential-helpers/pull/29 to bionic.

[Sergio Durigan Junior (sergiodj)]

Thanks for the patch and for working on this bug, Athos.

A few comments:

- Since this is an SRU, the version will be a bit different than in a normal upload to a development version. In this particular case, the version should be:

  0.5.0-2ubuntu0.1

- The "Origin:" DEP-3 header should contain the direct link to the commit. It should also specify if this patch is a backport or if it was applied cleanly from upstream. For example:

  Origin: upstream, https://link.to.commit

or

  Origin: backport, https://link.to.commit

Other than that, the patch LGTM.

The SRU template also LGTM, but there's a small nit in the Test Case section: the "docker build" command expects the directory where the build will happen as an argument, so I think the command should be:

   docker build -t golang-docker-credential-helpers:test .

Other than that, I followed the reproduction steps and verified that (1) the error is indeed hapening, and (2) it is fixed by this patch.

Let me know when you update the patch and I can upload it. Thanks.

[Athos Ribeiro (athos-ribeiro)]

Thanks for the review, Sergio.

I addressed your comments and pushed the proposed fix to the following PPA:

https://launchpad.net/~athos-ribeiro/+archive/ubuntu/lp-1813003-docker-credential-helpers/+packages

[Sergio Durigan Junior (sergiodj)]

Thanks for addressing the comments, Athos.

Uploaded:

$ dput golang-github-docker-docker-credential-helpers_0.5.0-2ubuntu0.1_source.changes
Trying to upload package to ubuntu
Checking signature on .changes
gpg: /home/sergio/work/golang-github-docker-docker-credential-helpers/golang-github-docker-docker-credential-helpers_0.5.0-2ubuntu0.1_source.changes: Valid signature from 106DA1C8C3CBBF14
Checking signature on .dsc
gpg: /home/sergio/work/golang-github-docker-docker-credential-helpers/golang-github-docker-docker-credential-helpers_0.5.0-2ubuntu0.1.dsc: Valid signature from 106DA1C8C3CBBF14
Uploading to ubuntu (via ftp to upload.ubuntu.com):
  Uploading golang-github-docker-docker-credential-helpers_0.5.0-2ubuntu0.1.dsc: done.
  Uploading golang-github-docker-docker-credential-helpers_0.5.0-2ubuntu0.1.debian.tar.xz: done.
  Uploading golang-github-docker-docker-credential-helpers_0.5.0-2ubuntu0.1_source.buildinfo: done.
  Uploading golang-github-docker-docker-credential-helpers_0.5.0-2ubuntu0.1_source.changes: done.
Successfully uploaded packages.

[Athos Ribeiro (athos-ribeiro)]

Thanks for the review and upload, Sergio!

[Robie Basak (racb)]

I verified that the patch is applied in Impish.

[Robie Basak (racb)]

Won't this need a subsequent rebuild of other packages for this change to have any effect? Have you tested the outcome against your PPA?

[Sergio Durigan Junior (sergiodj)]

Hi Robie,

I verified that it is not necessary to rebuild other packages in this case. Just updating golang-docker-credential-helpers_0.5.0-2ubuntu0.1_amd64.deb inside a VM is enough to fix the issue and have the "docker build" stop manifesting the error. But maybe Athos would like to complement this info somehow.

[Athos Ribeiro (athos-ribeiro)]

Hi Robie,

This will not need a subsequent rebuild of other packages here since it is the binary package (and not the -dev one) which is triggering the reported issue.

However, may the bogus call to free manifest in any other form for any other binary package building against the golang -dev package in question, then they may need a rebuild.

I did test this change against a locally built package (built from the same sources pushed to that PPA). I just re-triggered my tests against the package (x86_64) available in that PPA just to make sure everything is working as expected here.

[Robie Basak (racb)]

Ah, I see now - it actually ships a binary. Thank you for confirming!

[Robie Basak (racb)]

Hello Sitsofe, or anyone else affected,

Accepted golang-github-docker-docker-credential-helpers into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/golang-github-docker-docker-credential-helpers/0.5.0-2ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Athos Ribeiro (athos-ribeiro)]

I tested this in a bionic lxc container with the following steps:

# install affected golang-docker-credential-helpers version 0.5.0-2
apt update && apt install -y docker.io golang-docker-credential-helpers
# prepare Dockerfile and run build
mkdir /tmp/dummy-docker-build
cd /tmp/dummy-docker-build
cat <<EOF > Dockerfile
FROM ubuntu:focal
LABEL hello=label
EOF
docker build -t testing . # This will get the error to be reproduced.
# then we can enable -proposed and upgrade golang-docker-credential-helpers
cat <<EOF >/etc/apt/sources.list.d/ubuntu-$(lsb_release -cs)-proposed.list
# Enable Ubuntu proposed archive
deb http://archive.ubuntu.com/ubuntu/ $(lsb_release -cs)-proposed restricted main multiverse universe
EOF
# install fixed golang-docker-credential-helpers version 0.5.0-2ubuntu0.1
apt update
apt install -y golang-docker-credential-helpers
# Finally, re-run the build command and verify the error no long occurs
docker build -t testing .

Hence, I confirm the issue if fixed by the package in -proposed.

[Launchpad Janitor (janitor)]

This bug was fixed in the package golang-github-docker-docker-credential-helpers - 0.5.0-2ubuntu0.1

---------------
golang-github-docker-docker-credential-helpers (0.5.0-2ubuntu0.1) bionic; urgency=medium

  * d/p/free_unsafe_pointer_fix.patch: fix invalid free call when running docker
    build (LP: #1813003)

 -- Athos Ribeiro <email address hidden> Tue, 10 Aug 2021 09:43:00 -0300

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for golang-github-docker-docker-credential-helpers has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.