failing to start on ppc64el: R_PPC64_ADDR16_HA 277ef287d88 for symbol `' out of range

[Iain Lane]

I noticed that some autopkgtests were timing out & indefinitely restarting, because they were having problems installing their dependencies. The last lines in their log output are:

Setting up docker.io (1.12.6-0ubuntu7) ...
Adding group `docker' (GID 116) ...
Done.
Created symlink /etc/systemd/system/sockets.target.wants/docker.socket → /lib/systemd/system/docker.socket.

...upon SSHing into the instance, I can see that it's when we're trying to start docker

root 885 0.0 0.9 19584 14848 ? Ss 17:11 0:00 \_ sshd: ubuntu [priv]
ubuntu 905 0.0 0.7 19776 10880 ? S 17:11 0:00 | \_ sshd: ubuntu@notty
root 2140 0.0 0.5 11136 8576 ? Ss 17:26 0:00 | \_ sudo -n /tmp/autopkgtest-run-wrapper env DEBIAN_FRONTEND=noninteractive APT_LISTBUGS_FRONTEND=none APT_LISTCHANGES_FRONTEND=none /bin/sh -ec /usr/bin/e
root 2141 0.0 0.2 6848 4224 ? S 17:26 0:00 | \_ /bin/bash /tmp/autopkgtest-run-wrapper env DEBIAN_FRONTEND=noninteractive APT_LISTBUGS_FRONTEND=none APT_LISTCHANGES_FRONTEND=none /bin/sh -ec /usr
root 2143 0.0 0.0 3136 1344 ? S 17:26 0:00 | \_ /bin/sh -ec /usr/bin/eatmydata apt-get install --assume-yes --fix-broken -o APT::Status-Fd=3 -o APT::Install-Recommends=False -o Dpkg::Options:
root 2144 1.1 3.3 57408 51968 ? S 17:26 0:00 | \_ /usr/bin/apt-get install --assume-yes --fix-broken -o APT::Status-Fd=3 -o APT::Install-Recommends=False -o Dpkg::Options::=--force-confnew
root 2156 0.0 0.3 10112 5504 pts/0 Ss+ 17:26 0:00 | \_ /usr/bin/dpkg --force-confnew --status-fd 18 --configure --pending
root 2157 0.0 0.1 3520 1728 pts/0 S+ 17:26 0:00 | \_ /bin/sh /var/lib/dpkg/info/docker.io.postinst configure
root 2177 0.0 0.1 3520 1728 pts/0 S+ 17:26 0:00 | \_ /bin/sh /usr/sbin/invoke-rc.d docker start
root 2197 0.0 0.4 16192 6400 pts/0 S+ 17:26 0:00 | \_ systemctl start docker.service

Looking at systemctl status shows:

ubuntu@autopkgtest:~$ systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; disabled; vendor preset: enabled)
   Active: activating (start) since Sun 2017-08-20 17:26:37 UTC; 1min 26s ago
     Docs: https://docs.docker.com
 Main PID: 2199 (dockerd)
    Tasks: 6
   Memory: 8.5M
      CPU: 18ms
   CGroup: /system.slice/docker.service
           └─2199 /usr/bin/dockerd -H fd://

Aug 20 17:26:37 autopkgtest systemd[1]: Starting Docker Application Container Engine...
Aug 20 17:26:37 autopkgtest dockerd[2199]: time="2017-08-20T17:26:37.787523838Z" level=info msg="libcontainerd: new containerd process, pid: 2217"
Aug 20 17:26:37 autopkgtest dockerd[2199]: containerd: error while loading shared libraries: R_PPC64_ADDR16_HA 219eff67d88 for symbol `' out of range
Aug 20 17:26:43 autopkgtest dockerd[2199]: containerd: error while loading shared libraries: R_PPC64_ADDR16_HA 277ef287d88 for symbol `' out of range
Aug 20 17:26:43 autopkgtest dockerd[2199]: time="2017-08-20T17:26:43.794696672Z" level=info msg="libcontainerd: new containerd process, pid: 2240"

... any idea what's wrong here?

This is artful, with kernel

ubuntu@autopkgtest:~$ uname -a
Linux autopkgtest 4.12.0-11-generic #12-Ubuntu SMP Fri Aug 11 12:23:06 UTC 2017 ppc64le ppc64le ppc64le GNU/Linux

and docker.io version

ubuntu@autopkgtest:~$ apt-cache policy docker.io
docker.io:
  Installed: 1.12.6-0ubuntu7
  Candidate: 1.12.6-0ubuntu7
  Version table:
 *** 1.12.6-0ubuntu7 500
        500 http://ftpmaster.internal/ubuntu artful/universe ppc64el Packages
        100 /var/lib/dpkg/status

The same version of docker managed to get through its testsuite (failed, but didn't hang) on 2017-08-07 <https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-artful/artful/ppc64el/d/docker.io/20170807_220510_84af7@/log.gz> with kernel "Linux 4.11.0-10-generic #15-Ubuntu SMP Thu Jun 29 15:02:54 UTC 2017"

[Michael Hudson-Doyle (mwhudson)]

I can't reproduce this in an artful lxd container. If you or anyone can reproduce this and give me access to the environment it reproduces in I'd really love to have a look.

[Michael Hudson-Doyle (mwhudson)]

Can't reproduce in a VM either. Setting to incomplete until it can be reproduced.

[Michael Hudson-Doyle (mwhudson)]

I've managed to reproduce this enough, I think, and I actually think a no-change rebuild of containerd will fix it -- it was being linked as PIE but from code that had not been compiled as position independent which I think must have been a toolchain bug but in any case doesn't happen again if I rebuild the package. For some reason, this bogus PIE-ness is causing problems with the new kernel.