docker service failed to start on s390x (zVM) Yakkety

[Po-Hsu Lin]

docker.service failed to start on s390x(zVM) Yakkety 4.8.0-40-generic proposed kernel

# systemctl status docker.service
● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; disabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/docker.service.d
           └─http-proxy.conf
   Active: failed (Result: start-limit-hit) since Wed 2017-03-08 06:06:54 EST; 14min ago
     Docs: https://docs.docker.com
  Process: 3752 ExecStart=/usr/bin/dockerd -H fd:// $DOCKER_OPTS (code=exited, status=1/FAILURE)
 Main PID: 3752 (code=exited, status=1/FAILURE)

Mar 08 06:06:54 kernel02 systemd[1]: docker.service: Unit entered failed state.
Mar 08 06:06:54 kernel02 systemd[1]: docker.service: Failed with result 'exit-code'.
Mar 08 06:06:54 kernel02 systemd[1]: docker.service: cgroup is empty
Mar 08 06:06:54 kernel02 systemd[1]: docker.service: Trying to enqueue job docker.service/start/replace
Mar 08 06:06:54 kernel02 systemd[1]: docker.service: Installed new job docker.service/start as 1897
Mar 08 06:06:54 kernel02 systemd[1]: docker.service: Enqueued job docker.service/start as 1897
Mar 08 06:06:54 kernel02 systemd[1]: docker.service: Start request repeated too quickly.
Mar 08 06:06:54 kernel02 systemd[1]: docker.service: Job docker.service/start finished, result=failed
Mar 08 06:06:54 kernel02 systemd[1]: Failed to start Docker Application Container Engine.
Mar 08 06:06:54 kernel02 systemd[1]: docker.service: Failed with result 'start-limit-hit'.

Please find attachment for partial log from journalctl -xe

ProblemType: Bug
DistroRelease: Ubuntu 16.10
Package: docker.io 1.12.6-0ubuntu1~16.10.1
ProcVersionSignature: Ubuntu 4.8.0-40.43-generic 4.8.17
Uname: Linux 4.8.0-40-generic s390x
NonfreeKernelModules: zfs zunicode zcommon znvpair zavl
ApportVersion: 2.20.3-0ubuntu8.2
Architecture: s390x
Date: Wed Mar 8 06:17:52 2017
SourcePackage: docker.io
UpgradeStatus: No upgrade log present (probably fresh install)

[Po-Hsu Lin (cypressyew)]

It looks like this is a regression, I switched back to 4.8.0-38, and docker service can be started without any problem.

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1671058

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Joseph Salisbury (jsalisbury)]

Would you like me to assist with a bisect on this one?

[Po-Hsu Lin (cypressyew)]

Hi Joseph,
I will give it another shot first when the new Yakkety kernel is available.

[Po-Hsu Lin (cypressyew)]

Whoa!
It's still failing with the re-spinned 4.8.0-42-generic kernel.
But this time I can see more error message from systemctl status output, which is complaining about the driver this time.

(Test case: ubuntu_fan_smoke_test)

# systemctl status docker.service
docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; disabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/docker.service.d
           ~T~T~T~@http-proxy.conf
   Active: failed (Result: start-limit-hit) since Thu 2017-03-09 23:42:04 EST; 2min 7s ago
     Docs: https://docs.docker.com
  Process: 3482 ExecStart=/usr/bin/dockerd -H fd:// $DOCKER_OPTS (code=exited, status=1/FAILURE)
 Main PID: 3482 (code=exited, status=1/FAILURE)

Mar 09 23:42:03 kernel02 dockerd[3482]: time="2017-03-09T23:42:03.315347000-05:00" level=info msg="libcontainerd: new containerd process, pid: 3501"
Mar 09 23:42:04 kernel02 dockerd[3482]: time="2017-03-09T23:42:04.370521000-05:00" level=error msg="[graphdriver] prior storage driver \"aufs\" failed: driver not supported"
Mar 09 23:42:04 kernel02 dockerd[3482]: time="2017-03-09T23:42:04.370569000-05:00" level=fatal msg="Error starting daemon: error initializing graphdriver: driver not supported"
Mar 09 23:42:04 kernel02 systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Mar 09 23:42:04 kernel02 systemd[1]: Failed to start Docker Application Container Engine.
Mar 09 23:42:04 kernel02 systemd[1]: docker.service: Unit entered failed state.
Mar 09 23:42:04 kernel02 systemd[1]: docker.service: Failed with result 'exit-code'.
Mar 09 23:42:04 kernel02 systemd[1]: docker.service: Start request repeated too quickly.
Mar 09 23:42:04 kernel02 systemd[1]: Failed to start Docker Application Container Engine.
Mar 09 23:42:04 kernel02 systemd[1]: docker.service: Failed with result 'start-limit-hit'.

[Po-Hsu Lin (cypressyew)]

OK the missing driver error message give us a clue, the linux-image-extra-4.8.0-42-generic package was not installed.
It work well after that, I will close this bug. Thanks!

[Dimitri John Ledkov (xnox)]

We must support docker on cloud images which by default do not install -extra- package.

Thus missing default drivers should be in the main kernel package.

[Po-Hsu Lin (cypressyew)]

Verified with the older 4.8.0-38 kernel, docker.io version 1.12.1-0ubuntu15 and 1.12.6-0ubuntu1~16.10.1 will both fail without the -extra package as well.

So just like xnon mentioned above, we will need to add the missing driver to the main kernel package.

[Andy Whitcroft (apw)]

This bug was nominated against a series that is no longer supported, ie yakkety. The bug task representing the yakkety nomination is being closed as Won't Fix.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Dimitri John Ledkov (xnox)]

Checking in on this stale bug report.

Which kernel drivers are missing from the main linux-image package, and have we moved those in later releases and SRUed into xenial?

[Launchpad Janitor (janitor)]

[Expired for linux (Ubuntu) because there has been no activity for 60 days.]