Install of upstream Docker package only fails when running through cloud-init

[Matt Conway]

I'm not sure if this is a cloud-init issue, a systemd issue, or an issue with the package I'm trying to install, but given that I can install the package fine outside of cloud-init, I thought I'd mention it here in case this is an issue with cloud-init/wily/systemd, or perhaps someone more knowledgeable can suggest a workaround.

To summarize, this is easily duplicated by spinning up a new ec2 instance using ami-305d115a on a m3.medium and the following user-data:

#!/bin/bash
export DEBIAN_FRONTEND=noninteractive
sudo apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
sudo bash -c 'echo "deb https://apt.dockerproject.org/repo ubuntu-wily main" > /etc/apt/sources.list.d/docker.list'
sudo apt-get update && sudo apt-get install -y docker-engine

Expected:
To be able to install the package via ec2-user-data/cloud-init

Actual:
Package fails to install through cloud-init, but installs fine at terminal. Comparing the "journal -xe" from both cases seems to indicate that a dependent systemd service (docker.socket) installed by the package is not getting started when the install is run from cloud-init

More details:
https://github.com/docker/docker/issues/18444

System details:

# uname -a
Linux ip-10-229-17-174 4.2.0-19-generic #23-Ubuntu SMP Wed Nov 11 11:39:30 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 15.10
Release: 15.10
Codename: wily

# apt-cache policy cloud-init
cloud-init:
  Installed: 0.7.7~bzr1149-0ubuntu4
  Candidate: 0.7.7~bzr1149-0ubuntu5
  Version table:
     0.7.7~bzr1149-0ubuntu5 0
        500 http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ wily-updates/main amd64 Packages
 *** 0.7.7~bzr1149-0ubuntu4 0
        100 /var/lib/dpkg/status
     0.7.7~bzr1149-0ubuntu1 0
        500 http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ wily/main amd64 Packages

# apt-cache policy systemd
systemd:
  Installed: 225-1ubuntu9
  Candidate: 225-1ubuntu9
  Version table:
 *** 225-1ubuntu9 0
        500 http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ wily/main amd64 Packages
        100 /var/lib/dpkg/status

# apt-cache policy docker-engine
docker-engine:
  Installed: (none)
  Candidate: 1.9.1-0~wily
  Version table:
     1.9.1-0~wily 0
        500 https://apt.dockerproject.org/repo/ ubuntu-wily/main amd64 Packages
     1.9.0-0~wily 0
        500 https://apt.dockerproject.org/repo/ ubuntu-wily/main amd64 Packages
     1.8.3-0~wily 0
        500 https://apt.dockerproject.org/repo/ ubuntu-wily/main amd64 Packages
     1.8.2-0~wily 0
        500 https://apt.dockerproject.org/repo/ ubuntu-wily/main amd64 Packages
     1.8.1-0~wily 0
        500 https://apt.dockerproject.org/repo/ ubuntu-wily/main amd64 Packages
     1.8.0-0~wily 0
        500 https://apt.dockerproject.org/repo/ ubuntu-wily/main amd64 Packages
     1.7.1-0~wily 0
        500 https://apt.dockerproject.org/repo/ ubuntu-wily/main amd64 Packages
     1.6.2-0~wily 0
        500 https://apt.dockerproject.org/repo/ ubuntu-wily/main amd64 Packages
     1.6.1-0~wily 0
        500 https://apt.dockerproject.org/repo/ ubuntu-wily/main amd64 Packages
     1.6.0-0~wily 0
        500 https://apt.dockerproject.org/repo/ ubuntu-wily/main amd64 Packages
     1.5.0-0~wily 0
        500 https://apt.dockerproject.org/repo/ ubuntu-wily/main amd64 Packages

[Robie Basak (racb)]

I wonder if you can work around this by using policy-rc.d to ask for the docker service not to be started at install time, and instead when you manually try to start it (as you're reporting that doing it a bit later is working?)

That doesn't solve the actual problem though of course. I'd like to know what Scott thinks.

[Scott Moser (smoser)]

Ok. I've reproduced this, and collected some info here.
First, the following is a bit simpler user data to reproduce:
  #!/bin/bash
  exec >/run/docker.install.out 2>&1 </dev/null
  set -x
  export http_proxy=http://squid.internal:3128
  export DEBIAN_FRONTEND=noninteractive
  apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
  echo "deb https://apt.dockerproject.org/repo ubuntu-wily main" > /etc/apt/sources.list.d/docker.list
  apt-get update
  apt-get install -y docker-engine

The changes there are that cloud-init is executing as root, so no need for beign root. Second is that I just put all output to /run/docker.install.out. I've added the http_proxy for my own environment where that is needed.

And just an FYI, a similar cloud-config to accomplish the above looks like. It also reproduces the issue.
  #cloud-config
  apt_sources:
   - source: "deb https://apt.dockerproject.org/repo ubuntu-$RELEASE main"
     keyserver: hkp://p80.pool.sks-keyservers.net:80
     keyid: 58118E89F3A912897C070ADBF76221572C52609D
     filename: 'docker.list'
  packages:
   - docker-engine

I'll attach /run/docker.install.out and 'systemctl-status -l docker-service' and 'journal -xe' but I actually think we have to get docker run with more verbose output as I think its eating some useful error.

[Matt Conway (wr0ngway)]

Thanks for the tip, I was able to work around the problem using policy-rc.d

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Jason (japrice)]

This might be a different issue. But it feels related. In AWS if I build an instance, and install the docker.io package and then make an AMI.

Booting from that AMI, cloud-init is REALLY broken. Only runs the NoDataSource data source. None of the other cloud ones trigger (ec2, rightscale, etc.)

Steps to reproduce:

Launch amazon instance from the official canonical xenial AMI.
Install docker.io
Create a different shell user to log in with (cloud-init won't put your AWS SSH Key into the ubuntu user)
Create AMI from that instance.
Launch a second amazon instance from your new AMI