cloud-init doesn't always land files that one expects

[Amrith Kumar]

---- Begin SRU Template ----
[Impact]
Injected files functionality of OpenStack's config drive is broken.

[Test Case]
== Reproduce broken functionality ==
$ echo "hi mom" > my-file.txt
$ cat > "user-data" <<"EOF"
#!/bin/sh
logfile=/run/my.log
file="/my-file.txt"
if [ -e "$file" ]; then
   ( echo "=== PASS: file $file ====" ; cat $file ) | tee -a $logfile
   exit 0
else
   echo "=== FAIL: no file $file ====" | tee -a $logfile
   exit 0
fi
EOF

openstack server create --key-name=brickies --flavor=m1.small \
  --config-drive=1 --image=$img \
  --user-data=user-data --file=/my-file.txt=my-file.txt \
  injected-file0

The launched system will have a file in /run/my.log that shows 'FAIL'
and will not have /my-file.txt on disk.

== See Fix ==
# enable proposed
$ cat > enable-proposed <<"EOF"
#!/bin/sh
set -e
rel=$(lsb_release -sc)
awk '$1 == "deb" && $2 ~ /ubuntu.com/ {
  printf("%s %s %s-proposed main universe\n", $1, $2, rel); exit(0) };
  ' "rel=$rel" /etc/apt/sources.list |
    tee /etc/apt/sources.list.d/proposed.list
EOF
$ sudo sh ./enable-proposed
$ sudo apt-get update
$ sudo apt-get install cloud-init

# Remove /var/lib/cloud and /var/log/cloud-init* to remove state
# and indicate this is a new instance on reboot
$ sudo rm -Rf /var/lib/cloud /var/log/cloud-init*
$ sudo reboot

Now ssh back in after reboot, you should
a.) have /my-file.txt
b.) see PASS in /run/my.log
c.) see mention of the 'injected file' in /var/log/cloud-init.log

[Regression Potential]
Regression potential on Ubuntu should be very small.

---- End SRU Template ----

Trove launches instances using the servers.create() API with some files. Trove provides a dictionary of files that it wants on the instance and most of the time this works. Nova passes them to the launched VM as metadata on config drive.

Sometimes though, it doesn't.

When injection fails, I see a cloud-init.log that looks like this:

https://gist.github.com/amrith/7566d8fef4b6e813cca77e5e3b1f1d5a

When injection succeeds, I see a cloud-init.log that looks like this:

https://gist.github.com/amrith/50d9e3050d88ec51e13b0a510bd718c3

Observe that the one that succeeds properly injects three files:

/etc/injected_files (which is something I added just for debugging)
/etc/trove/conf.d/... two files here ...

On a machine where this injection failed:

root@m10:/tmp/zz/openstack/content# ls -l /etc/trove
total 4
drwxr-xr-x 2 amrith root 4096 Jul 12 16:55 conf.d
root@m10:/tmp/zz/openstack/content# ls -l /etc/trove/conf.d/
total 4
root@m10:/tmp/zz/openstack/content#

Clearly, no files made it over. Yet, the files are definitely there on the config drive ...

I've mounted the config drive.

root@m10:/tmp/zz/openstack/content# mount | grep zz
/dev/sr0 on /tmp/zz type iso9660 (ro,relatime)

root@m10:/tmp/zz/openstack/content# cd /tmp/zz
root@m10:/tmp/zz# find .
.
./ec2
./ec2/2009-04-04
./ec2/2009-04-04/meta-data.json
./ec2/latest
./ec2/latest/meta-data.json
./openstack
./openstack/2012-08-10
./openstack/2012-08-10/meta_data.json
./openstack/2013-04-04
./openstack/2013-04-04/meta_data.json
./openstack/2013-10-17
./openstack/2013-10-17/meta_data.json
./openstack/2013-10-17/vendor_data.json
./openstack/2015-10-15
./openstack/2015-10-15/meta_data.json
./openstack/2015-10-15/network_data.json
./openstack/2015-10-15/vendor_data.json
./openstack/2016-06-30
./openstack/2016-06-30/meta_data.json
./openstack/2016-06-30/network_data.json
./openstack/2016-06-30/vendor_data.json
./openstack/content
./openstack/content/0000
./openstack/content/0001
./openstack/content/0002
./openstack/latest
./openstack/latest/meta_data.json
./openstack/latest/network_data.json
./openstack/latest/vendor_data.json

root@m10:/tmp/zz# cd openstack/latest/
root@m10:/tmp/zz/openstack/latest# cat meta_data.json
{"files": [{"path": "/etc/injected_files", "content_path": "/content/0000"}, {"path": "/etc/trove/conf.d/guest_info.conf", "content_path": "/content/0001"}, {"path": "/etc/trove/conf.d/trove-guestagent.conf", "content_path": "/content/0002"}], "admin_pass": "pf2ng7tT8JSt", "random_seed": "uSNqQ3udiKspue4y8R8b4gg33Qe6klYYJa8ebvDS0t4i88rq2Owu4yC8TysAfsmsYpno0rbWpWisiTQ3raAOPsx+kGgkrunM9UR5khs/1XRh60tlpKJVIHZnKpLv4PcisLKL2DDoHaaFLV9lPcVtZi3iTqKu6RhcwFyhh+A0mD0xg2G89qACD/RNhWiAuQs4X8le+qgIeoRb3wwg7+4dpujLiqKyx7edDgs9zaEsng21YxO0kv47PiwQf7GoIXObR3xtBYClQQfWQ8a1o35gpDPkPSowqHNKJIxFnIrdFgVHBK5EFTAmwN2JY/GcLwQxLydK02+aalem+bznOpsa1Jl86rNAI9pwltyFYDVz8CGwy46pNYtCOJ2W5WZ+wF4KssE0LkxeWcq7w7CXoFCgz42k13ki0gcHYHw0ieJTMULQT3k65gLwGL2/EsJrZjgil5AaCQhLBLkY2sPhHKYs0k+331QttHHs7PjoX/0as63iNiY62M10givhwyrPDaeYSOAidD0PfRpvVZVcQym/jgKmytV6ZU8DBDwAujGtHjdXK4AovcAvqc96H97arahQr/1SHk8RktyedYjqLC6iYwbSFSHTs+7Fb07MspWKQ2cJLlpmytd/lcZHYAElF/b0VBhR8/eU0dUqdpxjvfGsx8T9/rflMaM5wrDlhad/ikQ=", "uuid": "38731f42-611e-465f-b3e4-1e80aa623caf", "availability_zone": "nova", "hostname": "m10.novalocal", "launch_index": 0, "devices": [], "project_id": "eeec3236ef004821a77ccbd26b81f18f", "name": "m10"}

root@m10:/tmp/zz/openstack/latest#

Sure enough, the files I want are in content/0000, 0001 and 0002

root@m10:/tmp/zz/openstack/content# ls -l
total 4
-r--r--r-- 1 root root 1329 Jul 12 16:55 0000
-r--r--r-- 1 root root 127 Jul 12 16:55 0001
-r--r--r-- 1 root root 1050 Jul 12 16:55 0002

[Matt Riedemann (mriedem)]

cloud-init != nova, if there is a bug with cloud-init then that needs to be reported upstream.

[Amrith Kumar (amrith)]

This appears to be a regression from

https://launchpad.net/ubuntu/+source/cloud-init/0.7.7~bzr1212-0ubuntu1

I reverted to that version, and things seem to work just fine.

[Scott Moser (smoser)]

fix-committed in 1255

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.7~bzr1256-0ubuntu1

---------------
cloud-init (0.7.7~bzr1256-0ubuntu1) yakkety; urgency=medium

  * New upstream snapshot.
    - distros/debian.py: fix eni renderer to not render .link files
    - fixes for execution in python2.6.
    - ConfigDrive: fix writing of 'injected' files and legacy networking
      (LP: #1602373)
    - improvements to /etc/network/interfaces rendering including rendering
      of 'lo' devices and sorting attributes within a interface section.
    - fix mcollective module that was completely broken if using python3
      (LP: #1597699)

 -- Scott Moser <email address hidden> Thu, 14 Jul 2016 14:54:05 -0400

[Chris J Arges (arges)]

Hello Amrith, or anyone else affected,

Accepted cloud-init into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.7~bzr1256-0ubuntu1~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Scott Moser (smoser)]

Verified.
# launch instance
sstack openstack server create --key-name=brickies --flavor=m1.small --config-drive=1 --image=$img --user-data=user-data --file=/my-file.txt=my-file.txt injected-file0

# on the instance (first boot, unpatched)
$ cat /etc/cloud/build.info
build_name: server
serial: 20160715
$ cat /run/my.log
=== FAIL: no file /my-file.txt ====
$ dpkg-query --show cloud-init
cloud-init 0.7.7~bzr1246-0ubuntu1~16.04.1

# update the system
$ sudo sh ./enable-proposed
$ sudo apt-get update
$ sudo apt-get install cloud-init
$ dpkg-query --show cloud-init
cloud-init 0.7.7~bzr1256-0ubuntu1~16.04.1

# similate first boot on pristine image
$ sudo rm -Rf /var/log/cloud-init /var/lib/cloud/
$ sudo reboot

# ssh in
$ cat /run/my.log
=== PASS: file /my-file.txt ====
hi mom

$ ls -l /my-file.txt
-rw-rw---- 1 root root 7 Jul 18 18:57 /my-file.txt
$ sudo cat /my-file.txt
hi mom

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.7~bzr1256-0ubuntu1~16.04.1

---------------
cloud-init (0.7.7~bzr1256-0ubuntu1~16.04.1) xenial-proposed; urgency=medium

  * New upstream snapshot.
    - distros/debian.py: fix eni renderer to not render .link files
    - fixes for execution in python2.6.
    - ConfigDrive: fix writing of 'injected' files and legacy networking
      (LP: #1602373)
    - improvements to /etc/network/interfaces rendering including rendering
      of 'lo' devices and sorting attributes within a interface section.
    - fix mcollective module that was completely broken if using python3
      (LP: #1597699)

 -- Scott Moser <email address hidden> Fri, 15 Jul 2016 13:27:04 -0400

[Adam Conrad (adconrad)]

The verification of the Stable Release Update for cloud-init has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Scott Moser (smoser)]

This is fixed in cloud-init 0.7.7

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/2692