cloud-init no longer processes user data on GCE in artful

[Dan Watkins]

=== Begin SRU Template ===
[Impact]
Any user-data provided when creating google cloud instances is ignored so no instance customization is observed. This is a silent failure and no tracebacks in cloud-init represent that failure to the user.

Providing a simple cloud-config to set a hostname will provide a quick validation of cloud-init observing user-data.

[Test Case]

# Create cloud-config which should change the hostname, and cli prompt
$ cat > sethostname.yaml <<EOF
#cloud-config
hostname: SRU-worked
EOF

# Deploy on GCE
$ gcloud compute instances create artful-sru-test --zone=us-central1-b --image-family ubuntu-1710 --image-project ubuntu-os-cloud-devel --metadata-from-file user-data=sethostname.yaml

$ gcloud compute ssh artful-sru-test

# Expect default hostname "artful-sru-test" instead of "SRU-worked"
# Look for failure symptom: populated instance-data but user-data: null
artful-sru-test:~# sudo grep user-data /run/cloud/instance-data.json
   "instance-data": "{\"user-data\":\"#cloud-config\\nhostname: SRU-worked\\n\"}",
  "user-data": null,

# Upgrade cloud-init

# add proposed
rel=$(lsb_release -sc);
line=$(awk '$1 == "deb" && $2 ~ /ubuntu.com/ {
  printf("%s %s %s-proposed main universe\n", $1, $2, rel); exit(0) };
  ' "rel=$rel" /etc/apt/sources.list);
echo "$line" | sudo tee /etc/apt/sources.list.d/proposed.list
sudo apt-get update -q;
sudo apt-get install cloud-init;
# Reboot 'clean' allowing new cloud-init to run against a fresh system
sudo cloud-init clean --reboot --logs

$ gcloud compute ssh artful-sru-test
# Expect to see proper hostname
SRU-worked:~# sudo cloud-init status --long;

# Validate user-data: non-null
SRU-worked:~# sudo grep user-data /run/cloud-init/instance-data.json
artful-sru-test:~# sudo grep user-data /run/cloud-init/instance-data.json
   "instance-data": "{\"user-data\":\"#cloud-config\\nhostname: SRU-worked\\n\"}",
  "user-data": "I2Nsb3VkLWNvbmZpZwpob3N0bmFtZTogbXliaW9uaWMK",

[Other Info]
Upstream commit at
  https://git.launchpad.net/cloud-init/commit/?id=40e7738

=== End SRU Template ===

=== Begin Original Description ===

If I pass in user data like so:

$ cat cfg
#!/bin/sh
touch /tmp/foobar

$ gcloud compute instances create aa-$(date +%y%m%d-%H%M) --image-family ubuntu-1710 --image-project ubuntu-os-cloud-devel --metadata-from-file user-data=cfg
...

Then in the instance:

$ ls /tmp/foobar
$ sudo cat /var/lib/cloud-init/instance/user-data.txt
$ curl "http://metadata.google.internal/computeMetadata/v1/instance/attributes/user-data" -H "Metadata-Flavor: Google"
#/bin/sh
touch /tmp/foobar

[Ryan Harper (raharper)]

Note that /tmp and /var/tmp (and possibly others) are not safe during boot:

https://www.freedesktop.org/software/systemd/man/systemd-tmpfiles.html

Can you reproduce with a touch to a non-tmp path ?

[Dan Watkins (oddbloke)]

I have reproduced with a non-/tmp path.

Log tarball attached.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 18.1-5-g40e77380-0ubuntu1

---------------
cloud-init (18.1-5-g40e77380-0ubuntu1) bionic; urgency=medium

  * New upstream snapshot.
    - GCE: fix reading of user-data that is not base64 encoded. (LP: #1752711)
    - doc: fix chef install from apt packages example in RTD.
    - Implement puppet 4 support [Romanos Skiadas] (LP: #1446804)
    - subp: Fix subp usage with non-ascii characters when no system locale.
      (LP: #1751051)
    - salt: configure grains in grains file rather than in minion config.
      [Daniel Wallace]

 -- Chad Smith <email address hidden> Thu, 01 Mar 2018 15:47:04 -0700

[Timo Aaltonen (tjaalton)]

Hello Dan, or anyone else affected,

Accepted cloud-init into artful-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/17.2-35-gf576b2a2-0ubuntu1~17.10.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-artful to verification-done-artful. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-artful. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Timo Aaltonen (tjaalton)]

Hello Dan, or anyone else affected,

Accepted cloud-init into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/17.2-35-gf576b2a2-0ubuntu1~16.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Chad Smith (chad.smith)]

=== Abridged manual verification
### Artful
root@publishing:~# gcloud compute ssh artful-sru-test
No zone specified. Using zone [us-central1-b] for instance: [artful-sru-test].
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:qnFD0APJ/VAfRHXrYshvasMsvAjmJ/R6/lmmXQzGx6w.
Please contact your system administrator.
Add correct host key in /root/.ssh/google_compute_known_hosts to get rid of this message.
Offending ECDSA key in /root/.ssh/google_compute_known_hosts:4
  remove with:
  ssh-keygen -f "/root/.ssh/google_compute_known_hosts" -R "compute.5802718818769095753"
Password authentication is disabled to avoid man-in-the-middle attacks.
Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
Enter passphrase for key '/root/.ssh/google_compute_engine':
Enter passphrase for key '/root/.ssh/google_compute_engine':
Welcome to Ubuntu 17.10 (GNU/Linux 4.13.0-36-generic x86_64)

 * Documentation: https://help.ubuntu.com
 * Management: https://landscape.canonical.com
 * Support: https://ubuntu.com/advantage

 * Meltdown, Spectre and Ubuntu: What are the attack vectors,
   how the fixes work, and everything else you need to know
   - https://ubu.one/u2Know

  Get cloud support with Ubuntu Advantage Cloud Guest:
    http://www.ubuntu.com/business/services/cloud

0 packages can be updated.
0 updates are security updates.

Last login: Fri Mar 2 15:51:05 2018 from 67.174.121.94
root@artful-sru-test:~# dpkg-query --show cloud-init
cloud-init 17.2-35-gf576b2a2-0ubuntu1~17.10.1
root@artful-sru-test:~# # add proposed
root@artful-sru-test:~# rel=$(lsb_release -sc);
root@artful-sru-test:~# line=$(awk '$1 == "deb" && $2 ~ /ubuntu.com/ {
> printf("%s %s %s-proposed main universe\n", $1, $2, rel); exit(0) };
> ' "rel=$rel" /etc/apt/sources.list);
root@artful-sru-test:~# echo "$line" | sudo tee /etc/apt/sources.list.d/proposed.list
deb http://us-central1.gce.archive.ubuntu.com/ubuntu/ artful-proposed main universe
root@artful-sru-test:~# sudo apt-get update -q;

Hit:1 http://us-central1.gce.archive.ubuntu.com/ubuntu artful InRelease
Get:2 http://us-central1.gce.archive.ubuntu.com/ubuntu artful-updates InRelease [78.6 kB]
Get:3 http://us-central1.gce.archive.ubuntu.com/ubuntu artful-backports InRelease [72.2 kB]
Get:4 http://us-central1.gce.archive.ubuntu.com/ubuntu artful-proposed InRelease [235 kB]
...
Reading package lists...
root@artful-sru-test:~# sudo apt-get install cloud-init;
...
The following packages will be upgraded:
  cloud-init
1 upgraded, 0 newly installed, 0 to remove and 14 not upgraded.
Need to get 342 kB of archives.
After this operation, 1024 B of additional disk space will be used.
Get:1 http://us-central1.gce.archive.ubuntu.com/ubuntu artful-proposed/main amd64 cloud-init all 17.2-35-gf576b2a2-0ubuntu1~17.10.2 [342 kB]
...

[Chad Smith (chad.smith)]

=== Abridged SRU verification ===h
NOTE: Xenial didn't publish an SRU regression per cloud-init 17.2.35, it still is on 17.1.46 which properly processes user-data.

# Xenial

root@publishing:~# gcloud compute instances create xenial-sru-test --zone=us-central1-b --image daily-ubuntu-1604-xenial-v20180224 --image-project ubuntu-os-cloud-devel --metadata-from-file user-data=sethostname.yaml
Created [https://www.googleapis.com/compute/v1/projects/cloud-init-testing/zones/us-central1-b/instances/xenial-sru-test].
NAME ZONE MACHINE_TYPE PREEMPTIBLE INTERNAL_IP EXTERNAL_IP STATUS
xenial-sru-test us-central1-b n1-standard-1 10.128.0.2 35.184.113.47 RUNNING
root@publishing:~# gcloud compute ssh xenial-sru-test
No zone specified. Using zone [us-central1-b] for instance: [xenial-sru-test].
Warning: Permanently added 'compute.2904988054415753140' (ECDSA) to the list of known hosts.
Enter passphrase for key '/root/.ssh/google_compute_engine':
Welcome to Ubuntu 16.04.4 LTS (GNU/Linux 4.13.0-1011-gcp x86_64)

 * Documentation: https://help.ubuntu.com
 * Management: https://landscape.canonical.com
 * Support: https://ubuntu.com/advantage

  Get cloud support with Ubuntu Advantage Cloud Guest:
    http://www.ubuntu.com/business/services/cloud

0 packages can be updated.
0 updates are security updates.

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

root@SRU-worked:~# dpkg-query --show cloud-init
cloud-init 17.1-46-g7acc9e68-0ubuntu1~16.04.1
root@SRU-worked:~# hostname xenial-sru-test # set it back so we can validate upgrade
root@SRU-worked:~# exit
logout
Connection to 35.184.113.47 closed.
root@publishing:~# gcloud compute ssh xenial-sru-test
... # Validating hostname presented in PS1
Last login: Fri Mar 2 16:55:23 2018 from 67.174.121.94
root@xenial-sru-test:~# rel=$(lsb_release -sc);
root@xenial-sru-test:~# line=$(awk '$1 == "deb" && $2 ~ /ubuntu.com/ {
> printf("%s %s %s-proposed main universe\n", $1, $2, rel); exit(0) };
> ' "rel=$rel" /etc/apt/sources.list);
root@xenial-sru-test:~# echo "$line" | sudo tee /etc/apt/sources.list.d/proposed.list
deb http://us-central1.gce.archive.ubuntu.com/ubuntu/ xenial-proposed main universe
root@xenial-sru-test:~# sudo apt-get update -q;
Hit:1 http://us-central1.gce.archive.ubuntu.com/ubuntu xenial InRelease
Get:2 http://us-central1.gce.archive.ubuntu.com/ubuntu xenial-updates InRelease [102 kB]
Get:3 http://us-central1.gce.archive.ubuntu.com/ubuntu xenial-backports InRelease [102 kB]
Get:4 http://us-central1.gce.archive.ubuntu.com/ubuntu xenial-proposed InRelease [253 kB]
...
Reading package lists...
root@xenial-sru-test:~# sudo apt-get install cloud-init;
...
The following packages will be upgraded:
  cloud-init
...
Setting up cloud-init (17.2-35-gf576b2a2-0ubuntu1~16.04.2) ...
root@xenial-sru-test:~# # Clean cloud-init artifacts so cloud-init runs against fresh instance
root@xenial-sru-test:~# sudo cloud-init clean --reboot --logs...

[Chad Smith (chad.smith)]

# Xenial automated integration tests nocloud-lxd from https://jenkins.ubuntu.com/server/view/cloud-init/job/cloud-init-integration-lxd-x/465/consoleFull

Started by timer
Building remotely on torkoal (metal-amd64) in workspace /var/lib/jenkins/slaves/torkoal/workspace/cloud-init-integration-lxd-x
[cloud-init-integration-lxd-x] $ /bin/sh -xe /tmp/jenkins5189693012157108797.sh
+ release=xenial
+ export http_proxy=http://squid.internal:3128
+ export apt_proxy=http://squid.internal:3128
+ sudo rm -Rf cloud-init
+ git clone https://git.launchpad.net/cloud-init
Cloning into 'cloud-init'...
+ cd cloud-init
+ ./packages/bddeb -S
Creating a temporary tarball using the 'make-tarball' helper
Extracting temporary tarball 'cloud-init_18.1-5-g40e77380.orig.tar.gz'
Creating a debian/ folder in '/tmp/tmpprhxfdyx/cloud-init-18.1-5-g40e77380'
Running 'debuild -S -us -uc' in '/tmp/tmpprhxfdyx/cloud-init-18.1-5-g40e77380'
Wrote 'cloud-init_18.1-5-g40e77380-1~bddeb.debian.tar.xz'
Wrote 'cloud-init_18.1-5-g40e77380.orig.tar.gz'
Wrote 'cloud-init_18.1-5-g40e77380-1~bddeb_source.build'
Wrote 'cloud-init_18.1-5-g40e77380-1~bddeb.dsc'
Linked 'cloud-init_18.1-5-g40e77380-1~bddeb.dsc' to 'cloud-init.dsc'
Wrote 'cloud-init_18.1-5-g40e77380-1~bddeb_source.changes'
+ sbuild --nolog --dist=xenial cloud-init_18.1-5-g40e77380-1~bddeb.dsc
sbuild (Debian sbuild) 0.71.0 (24 Aug 2016) on torkoal

+==============================================================================+
| cloud-init 18.1-5-g40e77380-1~bddeb (amd64) Fri, 02 Mar 2018 11:05:13 +0000 |
+==============================================================================+

Package: cloud-init
Version: 18.1-5-g40e77380-1~bddeb
Source Version: 18.1-5-g40e77380-1~bddeb
Distribution: xenial
Machine Architecture: amd64
Host Architecture: amd64
Build Architecture: amd64

I: NOTICE: Log filtering will replace 'var/lib/schroot/mount/xenial-amd64-b6cb28b7-fdea-41fd-b128-84de1b8f35c3' with '<<CHROOT>>'

+------------------------------------------------------------------------------+
| Update chroot |
+------------------------------------------------------------------------------+

Get:1 http://security.ubuntu.com/ubuntu xenial-security InRelease [102 kB]
Hit:2 http://archive.ubuntu.com/ubuntu xenial InRelease
Get:3 http://archive.ubuntu.com/ubuntu xenial-updates InRelease [102 kB]
Get:4 http://archive.ubuntu.com/ubuntu xenial-proposed InRelease [253 kB]
Get:5 http://security.ubuntu.com/ubuntu xenial-security/main Sources [117 kB]
Get:6 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages [460 kB]
Get:7 http://archive.ubuntu.com/ubuntu xenial-updates/main Sources [299 kB]
Get:8 http://archive.ubuntu.com/ubuntu xenial-updates/universe Sources [192 kB]
Get:9 http://security.ubuntu.com/ubuntu xenial-security/universe amd64 Packages [321 kB]
Get:10 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages [737 kB]
Get:11 http://archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages [593 kB]
Get:12 http://archive.ubuntu.com/ubuntu xenial-proposed/main Sources [18.0 kB]
Get:13 http://archive.ubuntu.com/ubuntu xenial-proposed/universe S...

[Chad Smith (chad.smith)]

# Artful automated nocloud-kvm integration test logs from https://jenkins.ubuntu.com/server/view/cloud-init/job/cloud-init-integration-lxd-a/275/consoleFull

Started by upstream project "cloud-init-integration-lxd-x" build number 465
originally caused by:
 Started by timer
Building remotely on torkoal (metal-amd64) in workspace /var/lib/jenkins/slaves/torkoal/workspace/cloud-init-integration-lxd-a
[cloud-init-integration-lxd-a] $ /bin/sh -xe /tmp/jenkins7866899822674870610.sh
+ release=artful
+ export http_proxy=http://squid.internal:3128
+ export apt_proxy=http://squid.internal:3128
+ sudo rm -Rf cloud-init
+ git clone https://git.launchpad.net/cloud-init
Cloning into 'cloud-init'...
+ cd cloud-init
+ ./packages/bddeb -S
Creating a temporary tarball using the 'make-tarball' helper
Extracting temporary tarball 'cloud-init_18.1-5-g40e77380.orig.tar.gz'
Creating a debian/ folder in '/tmp/tmpgiu1usk0/cloud-init-18.1-5-g40e77380'
Running 'debuild -S -us -uc' in '/tmp/tmpgiu1usk0/cloud-init-18.1-5-g40e77380'
Wrote 'cloud-init_18.1-5-g40e77380-1~bddeb.debian.tar.xz'
Wrote 'cloud-init_18.1-5-g40e77380.orig.tar.gz'
Wrote 'cloud-init_18.1-5-g40e77380-1~bddeb_source.build'
Wrote 'cloud-init_18.1-5-g40e77380-1~bddeb.dsc'
Linked 'cloud-init_18.1-5-g40e77380-1~bddeb.dsc' to 'cloud-init.dsc'
Wrote 'cloud-init_18.1-5-g40e77380-1~bddeb_source.changes'
+ sbuild --nolog --dist=artful cloud-init_18.1-5-g40e77380-1~bddeb.dsc
sbuild (Debian sbuild) 0.71.0 (24 Aug 2016) on torkoal

+==============================================================================+
| cloud-init 18.1-5-g40e77380-1~bddeb (amd64) Fri, 02 Mar 2018 11:17:19 +0000 |
+==============================================================================+

Package: cloud-init
Version: 18.1-5-g40e77380-1~bddeb
Source Version: 18.1-5-g40e77380-1~bddeb
Distribution: artful
Machine Architecture: amd64
Host Architecture: amd64
Build Architecture: amd64

I: NOTICE: Log filtering will replace 'var/lib/schroot/mount/artful-amd64-369965b1-1c33-49f9-9f64-484f33ba4eb8' with '<<CHROOT>>'

+------------------------------------------------------------------------------+
| Update chroot |
+------------------------------------------------------------------------------+

Hit:1 http://archive.ubuntu.com/ubuntu artful InRelease
Get:2 http://security.ubuntu.com/ubuntu artful-security InRelease [78.6 kB]
Get:3 http://archive.ubuntu.com/ubuntu artful-updates InRelease [78.6 kB]
Get:4 http://archive.ubuntu.com/ubuntu artful-proposed InRelease [235 kB]
Get:5 http://security.ubuntu.com/ubuntu artful-security/main Sources [43.2 kB]
Get:6 http://security.ubuntu.com/ubuntu artful-security/universe amd64 Packages [46.1 kB]
Get:7 http://security.ubuntu.com/ubuntu artful-security/main amd64 Packages [136 kB]
Get:8 http://archive.ubuntu.com/ubuntu artful-updates/universe Sources [26.0 kB]
Get:9 http://archive.ubuntu.com/ubuntu artful-updates/main Sources [86.6 kB]
Get:10 http://archive.ubuntu.com/ubuntu artful-updates/universe amd64 Packages [89.9 kB]
Get:11 http://archive.ubuntu.com/ubuntu artful-updates/main amd64 Packages [216 kB]
Get:12 http://archive.ubuntu.com/ub...