GCE cloudinit and ubuntu keys from metadata to ubuntu authorized_keys

[Shane Peters]

[Impact]

 * Per documentation at https://wiki.ubuntu.com/GoogleComputeEngineSSHKeys ssh keys for cloudinit and ubuntu users should both be added to the 'ubuntu' users authorized_keys file.

 * This works fine in Xenial (16.04) and higher, but doesn't work for Trusty (14.04).

[Test Case]

 * Create a file that contains ssh public keys

   $ cat googlekeys
   test:ssh-rsa <one example key> <email address hidden>
   ubuntu:ssh-rsa <a second example key> <email address hidden>
   cloudinit:ssh-rsa <a third example key> <email address hidden>

  * Create an ubuntu 14.04 instance

    gcloud compute instances create ubuntu1404cloudinit --image-family ubuntu-1404-lts --image-project ubuntu-os-cloud --metadata-from-file=ssh-keys=googlekeys --metadata=block-project-ssh-keys=True

  * Create an ubuntu 16.04 instance

    gcloud compute instances create ubuntu1604cloudinit --image-family ubuntu-1604-lts --image-project ubuntu-os-cloud --metadata-from-file=ssh-keys=googlekeys --metadata=block-project-ssh-keys=True

  * Notice that the ubuntu user in the ubuntu 14.04 instance contains no keys from cloud-init (the keys there are added by the google daemon):

    $ sudo cat /home/ubuntu/.ssh/authorized_keys
    # Added by Google
    ssh-rsa <the second example key but added by google daemon> <email address hidden>

  * However, in 16.04,

    $ sudo cat /home/ubuntu/.ssh/authorized_keys
    ssh-rsa <the third example key added by cloud-init> <email address hidden>
    ssh-rsa <the second example key added by cloud-init> <email address hidden>
    # Added by Google
    ssh-rsa <the second example key added by the google daemon> <email address hidden>

[Regression Potential]

 * DatasourceGCE.py is heavily modified to fix this behavior in 14.04. That said, there is a medium amount of regression potential when using the GCE datasource. More specificallly, there is now stricter checking of the metadata source when used(platform_check=True).

 * Significant testing has been completed via the Google Compute platform as well as other none-GCE datasources (lxd) to confirm functionality and to test for possible regressions.

[Shane Peters (shaner)]

I've tested a "first boot" scenario using a customised image with this patched cloud-init and it works as expected (you can see both cloudinit and ubuntu keys above the 'Added by google' comment).

$ cat googlekeys
carbon:ssh-rsa AAAAB....ZCRI2M shaner@carbon
ubuntu:ssh-rsa AAAAB....65Otq/ shaner@ubuntu
cloudinit:ssh-rsa AAAA..6s3oSv shaner@cloudinit

# USING CUSTOM IMAGE WITH UPGRADED CLOUD-INIT
#############################################
$ gcloud compute instances create trusty1 --image custom-cloud-init --image-project firm-lacing-209716 --metadata-from-file=ssh-keys=googlekeys --metadata=block-project-ssh-keys=True
$ ssh ubuntu@x.x.x.x cat .ssh/authorized_keys
ssh-rsa AAAAB....65Otq/ shaner@ubuntu
ssh-rsa AAAA..6s3oSv shaner@cloudinit
# Added by Google
ssh-rsa AAAAB....65Otq/ shaner@ubuntu

# USING EXISTING IMAGE
################################
$ gcloud compute instances create trust2 --image-family ubuntu-1404-lts --image-project ubuntu-os-cloud --metadata-from-file=ssh-keys=googlekeys --metadata=block-project-ssh-keys=True

$ ssh ubuntu@${IP} cat .ssh/authorized_keys
# Added by Google
ssh-rsa AAAAB.....z65Otq/ shaner@ubuntu

$ scp cloud-init_0.7.5-0ubuntu1.23_all.deb ubuntu@${IP}:~/
$ ssh ubuntu@${IP} sudo dpkg -i cloud-init_0.7.5-0ubuntu1.23_all.deb
$ ssh ubuntu@${IP} sudo rm -rf /var/lib/cloud/instances
$ ssh ubuntu@${IP} sudo reboot
$ ssh ubuntu@${IP} cat .ssh/authorized_keys
ssh-rsa AAAA.....6s3oSv shaner@cloudinit
# Added by Google
ssh-rsa AAAAB....65Otq/ shaner@ubuntu

You'll notice in this second example, the 'ubuntu' key from my googlekeys file isn't added on top like you would expect if the image had an upgraded cloud-init. This is because the code is duplicate checking and won't add a key if it already exists.

[Scott Moser (smoser)]

Hi Shane,
Could you put this into a git branch and propose it for merge into
ubuntu/trusty branch of upstream?
Similar to

please feel free to ping me if what i'm asking isn't clear.

Scott.

[Scott Moser (smoser)]

Bah. similar to described in
 http://cloudinit.readthedocs.io/en/latest/topics/hacking.html
but instead of 'master' you'll propose merge into ubuntu/trusty.

Also, you will need to do the change as a "quilt 3.0" format...
that is to say your change will add a file to debian/patches/
and will update debian/patches/series.

[Shane Peters (shaner)]

Hi Scott,
Thanks for the guidance! I think I got it now, let me know if you see the merge proposal.

[Scott Moser (smoser)]

uploaded. 0.7.5-0ubuntu1.23
Currently in the SRU queue.

Thanks Shane.

[Robie Basak (racb)]

12:36 <rbasak> smoser: could you add SRU information to bug 1781039 please?

12:36 <rbasak> smoser: in particular Regression Potential. Looks like the entire cloudinit/sources/DataSourceGCE.py file has been rewritten or wholesale backported?

Or are you intending to follow the full process documented at https://wiki.ubuntu.com/CloudinitUpdates?

[Scott Moser (smoser)]

Shane,
Could you pleaes fill in a Stable Release Updates template on this bug?

https://wiki.ubuntu.com/StableReleaseUpdates

Thanks.

[Shane Peters (shaner)]

Updated SRU template. Let me know if there's anything I'm missing.

[Brian Murray (brian-murray)]

Hello Shane, or anyone else affected,

Accepted cloud-init into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.5-0ubuntu1.23 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-trusty to verification-done-trusty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-trusty. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[David Coronel (davecore)]

I tested the package cloud-init 0.7.5-0ubuntu1.23 from trusty-proposed and confirm it works as expected.

# TEST KEYS

$ cat googlekeys
test:ssh-rsa AAAA[...]+1LRl <email address hidden>
ubuntu:ssh-rsa AAAA[...]+2LRl <email address hidden>
cloudinit:ssh-rsa AAAA[...]+3LRl test

# CREATE THE INSTANCE

$ gcloud compute instances create ubuntu1404cloudinittest --image-family ubuntu-1404-lts --image-project ubuntu-os-cloud --metadata-from-file=ssh-keys=googlekeys --metadata=block-project-ssh-keys=True --zone us-central1-a

# BEFORE THE UPDATE

$ gcloud compute --project "ubuntu-os-support" ssh --zone "us-central1-a" ubuntu1404cloudinittest

$ sudo cat /home/ubuntu/.ssh/authorized_keys
ssh-rsa AAAA[...]+fVtBKqZ fubar@famine
# Added by Google
ssh-rsa AAAA[...]+2LRl <email address hidden>

# AFTER THE UPDATE

$ sudo sh -c 'echo deb http://us-central1.gce.archive.ubuntu.com/ubuntu/ trusty-proposed main restricted universe multiverse >> /etc/apt/sources.list'

$ sudo apt update

$ sudo apt install cloud-init
[...]
Setting up cloud-init (0.7.5-0ubuntu1.23) ...
[...]

$ sudo sh -c 'cat /dev/null > /home/ubuntu/.ssh/authorized_keys'

$ sudo rm -rf /var/lib/cloud/instance*

$ sudo reboot

$ gcloud compute --project "ubuntu-os-support" ssh --zone "us-central1-a" ubuntu1404cloudinittest

$ sudo cat /home/ubuntu/.ssh/authorized_keys
ssh-rsa AAAA[...]+2LRl <email address hidden>
ssh-rsa AAAA[...]+3LRl test
# Added by Google
ssh-rsa AAAA[...]+2LRl <email address hidden>

cloud-init now works the same way as Ubuntu 16.04 does. Thanks!

Bonus test: I tested that oslogin also works well:

$ gcloud compute instances add-metadata ubuntu1404cloudinittest --metadata enable-oslogin=TRUE --zone "us-central1-a"
Updated [https://www.googleapis.com/compute/v1/projects/ubuntu-os-support/zones/us-central1-a/instances/ubuntu1404cloudinittest].

$ gcloud compute --project "ubuntu-os-support" ssh --zone "us-central1-a" ubuntu1404cloudinittest
[...]
Welcome to Ubuntu 14.04.5 LTS (GNU/Linux 4.4.0-137-generic x86_64)
[...]

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for cloud-init has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.5-0ubuntu1.23

---------------
cloud-init (0.7.5-0ubuntu1.23) trusty; urgency=medium

  - debian/control: added python-six dependency.
  - debian/patches/lp-1781039-gce-datasource-update.patch:
    Backport GCE datasource functionality from Xenial (LP: #1781039).

 -- Shane Peters <email address hidden> Tue, 06 Sep 2018 17:57:23 -0400

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/3208