salt minion module writes minion keys to the wrong directory

[Nathan Grennan]

==== Begin SRU Template ====
[Impact]
Salt minion config module of cloud-init would not work by default
if 'public_key' and 'private_key' were provided.

[Test Case]
## Recreate failure
$ cat >user-data <<EOF
#cloud-config
salt_minion:
  public_key: "foo public"
  private_key: "foo private"
EOF

$ lxc launch ubuntu-daily:xenial x1 "--config=user.user-data=$(cat user-data)"
$ lxc exec x1 -- grep salt/pki/ /var/log/cloud-init.log
Sep 13 21:04:55 ubuntu [CLOUDINIT] util.py[DEBUG]: Writing to /etc/salt/pki/minion.pub - wb: [420] 10 bytes
Sep 13 21:04:55 ubuntu [CLOUDINIT] util.py[DEBUG]: Writing to /etc/salt/pki/minion.pem - wb: [420] 11 bytes

## Note, that ubuntu's packaging actuall moves these files to their proper
## location, so checking the log is all we can do to show failure.

## Now update container, clean and reboot to show first boot
$ lxc exec x1 -- sh -c '
    p=/etc/apt/sources.list.d/proposed.list
    echo deb http://archive.ubuntu.com/ubuntu xenial-proposed main > "$p" &&
    apt-get update -q && apt-get -qy install cloud-init'
$ lxc exec x1 -- sh -c 'apt-get -qy --purge remove salt-minion && rm -Rf /etc/salt'
$ lxc exec x1 -- sh -c '
    cd /var/lib/cloud && for d in *; do [ "$d" = "seed" ] || rm -Rf "$d"; done
    rm -Rf /var/log/cloud-init*'

$ lxc exec x1 reboot

$ lxc exec x1 -- grep salt/pki/ /var/log/cloud-init.log
Sep 13 21:10:52 x1 [CLOUDINIT] util.py[DEBUG]: Writing to /etc/salt/pki/minion/minion.pub - wb: [420] 10 bytes
Sep 13 21:10:52 x1 [CLOUDINIT] util.py[DEBUG]: Writing to /etc/salt/pki/minion/minion.pem - wb: [420] 11 bytes

[Regression Potential]
Low chance for regression, especially since the packaging does the right thing.
==== End SRU Template ====

Cloud-init's salt minion module writes minion.pem, and minion.pub to the wrong directory. Salt-minion expects them in /etc/salt/pki/minion, but /etc/salt/pki is used by cloud-init's salt minion module. Somehow in the past this worked out, and the files would be moved to /etc/salt/pki/minion. This part I don't understand, but currently on Ubuntu 16.04 Xenial with cloud-init 0.7.7 it doesn't work out. What happens is cloud-init writes to /etc/salt/pki, and salt-minion ignores the /etc/salt/pki files and writes it's own /etc/salt/pki/minion files. This results in the salt minion generated keys being rejected by the salt master.

Current:
pki_dir = salt_cfg.get('pki_dir', '/etc/salt/pki')

Fixed:
pki_dir = salt_cfg.get('pki_dir', '/etc/salt/pki/minion')

[Nathan Grennan (9-ubuntuone-g)]

The answer to the mystery is that cloud-init's salt minion module was originally written for salt two years ago when /etc/salt/pki was the path. At some point the changes to /etc/salt/pki/minion, and added an auto migration function. But the auto migration function is wrapped in an if statement that says only run it if the transport is zeromq. The default transport seems to not be zeromq anymore, so the migration no longer runs.

I am going to file a bug with salt, but cloud-init should still be fixed to use the new path. Then it won't depend on the migration.

[Nathan Grennan (9-ubuntuone-g)]

Correction: The if statement is only around the master migration, not the minion. It also seems to work just fine on trusty, but not xenial.

[Nathan Grennan (9-ubuntuone-g)]

diff -uNr cloud-init-0.7.7~bzr1256/cloudinit/config/cc_salt_minion.py cloud-init-0.7.7~bzr1256-salt-pki-path/cloudinit/config/cc_salt_minion.py
--- cloud-init-0.7.7~bzr1256/cloudinit/config/cc_salt_minion.py 2016-07-14 11:49:20.000000000 -0700
+++ cloud-init-0.7.7~bzr1256-salt-pki-path/cloudinit/config/cc_salt_minion.py 2016-08-04 11:34:39.092898311 -0700
@@ -46,7 +46,7 @@

     # ... copy the key pair if specified
     if 'public_key' in salt_cfg and 'private_key' in salt_cfg:
- pki_dir = salt_cfg.get('pki_dir', '/etc/salt/pki')
+ pki_dir = salt_cfg.get('pki_dir', '/etc/salt/pki/minion')
         with util.umask(0o77):
             util.ensure_dir(pki_dir)
             pub_name = os.path.join(pki_dir, 'minion.pub')

[Scott Moser (smoser)]

Hi, is there any way to determine generically if it should use one path or the other?
could we use /etc/salt/pki/minion if thats an existing directory and /etc/salt/pki otherwise ? That would assume the package installed that directory.

Also note, you can pass in configuration in cloud-config to override the default that is there

#cloud-config
salt_minion:
  pki_dir: /etc/salt/pki/minion

Clearly you should not have to do that, but it works in all cases if you specify it correctly.

[Scott Moser (smoser)]

Hi,
See the linked branch. Would that work ?

Or is there a different approach that could assume the new directory and fallback to the old one.

Thoughts?

[Scott Moser (smoser)]

fixed in 0.7.8.

[Chris J Arges (arges)]

Hello Nathan, or anyone else affected,

Accepted cloud-init into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.7-31-g65ace7b-0ubuntu1~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Scott Moser (smoser)]

I've walked through the lxc test case as above with the -proposed package. Looks good.

[Martin Pitt (pitti)]

Hello Nathan, or anyone else affected,

Accepted cloud-init into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.8-1-g3705bb5-0ubuntu1~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Scott Moser (smoser)]

verified as per sru comments.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.8-1-g3705bb5-0ubuntu1~16.04.1

---------------
cloud-init (0.7.8-1-g3705bb5-0ubuntu1~16.04.1) xenial-proposed; urgency=medium

  * New upstream release 0.7.8.
  * New upstream snapshot.
    - systemd: put cloud-init.target After multi-user.target (LP: #1623868)

cloud-init (0.7.7-31-g65ace7b-0ubuntu1~16.04.2) xenial-proposed; urgency=medium

  * debian/control: add Breaks of older versions of walinuxagent (LP: #1623570)

cloud-init (0.7.7-31-g65ace7b-0ubuntu1~16.04.1) xenial-proposed; urgency=medium

  * debian/control: fix missing dependency on python3-serial,
    and make SmartOS datasource work.
  * debian/cloud-init.templates fix capitalisation in template so
    dpkg-reconfigure works to select OpenStack. (LP: #1575727)
  * d/README.source, d/control, d/new-upstream-snapshot, d/rules: sync
    with yakkety for changes due to move to git.
  * d/rules: change PYVER=python3 to PYVER=3 to adjust to upstream change.
  * debian/rules, debian/cloud-init.install: remove install file
    to ensure expected files are collected into cloud-init deb.
    (LP: #1615745)
  * debian/dirs: remove obsolete / unused file.
  * upstream move from bzr to git.
  * New upstream snapshot.
    - Allow link type of null in network_data.json [Jon Grimm] (LP: #1621968)
    - DataSourceOVF: fix user-data as base64 with python3 (LP: #1619394)
    - remove obsolete .bzrignore
    - systemd: Better support package and upgrade. (LP: #1576692, #1621336)
    - tests: cleanup tempdirs in apt_source tests
    - apt config conversion: treat empty string as not provided. (LP: #1621180)
    - Fix typo in default keys for phone_home [Roland Sommer] (LP: #1607810)
    - salt minion: update default pki directory for newer salt minion.
      (LP: #1609899)
    - bddeb: add --release flag to specify the release in changelog.
    - apt-config: allow both old and new format to be present.
      [Christian Ehrhardt] (LP: #1616831)
    - python2.6: fix dict comprehension usage in _lsb_release. [Joshua Harlow]
    - Add a module that can configure spacewalk. [Joshua Harlow]
    - add install option for openrc [Matthew Thode]
    - Generate a dummy bond name for OpenStack (LP: #1605749)
    - network: fix get_interface_mac for bond slave, read_sys_net for ENOTDIR
    - azure dhclient-hook cleanups
    - Minor cleanups to atomic_helper and add unit tests.
    - Fix Gentoo net config generation [Matthew Thode]
    - distros: fix get_primary_arch method use of os.uname [Andrew Jorgensen]
    - Apt: add new apt configuration format [Christian Ehrhardt]
    - Get Azure endpoint server from DHCP client [Brent Baude]
    - DigitalOcean: use the v1.json endpoint [Ben Howard]
    - MAAS: add vendor-data support (LP: #1612313)
    - Upgrade to a configobj package new enough to work [Joshua Harlow]
    - ConfigDrive: recognize 'tap' as a link type. (LP: #1610784)
    - NoCloud: fix bug providing network-interfaces via meta-data.
      (LP: 1577982)
    - Add distro tags on config modules that should have it [Joshua Harlow]
    - ChangeLog: update changelog for previous commit.
    - add ntp config module [Ryan Harper]
    - SmartOS: more improvement...

[Chris J Arges (arges)]

The verification of the Stable Release Update for cloud-init has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/2703