cloud-init useradd/groupadd fails on ubuntu-core-16 with readonly /etc/passwd

[Ryan Harper]

=== Begin SRU Template ===
[Impact]
When running under ubuntu-core 16 images, /etc/passwd is read-only.

If my user-data includes any non-default username, creation fails due to
the read-only nature of the image.

This is addressed by useradd/groupadd including a command line flag, --extrausers
which instructs the command to look for a different user/group database in
/var/lib/extrausers , which is writable in the ubuntu-core 16 image.

[Test Case]
In a snappy image that has cloud-init enabled, launch image with the
following user-data:
 #cloud-config
 users:
   - name: bob
     snapuser: <email address hidden>

And also:
 #cloud-config
 snappy:
   email: <email address hidden>

where '<email address hidden>' is your launchpad registered email address.
Assume you can log in.

[Regression Potential]
The code is intended to be backwards compatible and inert unless
cloud-config provided turns it on. It is also gated by a 'system_is_snappy'
method that checks if the system is snappy (ubuntu core).

Unit tests are provided, so regression should be somewhat reduced.

Some code was moved around to implement this, and a new config module was added.

[Other Info]
The upstream change made here is at [1]

[1] https://git.launchpad.net/cloud-init/commit?id=d8534561ba76db25b6fc0044eb1bfda63686e859

=== End SRU Template ===

When running under ubuntu-core 16 images, /etc/passwd is read-only.

If my user-data includes any non-default username, creation fails due to
the read-only nature of the image.

This is addressed by useradd/groupadd including a command line flag, --extrausers
which instructs the command to look for a different user/group database in
/var/lib/extrausers , which is writable in the ubuntu-core 16 image.

The cc_user_groups module though is not aware of this.

The Distro base-class could check if the system it's running on is snappy (see cc_snappy.py)
and if so, append the --extrausers parameter to the useradd/groupadd commands.

1) release is Xenial (ubuntu-core 16)
2) cloud-init present is: 0.7.7~bzr1256-0ubuntu1~16.04.1
3) useradd bob -m should create the user bob
4) useradd fails due to readonly /etc/{passwd,group,shadow}

[Scott Moser (smoser)]

fixed in trunk at d8534561ba76db25b6fc0044eb1bfda63686e859

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.8-27-g29348af-0ubuntu1

---------------
cloud-init (0.7.8-27-g29348af-0ubuntu1) zesty; urgency=medium

  * debian/cloud-init.templates: enable DigitalOcean by default [Ben Howard]
  * New upstream snapshot.
    - disk-config: udev settle after partitioning in gpt format. (LP: #1626243)
    - unittests: do not read system /etc/cloud/cloud.cfg.d (LP: #1635350)
    - Add documentation for logging features. [Wesley Wiedenmeier]
    - Add support for snap create-user on Ubuntu Core images. [Ryan Harper]
      (LP: #1619393)
    - Fix sshd restarts for rhel distros. [Jim Gorz] (LP: #1470433)
    - OpenNebula: replace 'ip' parsing with cloudinit.net usage.
    - Fix python2.6 things found running in centos 6.
    - Move user/group functions to new ug_util file [Joshua Harlow]
    - DigitalOcean: enable usage of data source by default.
    - update Gentoo initscripts to run in the correct order [Matthew Thode]
    - MAAS: improve the main of datasource to look at kernel cmdline config.
    - tests: silence the Cheetah UserWarning about NameMapper C version.

 -- Scott Moser <email address hidden> Tue, 25 Oct 2016 17:06:59 -0400

[Scott Moser (smoser)]

This change was not included in the changelog, but is being proposed to go back to xenial,
and should appear there with 0.7.8-34-ga1cdebd-0ubuntu1~16.04.1

[Steve Langasek (vorlon)]

Hello Ryan, or anyone else affected,

Accepted cloud-init into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.8-47-gb6561a1-0ubuntu1~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Ryan Harper (raharper)]

I've built an updated UC16 image with cloud-init from the xenial-proposed archive, booted it in local kvm as well OpenStack cloud and confirmed user creation and snapuser creation work as expected.

[Steve Langasek (vorlon)]

Hello Ryan, or anyone else affected,

Accepted cloud-init into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.8-49-g9e904bb-0ubuntu1~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Ryan Harper (raharper)]

I've verified 0.7.8-49 cloud-init from xenial-proposed archive in UC16 cloud-image, using core snap here:

https://launchpadlibrarian.net/294392203/core_16.04.1_amd64.manifest

Tested under local kvm as well OpenStack cloud and confirmed user creation and snapuser creation work as expected.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.8-49-g9e904bb-0ubuntu1~16.04.1

---------------
cloud-init (0.7.8-49-g9e904bb-0ubuntu1~16.04.1) xenial-proposed; urgency=medium

  * debian/cloud-init.postinst: update /etc/fstab on Azure to fix
    future resize operations. (LP: #1611074)
  * New upstream snapshot.
    - Add activate_datasource, for datasource specific code paths.
      (LP: #1611074)
    - systemd: cloud-init-local use RequiresMountsFor=/var/lib/cloud
      (LP: #1642062)

cloud-init (0.7.8-47-gb6561a1-0ubuntu1~16.04.1) xenial-proposed; urgency=medium

  * debian/cloud-init.templates: enable DigitalOcean by default [Ben Howard]
  * New upstream snapshot.
    - systemd/cloud-init-local.service:
      + replace 'Wants' and 'After' on local-fs.target with more granular
        After=systemd-remount-fs.service and RequiresMountsFor=/var/lib
        and Before=sysinit.target.
        This is done run sufficiently early enough to update /etc/fstab.
        (LP: #1611074)
      + add Before=NetworkManager.service so that cloud-init can render
        NetworkManager network config before it would apply them.
    - systemd/cloud-init.service:
      + add Before=sysinit.target and DefaultDependencies=no (LP: #1611074)
      + drop Requires=networking.service to work where networking.service is
        not needed.
      + add Conflicts=shutdown.target
      + drop unnecessary Wants=local-fs.target
    - net: support reading ipv6 dhcp config from initramfs [LaMont Jones]
      (LP: #1621615)
    - dmidecode: Allow dmidecode to be used on aarch64, and only attempt
      usage on x86, x86_64, and aarch64. [Robert Schweikert]
    - disk-config: udev settle after partitioning in gpt format.
      (LP: #1626243)
    - Add support for snap create-user on Ubuntu Core images. [Ryan Harper]
      (LP: #1619393)
    - Fix sshd restarts for rhel distros. [Jim Gorz]
    - Move user/group functions to new ug_util file [Joshua Harlow]
    - update Gentoo initscripts to run in the correct order [Matthew Thode]
    - MAAS: improve the debugging tool in datasource to consider
      config provided on kernel cmdline.
    - lxd: Update network config for LXD 2.3 [Stéphane Graber] (LP: #1640556)
    - Decode unicode types in decode_binary [Robert Schweikert]
    - Allow ephemeral drive to be unpartitioned [Paul Meyer]
    - subp: add 'update_env' argument which allows for more easily adding
      environment variables to a subprocess call.
    - Adjust mounts and disk configuration for systemd. (LP: #1611074)
    - DataSources:
      + Ec2: protect against non-dictionary in block-device-mapping.
      + AliYun: Add new datasource for Ali-Cloud ECS, that is
        available but not enabled by default [kaihuan.pkh]
      + DigitalOcean: use meta-data for network configuration and
        enable data source by default. [Ben Howard]
      + OpenNebula: replace parsing of 'ip' command with similar function
        available in cloudinit.net. This fixed unit tests when running
        in environment with no networking.
    - doc changes:
      + Add documentation on stages of boot.
      + make the RST files consistently formated and other improvements.
     ...

[Adam Conrad (adconrad)]

The verification of the Stable Release Update for cloud-init has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Chris Halse Rogers (raof)]

Hello Ryan, or anyone else affected,

Accepted cloud-init into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.8-49-g9e904bb-0ubuntu1~16.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Ryan Harper (raharper)]

This fix was needed for cloud-init in Ubuntu-Core 16 images which uses Xenial archive. There is not a UC16 snappy image based on Yakkety archive, so we do not have any users of this path in yakkety and no way to test in a non-existent image. I'm marking verification-done.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.8-49-g9e904bb-0ubuntu1~16.10.1

---------------
cloud-init (0.7.8-49-g9e904bb-0ubuntu1~16.10.1) yakkety; urgency=medium

  * debian/cloud-init.templates: enable DigitalOcean by default [Ben Howard]
  * debian/cloud-init.postinst: update /etc/fstab on Azure to fix
    future resize operations. (LP: #1611074)
  * New upstream snapshot.
    - systemd/cloud-init-local.service:
      + replace 'Wants' and 'After' on local-fs.target with more granular
        After=systemd-remount-fs.service and RequiresMountsFor=/var/lib
        and Before=sysinit.target.
        This is done run sufficiently early enough to update /etc/fstab.
        (LP: #1611074)
    - systemd/cloud-init.service:
      + add Before=sysinit.target and DefaultDependencies=no (LP: #1611074)
      + drop Requires=networking.service to work where networking.service is
        not needed.
      + add Conflicts=shutdown.target
      + drop unnecessary Wants=local-fs.target
    - net: support reading ipv6 dhcp config from initramfs [LaMont Jones]
      (LP: #1621615)
    - dmidecode: Allow dmidecode to be used on aarch64, and only attempt
      usage on x86, x86_64, and aarch64. [Robert Schweikert]
    - disk-config: udev settle after partitioning in gpt format.
      (LP: #1626243)
    - Add support for snap create-user on Ubuntu Core images. [Ryan Harper]
      (LP: #1619393)
    - Fix sshd restarts for rhel distros. [Jim Gorz]
    - Move user/group functions to new ug_util file [Joshua Harlow]
    - update Gentoo initscripts to run in the correct order [Matthew Thode]
    - MAAS: improve the debugging tool in datasource to consider
      config provided on kernel cmdline.
    - DataSources:
      + Ec2: protect against non-dictionary in block-device-mapping.
      + AliYun: Add new datasource for Ali-Cloud ECS, that is
        available but not enabled by default [kaihuan.pkh]
      + OpenNebula: replace parsing of 'ip' command with similar function
        available in cloudinit.net. This fixed unit tests when running
        in environment with no networking.
    - doc changes:
      + Add documentation on stages of boot.
      + make the RST files consistently formated and other improvements.
      + fixed example to not overwrite /etc/hosts [Chris Glass]
      + fix spelling / typos in ca_certs and scripts_vendor.
      + improve HACKING.rst file
      + Add documentation for logging features. [Wesley Wiedenmeier]
    - code style and unit test changes:
      + pep8: fix style errors reported by pycodestyle 2.1.0
      + pyflakes: fix issue with pyflakes 1.3 found in ubuntu zesty-proposed.
      + Add coverage dependency to bddeb to fix package build.
      + Add coverage collection to tox unit tests. [Joshua Powers]
      + do not read system /etc/cloud/cloud.cfg.d (LP: #1635350)
      + tests: silence the Cheetah UserWarning about NameMapper C version.
      + Fix python2.6 things found running in centos 6.

 -- Scott Moser <email address hidden> Tue, 22 Nov 2016 17:04:36 -0500

[Scott Moser (smoser)]

this was fixed in 0.7.9

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/2714