bionic cloud-init 18.2 WARNING Juju's 'runcmd' stanza

[John A Meinel]

I haven't quite figured out what is wrong, but I tried bootstrapping bionic with Juju 2.3.6 (proposed) today. I had been successfully bootstrapping on LXD bionic as of last week. This was my first attempt to bootstrap on a MAAS image of bionic.

The cloud init version reported in /var/log/cloud-init-output.log is listed as 18.2

(It may be that it has been "successfully" bootstrapping, but this error has been in the logs and we just didn't notice it.)

Cloud-init v. 18.2 running 'modules:config' at Mon, 16 Apr 2018 05:51:08 +0000. Up 28.17 seconds.
2018-04-16 05:51:08,730 - util.py[WARNING]: Running module apt-configure (<module 'cloudinit.config.cc_apt_configure' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_apt_configure.py'>) failed
2018-04-16 05:51:08,930 - schema.py[WARNING]: Invalid config:
runcmd: ['set -xe', "mkdir -p '/var/lib/juju'\ncat > '/var/lib/juju/MAASmachine.txt' << 'EOF'\n'hostname: nuc7\n'\nEOF\nchmod 0755 '/var/lib/juju/MAASmachine.txt'", 'set -xe', "install -D -m 644 /dev/null '/etc/systemd/system/juju-clean-shutdown.service'", "printf '%s\\n' '\n[Unit]\nDescription=Stop all network interfaces on shutdown\nDefaultDependencies=false\nAfter=final.target\n\n[Service]\nType=oneshot\nExecStart=/sbin/ifdown -a -v --force\nStandardOutput=tty\nStandardError=tty\n\n[Install]\nWantedBy=final.target\n' > '/etc/systemd/system/juju-clean-shutdown.service'", "/bin/systemctl enable '/etc/systemd/system/juju-clean-shutdown.service'", "install -D -m 644 /dev/null '/var/lib/juju/nonce.txt'", "printf '%s\\n' 'user-admin:bootstrap' > '/var/lib/juju/nonce.txt'"] has non-unique elements

I wasn't able to easily figure out what the schema is for cloud-init, as it seems to be read from a file. I imagine it is available somewhere.

I don't know if we're doing something wrong, or if the schema is incorrectly stating that "runcmd" cannot have the same bit of text twice. I'm guessing its complaining because we pass "set -xe" in 2 different places?

[John A Meinel (jameinel)]

I think for Juju this is a case where we do different setup on MAAS than on LXD. Specifically w/ the same 2.3.6 and LXD I see:
Cloud-init v. 18.2 running 'modules:config' at Mon, 16 Apr 2018 06:11:54 +0000. Up 16.00 seconds.
+ install -D -m 644 /dev/null /etc/systemd/system/juju-clean-shutdown.service
+ printf %s\n
[Unit]
Description=Stop all network interfaces on shutdown
DefaultDependencies=false
After=final.target

[Service]
Type=oneshot
ExecStart=/sbin/ifdown -a -v --force
StandardOutput=tty
StandardError=tty

[Install]
WantedBy=final.target

+ /bin/systemctl enable /etc/systemd/system/juju-clean-shutdown.service
Created symlink /etc/systemd/system/final.target.wants/juju-clean-shutdown.service → /etc/systemd/system/juju-clean-shutdown.service.
+ install -D -m 644 /dev/null /var/lib/juju/nonce.txt
+ printf %s\n user-admin:bootstrap
Cloud-init v. 18.2 running 'modules:final' at Mon, 16 Apr 2018 06:11:55 +0000. Up 17.00 seconds.
Cloud-init v. 18.2 finished at Mon, 16 Apr 2018 06:11:55 +0000. Datasource DataSourceNoCloud [seed=/var/lib/cloud/seed/nocloud-net][dsmode=net]. Up 17.00 seconds

while on MAAS I had:
Cloud-init v. 18.2 running 'modules:config' at Mon, 16 Apr 2018 05:51:08 +0000. Up 28.17 seconds.
2018-04-16 05:51:08,730 - util.py[WARNING]: Running module apt-configure (<module 'cloudinit.config.cc_apt_configure' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_apt_configure.py'>) failed
2018-04-16 05:51:08,930 - schema.py[WARNING]: Invalid config:
runcmd: ['set -xe', "mkdir -p '/var/lib/juju'\ncat > '/var/lib/juju/MAASmachine.txt' << 'EOF'\n'hostname: nuc7\n'\nEOF\nchmod 0755 '/var/lib/juju/MAASmachine.txt'", 'set -xe', "install -D -m 644 /dev/null '/etc/systemd/system/juju-clean-shutdown.service'", "printf '%s\\n' '\n[Unit]\nDescription=Stop all network interfaces on shutdown\nDefaultDependencies=false\nAfter=final.target\n\n[Service]\nType=oneshot\nExecStart=/sbin/ifdown -a -v --force\nStandardOutput=tty\nStandardError=tty\n\n[Install]\nWantedBy=final.target\n' > '/etc/systemd/system/juju-clean-shutdown.service'", "/bin/systemctl enable '/etc/systemd/system/juju-clean-shutdown.service'", "install -D -m 644 /dev/null '/var/lib/juju/nonce.txt'", "printf '%s\\n' 'user-admin:bootstrap' > '/var/lib/juju/nonce.txt'"] has non-unique elements
Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease
Hit:2 http://archive.ubuntu.com/ubuntu bionic-updates InRelease
Hit:3 http://archive.ubuntu.com/ubuntu bionic-backports InRelease
Hit:4 http://archive.ubuntu.com/ubuntu bionic-security InRelease
Get:5 http://archive.ubuntu.com/ubuntu bionic/main Translation-en [516 kB]
Get:6 http://archive.ubuntu.com/ubuntu bionic/restricted Translation-en [3452 B]
Get:7 http://archive.ubuntu.com/ubuntu bionic/universe Translation-en [4942 kB]
Get:8 http://archive.ubuntu.com/ubuntu bionic/multiverse Translation-en [108 kB]
Fetched 5570 kB in 3s (2055 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
Suggested packages:
  ifupdown
The following NEW packages will be installed:
  bridge-utils
0 upgraded, 1 newly installed, 0 to remove and 1 not ...

[John A Meinel (jameinel)]

Note that after running, things seem to have actually worked, but "systemctl list-units" seems to say that cloud-init is flagged as failed.

[John A Meinel (jameinel)]

As near as I can tell, the cloud-init lines:
2018-04-16 07:14:36,809 - util.py[WARNING]: Running module apt-configure (<module 'cloudinit.config.cc_apt_configure' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_apt_configure.py'>) failed
2018-04-16 07:14:37,036 - schema.py[WARNING]: Invalid config:
...

Are just warnings and not actually failing. Certainly the script itself seems to be run.

It does seem like a cloud-init schema validation bug if it wants to claim that you can't repeat the same content in "runcmd".

Consider a script like:

if [ $a == "a" ] ; then
  a()
fi
if [ $b == "b" ] ; then
  b()
fi

In that case you'll have 'fi' repeated 2x, and trigger the same failure.

I think both 'runcmd' and 'bootcmd' should be allowed to have duplicate entries.

[Scott Moser (smoser)]

The issue with runcmd and bootcmd is confirmed. you're right there.
but in your log, you have an error (probably to /var/log/cloud-init-output.log) for your apt configure module that is fatal.

[Scott Moser (smoser)]

on a mostly unrelated note, you're kind of relying on an implementation detail of cloud-init's runcmd behavior.
runcmd: ['set -xe', "mkdir -p '/var/lib/juju'\ncat > '/var/lib/juju/MAASmachine.txt' << 'EOF'\n'hostname: nuc7\n'\nEOF\nchmod 0755 '/var/lib/juju/MAASmachine.txt'", 'set -xe', ..]

I'd much rather see that as:
runcmd:
 - [sh, '-c', 'set -xe\nmkdir -p\ncat >'/var/lib/...']

the difference being that you are assuming the 'set -xe' from the first entry in runcmd changes the state of the shell that is interpreting 'mkdir -p'.

I'd like to make a change in the future for your command to essentially get invoked as:
 sh -c 'set -xe'
 sh -c 'mkdir -p ...'
 sh -c '....'

[Chad Smith (chad.smith)]

An upstream commit landed for this bug.

To view that commit see the following URL:
https://git.launchpad.net/cloud-init/commit/?id=6811926f

[John A Meinel (jameinel)]

We can certainly change our assumption that all entries are a single
script, and instead break it up into each entry in the list is an
independent script.
However, that does make quoting, etc, a bit trickier. Coupled with things
like "python inside bash inside a string inside yaml" it gets...
interesting.

Anyway, if you want us to make the switch, we should. As it should be
backward compatible and maintain forward compatibility.

On Thu, Apr 19, 2018 at 6:09 AM, Scott Moser <email address hidden>
wrote:

> on a mostly unrelated note, you're kind of relying on an implementation
> detail of cloud-init's runcmd behavior.
> runcmd: ['set -xe', "mkdir -p '/var/lib/juju'\ncat >
> '/var/lib/juju/MAASmachine.txt' << 'EOF'\n'hostname: nuc7\n'\nEOF\nchmod
> 0755 '/var/lib/juju/MAASmachine.txt'", 'set -xe', ..]
>
> I'd much rather see that as:
> runcmd:
> - [sh, '-c', 'set -xe\nmkdir -p\ncat >'/var/lib/...']
>
> the difference being that you are assuming the 'set -xe' from the first
> entry in runcmd changes the state of the shell that is interpreting
> 'mkdir -p'.
>
> I'd like to make a change in the future for your command to essentially
> get invoked as:
> sh -c 'set -xe'
> sh -c 'mkdir -p ...'
> sh -c '....'
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1764264
>
> Title:
> bionic cloud-init 18.2 WARNING Juju's 'runcmd' stanza
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/cloud-init/+bug/1764264/+subscriptions
>

[Chad Smith (chad.smith)]

This bug is believed to be fixed in cloud-init (Ubuntu) in version 18.2-27-g6ef92c98-0ubuntu1~18.04.1. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

[Scott Moser (smoser)]

This bug is believed to be fixed in cloud-init in version 18.3. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

[Anastasia (anastasia-macmood)]

Removing form milestone to let the report expire.

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/3154