bionic cloud-init 18.2 WARNING Juju's 'runcmd' stanza

Bug #1764264 reported by John A Meinel on 2018-04-16
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
cloud-init
Medium
Scott Moser
juju
High
Unassigned
2.3
High
Unassigned
cloud-init (Ubuntu)
Undecided
Unassigned
Nominated for Bionic by Chad Smith
Nominated for Cosmic by Chad Smith

Bug Description

I haven't quite figured out what is wrong, but I tried bootstrapping bionic with Juju 2.3.6 (proposed) today. I had been successfully bootstrapping on LXD bionic as of last week. This was my first attempt to bootstrap on a MAAS image of bionic.

The cloud init version reported in /var/log/cloud-init-output.log is listed as 18.2

(It may be that it has been "successfully" bootstrapping, but this error has been in the logs and we just didn't notice it.)

Cloud-init v. 18.2 running 'modules:config' at Mon, 16 Apr 2018 05:51:08 +0000. Up 28.17 seconds.
2018-04-16 05:51:08,730 - util.py[WARNING]: Running module apt-configure (<module 'cloudinit.config.cc_apt_configure' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_apt_configure.py'>) failed
2018-04-16 05:51:08,930 - schema.py[WARNING]: Invalid config:
runcmd: ['set -xe', "mkdir -p '/var/lib/juju'\ncat > '/var/lib/juju/MAASmachine.txt' << 'EOF'\n'hostname: nuc7\n'\nEOF\nchmod 0755 '/var/lib/juju/MAASmachine.txt'", 'set -xe', "install -D -m 644 /dev/null '/etc/systemd/system/juju-clean-shutdown.service'", "printf '%s\\n' '\n[Unit]\nDescription=Stop all network interfaces on shutdown\nDefaultDependencies=false\nAfter=final.target\n\n[Service]\nType=oneshot\nExecStart=/sbin/ifdown -a -v --force\nStandardOutput=tty\nStandardError=tty\n\n[Install]\nWantedBy=final.target\n' > '/etc/systemd/system/juju-clean-shutdown.service'", "/bin/systemctl enable '/etc/systemd/system/juju-clean-shutdown.service'", "install -D -m 644 /dev/null '/var/lib/juju/nonce.txt'", "printf '%s\\n' 'user-admin:bootstrap' > '/var/lib/juju/nonce.txt'"] has non-unique elements

I wasn't able to easily figure out what the schema is for cloud-init, as it seems to be read from a file. I imagine it is available somewhere.

I don't know if we're doing something wrong, or if the schema is incorrectly stating that "runcmd" cannot have the same bit of text twice. I'm guessing its complaining because we pass "set -xe" in 2 different places?

Related branches

John A Meinel (jameinel) wrote :
Download full text (5.2 KiB)

I think for Juju this is a case where we do different setup on MAAS than on LXD. Specifically w/ the same 2.3.6 and LXD I see:
Cloud-init v. 18.2 running 'modules:config' at Mon, 16 Apr 2018 06:11:54 +0000. Up 16.00 seconds.
+ install -D -m 644 /dev/null /etc/systemd/system/juju-clean-shutdown.service
+ printf %s\n
[Unit]
Description=Stop all network interfaces on shutdown
DefaultDependencies=false
After=final.target

[Service]
Type=oneshot
ExecStart=/sbin/ifdown -a -v --force
StandardOutput=tty
StandardError=tty

[Install]
WantedBy=final.target

+ /bin/systemctl enable /etc/systemd/system/juju-clean-shutdown.service
Created symlink /etc/systemd/system/final.target.wants/juju-clean-shutdown.service → /etc/systemd/system/juju-clean-shutdown.service.
+ install -D -m 644 /dev/null /var/lib/juju/nonce.txt
+ printf %s\n user-admin:bootstrap
Cloud-init v. 18.2 running 'modules:final' at Mon, 16 Apr 2018 06:11:55 +0000. Up 17.00 seconds.
Cloud-init v. 18.2 finished at Mon, 16 Apr 2018 06:11:55 +0000. Datasource DataSourceNoCloud [seed=/var/lib/cloud/seed/nocloud-net][dsmode=net]. Up 17.00 seconds

while on MAAS I had:
Cloud-init v. 18.2 running 'modules:config' at Mon, 16 Apr 2018 05:51:08 +0000. Up 28.17 seconds.
2018-04-16 05:51:08,730 - util.py[WARNING]: Running module apt-configure (<module 'cloudinit.config.cc_apt_configure' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_apt_configure.py'>) failed
2018-04-16 05:51:08,930 - schema.py[WARNING]: Invalid config:
runcmd: ['set -xe', "mkdir -p '/var/lib/juju'\ncat > '/var/lib/juju/MAASmachine.txt' << 'EOF'\n'hostname: nuc7\n'\nEOF\nchmod 0755 '/var/lib/juju/MAASmachine.txt'", 'set -xe', "install -D -m 644 /dev/null '/etc/systemd/system/juju-clean-shutdown.service'", "printf '%s\\n' '\n[Unit]\nDescription=Stop all network interfaces on shutdown\nDefaultDependencies=false\nAfter=final.target\n\n[Service]\nType=oneshot\nExecStart=/sbin/ifdown -a -v --force\nStandardOutput=tty\nStandardError=tty\n\n[Install]\nWantedBy=final.target\n' > '/etc/systemd/system/juju-clean-shutdown.service'", "/bin/systemctl enable '/etc/systemd/system/juju-clean-shutdown.service'", "install -D -m 644 /dev/null '/var/lib/juju/nonce.txt'", "printf '%s\\n' 'user-admin:bootstrap' > '/var/lib/juju/nonce.txt'"] has non-unique elements
Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease
Hit:2 http://archive.ubuntu.com/ubuntu bionic-updates InRelease
Hit:3 http://archive.ubuntu.com/ubuntu bionic-backports InRelease
Hit:4 http://archive.ubuntu.com/ubuntu bionic-security InRelease
Get:5 http://archive.ubuntu.com/ubuntu bionic/main Translation-en [516 kB]
Get:6 http://archive.ubuntu.com/ubuntu bionic/restricted Translation-en [3452 B]
Get:7 http://archive.ubuntu.com/ubuntu bionic/universe Translation-en [4942 kB]
Get:8 http://archive.ubuntu.com/ubuntu bionic/multiverse Translation-en [108 kB]
Fetched 5570 kB in 3s (2055 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
Suggested packages:
  ifupdown
The following NEW packages will be installed:
  bridge-utils
0 upgraded, 1 newly installed, 0 to remove and 1 not ...

Read more...

John A Meinel (jameinel) on 2018-04-16
Changed in juju:
status: New → Triaged
importance: Undecided → High
John A Meinel (jameinel) on 2018-04-16
summary: - bionic cloud-init 18.2 refuses Juju's 'runcmd' stanza
+ bionic cloud-init 18.2 WARNING Juju's 'runcmd' stanza
John A Meinel (jameinel) wrote :

Note that after running, things seem to have actually worked, but "systemctl list-units" seems to say that cloud-init is flagged as failed.

John A Meinel (jameinel) wrote :

As near as I can tell, the cloud-init lines:
2018-04-16 07:14:36,809 - util.py[WARNING]: Running module apt-configure (<module 'cloudinit.config.cc_apt_configure' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_apt_configure.py'>) failed
2018-04-16 07:14:37,036 - schema.py[WARNING]: Invalid config:
...

Are just warnings and not actually failing. Certainly the script itself seems to be run.

It does seem like a cloud-init schema validation bug if it wants to claim that you can't repeat the same content in "runcmd".

Consider a script like:

if [ $a == "a" ] ; then
  a()
fi
if [ $b == "b" ] ; then
  b()
fi

In that case you'll have 'fi' repeated 2x, and trigger the same failure.

I think both 'runcmd' and 'bootcmd' should be allowed to have duplicate entries.

Scott Moser (smoser) wrote :

The issue with runcmd and bootcmd is confirmed. you're right there.
but in your log, you have an error (probably to /var/log/cloud-init-output.log) for your apt configure module that is fatal.

Scott Moser (smoser) on 2018-04-19
Changed in cloud-init:
status: New → In Progress
assignee: nobody → Scott Moser (smoser)
importance: Undecided → Medium
Scott Moser (smoser) wrote :

on a mostly unrelated note, you're kind of relying on an implementation detail of cloud-init's runcmd behavior.
runcmd: ['set -xe', "mkdir -p '/var/lib/juju'\ncat > '/var/lib/juju/MAASmachine.txt' << 'EOF'\n'hostname: nuc7\n'\nEOF\nchmod 0755 '/var/lib/juju/MAASmachine.txt'", 'set -xe', ..]

I'd much rather see that as:
runcmd:
 - [sh, '-c', 'set -xe\nmkdir -p\ncat >'/var/lib/...']

the difference being that you are assuming the 'set -xe' from the first entry in runcmd changes the state of the shell that is interpreting 'mkdir -p'.

I'd like to make a change in the future for your command to essentially get invoked as:
 sh -c 'set -xe'
 sh -c 'mkdir -p ...'
 sh -c '....'

Chad Smith (chad.smith) wrote :

An upstream commit landed for this bug.

To view that commit see the following URL:
https://git.launchpad.net/cloud-init/commit/?id=6811926f

Changed in cloud-init:
status: In Progress → Fix Committed

We can certainly change our assumption that all entries are a single
script, and instead break it up into each entry in the list is an
independent script.
However, that does make quoting, etc, a bit trickier. Coupled with things
like "python inside bash inside a string inside yaml" it gets...
interesting.

Anyway, if you want us to make the switch, we should. As it should be
backward compatible and maintain forward compatibility.

On Thu, Apr 19, 2018 at 6:09 AM, Scott Moser <email address hidden>
wrote:

> on a mostly unrelated note, you're kind of relying on an implementation
> detail of cloud-init's runcmd behavior.
> runcmd: ['set -xe', "mkdir -p '/var/lib/juju'\ncat >
> '/var/lib/juju/MAASmachine.txt' << 'EOF'\n'hostname: nuc7\n'\nEOF\nchmod
> 0755 '/var/lib/juju/MAASmachine.txt'", 'set -xe', ..]
>
> I'd much rather see that as:
> runcmd:
> - [sh, '-c', 'set -xe\nmkdir -p\ncat >'/var/lib/...']
>
> the difference being that you are assuming the 'set -xe' from the first
> entry in runcmd changes the state of the shell that is interpreting
> 'mkdir -p'.
>
> I'd like to make a change in the future for your command to essentially
> get invoked as:
> sh -c 'set -xe'
> sh -c 'mkdir -p ...'
> sh -c '....'
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1764264
>
> Title:
> bionic cloud-init 18.2 WARNING Juju's 'runcmd' stanza
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/cloud-init/+bug/1764264/+subscriptions
>

This bug is believed to be fixed in cloud-init (Ubuntu) in version 18.2-27-g6ef92c98-0ubuntu1~18.04.1. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

Changed in cloud-init (Ubuntu):
status: New → Fix Released

This bug is believed to be fixed in cloud-init in version 18.3. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

Changed in cloud-init:
status: Fix Committed → Fix Released
Changed in juju:
status: Triaged → Incomplete
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers