When using Juju to deploy openstack using nova-compute charm revision 287, approximately 10% of the time, the deployment fails with nova-compute/0 going into error state (hook failed: "config-changed". This was specifically tested deploying nova-compute into an LXD container.
This appears to be because /etc/init is not created due to a race condition intermittedly. Note that logging onto the offending LXD container and manually creating the directory (sudo mkdir /etc/init) and resolving the charm state is sufficient to continue deployment.
Extract from /var/log/juju/unit-nova-compute-0.log:
2018-11-26 17:21:51 DEBUG config-changed ERROR no relation id specified
2018-11-26 17:21:51 INFO juju-log CPU architecture: x86_64
2018-11-26 17:21:51 INFO juju-log Loaded template from templates/qemu.conf
2018-11-26 17:21:51 INFO juju-log Rendering from template: /etc/libvirt/qemu.conf
2018-11-26 17:21:51 INFO juju-log Wrote template /etc/libvirt/qemu.conf.
2018-11-26 17:21:51 INFO juju-log Loaded template from templates/qemu-kvm
2018-11-26 17:21:51 INFO juju-log Rendering from template: /etc/default/qemu-kvm
2018-11-26 17:21:51 INFO juju-log Wrote template /etc/default/qemu-kvm.
2018-11-26 17:21:52 INFO juju-log Loaded template from templates/libvirtd.conf
2018-11-26 17:21:52 INFO juju-log Rendering from template: /etc/libvirt/libvirtd.conf
2018-11-26 17:21:52 INFO juju-log Wrote template /etc/libvirt/libvirtd.conf.
2018-11-26 17:21:52 INFO juju-log Loaded template from templates/libvirt-bin
2018-11-26 17:21:52 INFO juju-log Rendering from template: /etc/default/libvirt-bin
2018-11-26 17:21:52 INFO juju-log Wrote template /etc/default/libvirt-bin.
2018-11-26 17:21:52 INFO juju-log Loaded template from templates/libvirt-bin.override
2018-11-26 17:21:52 INFO juju-log Rendering from template: /etc/init/libvirt-bin.override
2018-11-26 17:21:52 DEBUG config-changed Traceback (most recent call last):
2018-11-26 17:21:52 DEBUG config-changed File "/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/config-changed", line 579, in <module>
2018-11-26 17:21:52 DEBUG config-changed main()
2018-11-26 17:21:52 DEBUG config-changed File "/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/config-changed", line 572, in main
2018-11-26 17:21:52 DEBUG config-changed hooks.execute(sys.argv)
2018-11-26 17:21:52 DEBUG config-changed File "/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/charmhelpers/core/hookenv.py", line 847, in execute
2018-11-26 17:21:52 DEBUG config-changed self._hooks[hook_name]()
2018-11-26 17:21:52 DEBUG config-changed File "/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1466, in wrapped_f
2018-11-26 17:21:52 DEBUG config-changed restart_functions)
2018-11-26 17:21:52 DEBUG config-changed File "/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/charmhelpers/core/host.py", line 730, in restart_on_change_helper
2018-11-26 17:21:52 DEBUG config-changed r = lambda_f()
2018-11-26 17:21:52 DEBUG config-changed File "/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1465, in <lambda>
2018-11-26 17:21:52 DEBUG config-changed (lambda: f(*args, **kwargs)), restart_map, stopstart,
2018-11-26 17:21:52 DEBUG config-changed File "/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/charmhelpers/contrib/hardening/harden.py", line 79, in _harden_inner2
2018-11-26 17:21:52 DEBUG config-changed return f(*args, **kwargs)
2018-11-26 17:21:52 DEBUG config-changed File "/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/config-changed", line 225, in config_changed
2018-11-26 17:21:52 DEBUG config-changed CONFIGS.write_all()
2018-11-26 17:21:52 DEBUG config-changed File "/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/charmhelpers/contrib/openstack/templating.py", line 334, in write_all
2018-11-26 17:21:52 DEBUG config-changed [self.write(k) for k in six.iterkeys(self.templates)]
2018-11-26 17:21:52 DEBUG config-changed File "/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/charmhelpers/contrib/openstack/templating.py", line 334, in <listcomp>
2018-11-26 17:21:52 DEBUG config-changed [self.write(k) for k in six.iterkeys(self.templates)]
2018-11-26 17:21:52 DEBUG config-changed File "/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/charmhelpers/contrib/openstack/templating.py", line 325, in write
2018-11-26 17:21:52 DEBUG config-changed with open(config_file, 'wb') as out:
2018-11-26 17:21:52 DEBUG config-changed FileNotFoundError: [Errno 2] No such file or directory: '/etc/init/libvirt-bin.override'
ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: nova-compute 2:18.0.1-0ubuntu1~cloud0 [origin: Canonical]
ProcVersionSignature: Ubuntu 4.15.0-1024.25-gcp 4.15.18
Uname: Linux 4.15.0-1024-gcp x86_64
NonfreeKernelModules: iptable_raw br_netfilter kvm_intel openvswitch nsh binfmt_misc veth ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_CHECKSUM xt_comment xt_tcpudp iptable_mangle xt_nat iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat vxlan ip6_udp_tunnel udp_tunnel bridge stp llc ufs msdos xfs ip6table_filter ip6_tables iptable_filter nls_iso8859_1 kvm input_leds serio_raw irqbypass pvpanic sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 raid0 multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd glue_helper cryptd psmouse virtio_net
ApportVersion: 2.20.9-0ubuntu7.5
Architecture: amd64
CrashDB:
{
"impl": "launchpad",
"project": "cloud-archive",
"bug_pattern_url": "http://people.canonical.com/~ubuntu-archive/bugpatterns/bugpatterns.xml",
}
Date: Mon Nov 26 17:19:27 2018
PackageArchitecture: all
ProcEnviron:
TERM=xterm-256color
PATH=(custom, no user)
LANG=C.UTF-8
SHELL=/bin/bash
SourcePackage: nova
UpgradeStatus: No upgrade log present (probably fresh install)
Moving to charm as this is where the problem is.