nova-cloud-controller unit doesn' start when upgrading to 2.8.2 model

[Mark Cave-Ayland]

I recently upgraded our openstack from 2.6 to the latest 2.8.2 model and found that the nova-cloud-controller unit services didn't start. Removing and then re-adding the relations didn't seem to make any difference.

Investigating with help from members of the #openstack-charms IRC channel it was found that all of the systemd services for the cloud controller unit had been masked. With this knowledge I was able to manually unmask, enable and start all of the nova, memcached, haproxy and apache2 services which brought the unit back to life.

gnuoy suggested that it may be because the install hook was run as part of running "juju upgrade-charm" as part of the upgrade to 2.8.

[Mark Cave-Ayland (mark-cave-ayland)]

Here is the original output from /var/log/juju/unit-nova-cloud-controller.log after the 2.8 upgrade when the unit failed to start:

2020-08-05 06:53:33 INFO juju-log Registered config file: /etc/nova/nova.conf
2020-08-05 06:53:33 INFO juju-log Registered config file: /etc/nova/api-paste.ini
2020-08-05 06:53:33 INFO juju-log Registered config file: /etc/nova/vendor_data.json
2020-08-05 06:53:33 INFO juju-log Registered config file: /etc/haproxy/haproxy.cfg
2020-08-05 06:53:33 INFO juju-log Registered config file: /etc/apache2/sites-available/openstack_https_frontend.conf
2020-08-05 06:53:33 INFO juju-log Registered config file: /etc/apache2/ports.conf
2020-08-05 06:53:33 INFO juju-log Registered config file: /etc/memcached.conf
2020-08-05 06:53:33 INFO juju-log Registered config file: /etc/apache2/sites-enabled/wsgi-api-os-compute.conf
2020-08-05 06:53:33 INFO juju-log Registered config file: /etc/apache2/sites-enabled/wsgi-placement-api.conf
2020-08-05 06:53:33 INFO juju-log Registered config file: /etc/apache2/sites-enabled/wsgi-openstack-metadata.conf
2020-08-05 06:53:33 DEBUG juju-log Hardening function 'update_status'
2020-08-05 06:53:33 DEBUG juju-log No hardening applied to 'update_status'
2020-08-05 06:53:33 INFO juju-log Updating status.
2020-08-05 06:53:33 INFO juju-log Registered config file:
2020-08-05 06:53:33 DEBUG juju-log 0 section(s) found
2020-08-05 06:53:34 DEBUG juju-log Generating template context for amqp
2020-08-05 06:53:34 DEBUG update-status ERROR no relation id specified
2020-08-05 06:53:34 DEBUG juju-log Generating template context for image-service.
2020-08-05 06:53:34 DEBUG juju-log 0 section(s) found
2020-08-05 06:53:34 INFO juju-log HAProxy context is incomplete, this unit has no peers.
2020-08-05 06:53:34 DEBUG juju-log Generating template context for identity-service
2020-08-05 06:53:34 INFO juju-log Generating template context from neutron api relation
2020-08-05 06:53:34 DEBUG juju-log Generating template context for identity-service
2020-08-05 06:53:35 DEBUG juju-log Ensuring haproxy enabled in /etc/default/haproxy.
2020-08-05 06:53:35 INFO juju-log HAProxy context is incomplete, this unit has no peers.
2020-08-05 06:53:35 DEBUG juju-log 0 section(s) found
2020-08-05 06:53:35 INFO juju-log HAProxy context is incomplete, this unit has no peers.
2020-08-05 06:53:35 INFO juju-log HAProxy context is incomplete, this unit has no peers.
2020-08-05 06:53:35 INFO juju-log HAProxy context is incomplete, this unit has no peers.
2020-08-05 06:53:35 INFO juju-log Generating template context from neutron api relation
2020-08-05 06:53:35 DEBUG update-status inactive
2020-08-05 06:53:35 DEBUG update-status inactive
2020-08-05 06:53:35 DEBUG update-status inactive
2020-08-05 06:53:35 DEBUG update-status inactive
2020-08-05 06:53:35 DEBUG update-status active
2020-08-05 06:53:35 DEBUG update-status inactive
2020-08-05 06:53:35 DEBUG update-status active

[Mark Cave-Ayland (mark-cave-ayland)]

Attached is the log file from the period of the upgrade.

[Liam Young (gnuoy)]

The unit log definitely shows the install hook being run after the upgrade to 2.8.2:

ERROR must restart: an agent upgrade is available
2020-08-04 17:38:52 INFO juju.cmd supercommand.go:54 running jujud [2.8.2 0 1d5703e8a6cef8b08c24132548644f43bb468b93 gc go1.14.6]
2020-08-04 17:38:52 DEBUG juju.cmd supercommand.go:55 args: []string{"/var/lib/juju/tools/unit-nova-cloud-controller-3/jujud", "unit", "--data-dir", "/var/lib/juju", "--unit-name", "nova-cloud-controller/3", "--debug"}
2020-08-04 17:38:52 DEBUG juju.agent agent.go:583 read agent config, format "2.0"
2020-08-04 17:38:52 INFO juju.cmd.jujud agent.go:138 setting logging config to "<root>=WARNING;unit=DEBUG"
2020-08-04 17:39:05 DEBUG juju-log 0 section(s) found
2020-08-04 17:39:05 INFO juju-log Registered config file: /etc/nova/nova.conf
2020-08-04 17:39:05 INFO juju-log Registered config file: /etc/nova/api-paste.ini
2020-08-04 17:39:05 INFO juju-log Registered config file: /etc/nova/vendor_data.json
2020-08-04 17:39:05 INFO juju-log Registered config file: /etc/haproxy/haproxy.cfg
2020-08-04 17:39:05 INFO juju-log Registered config file: /etc/apache2/sites-available/openstack_https_frontend.conf
2020-08-04 17:39:05 INFO juju-log Registered config file: /etc/memcached.conf
2020-08-04 17:39:05 INFO juju-log Registered config file: /etc/apache2/sites-enabled/wsgi-api-os-compute.conf
2020-08-04 17:39:05 INFO juju-log Registered config file: /etc/apache2/sites-enabled/wsgi-placement-api.conf
2020-08-04 17:39:05 INFO juju-log Registered config file: /etc/apache2/sites-enabled/wsgi-openstack-metadata.conf
2020-08-04 17:39:05 DEBUG juju-log Hardening function 'install'
2020-08-04 17:39:05 DEBUG juju-log No hardening applied to 'install'
2020-08-04 17:39:05 INFO juju-log DEPRECATION WARNING: Function configure_installation_source is being removed on/around 2017-07 : use charmhelpers.fetch.add_source() instead.
2020-08-04 17:39:06 INFO juju-log Installing [] with options: ['--option=Dpkg::Options::=--force-confold']
2020-08-04 17:39:06 DEBUG install Reading package lists...

So far I have been unable to reproduce this. Upgrading to 2.8.1 seems to run all the relation hooks but did not seem to trigger the install hook. Still investigating.

[Liam Young (gnuoy)]

Running then install hook on a previously running node masks the services. That is a bug in the charm. The fact that the install hook is called is probably a bug in juju

[Liam Young (gnuoy)]

Raised a bug against Juju here: https://bugs.launchpad.net/juju/+bug/1890465