application-version-set may error during an install hook

[Chris MacNaughton]

During a deploy on the ceilometer charm (~openstack-charmers-next/ceilometer) on xenial (openstack queens), the ceilometer charm had an error calling application-version-set:

2018-11-06 19:31:04 DEBUG install ERROR connection is shut down
2018-11-06 19:31:04 DEBUG install Traceback (most recent call last):
2018-11-06 19:31:04 DEBUG install File "/var/lib/juju/agents/unit-ceilometer-0/charm/hooks/install.real", line 504, in <module>
2018-11-06 19:31:04 DEBUG install assess_status(CONFIGS)
2018-11-06 19:31:04 DEBUG install File "/var/lib/juju/agents/unit-ceilometer-0/charm/hooks/ceilometer_utils.py", line 501, in assess_status
2018-11-06 19:31:04 DEBUG install os_application_version_set(VERSION_PACKAGE)
2018-11-06 19:31:04 DEBUG install File "/var/lib/juju/agents/unit-ceilometer-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1580, in os_application_version_set
2018-11-06 19:31:04 DEBUG install application_version_set(application_version)
2018-11-06 19:31:04 DEBUG install File "/var/lib/juju/agents/unit-ceilometer-0/charm/hooks/charmhelpers/core/hookenv.py", line 995, in application_version_set
2018-11-06 19:31:04 DEBUG install subprocess.check_call(cmd)
2018-11-06 19:31:04 DEBUG install File "/usr/lib/python2.7/subprocess.py", line 541, in check_call
2018-11-06 19:31:04 DEBUG install raise CalledProcessError(retcode, cmd)
2018-11-06 19:31:04 DEBUG install subprocess.CalledProcessError: Command '['application-version-set', '10.0.1']' returned non-zero exit status 1
2018-11-06 19:31:04 ERROR juju.worker.uniter.operation runhook.go:117 hook "install" failed: exit status 1

[Tim Penhey (thumper)]

It looks like the underlying error is this:

2018-11-06 19:31:04 DEBUG install ERROR connection is shut down

It is possible that the controller that this unit was talking to was restarted. If that is the case, the hook will fail. When the controller comes back up and the connection is reestablished, the hook will be executed again (I think).

[Chris MacNaughton (chris.macnaughton)]

Tim, I wonder if it would be worth discussion potential reasons that the controller would randomly shut down - this wasn't on a long lived controller that might have undergone maintenance, it is on a controller in our CI environment that would have been, at most, one week old (and likely less than two days). We never intentionally shut them down except when we destroy and re-create them, and that never happens during a test run.

[Tim Penhey (thumper)]

Sure, did you want the conversation on this bug or in person?

[Pedro Guimarães (pguimaraes)]

Hi, I am facing this same issue. After application-version-set fails, jujud seems to shutdown and the unit is kept on "waiting" status.

Here is the full crashdump: https://drive.google.com/file/d/1BF8UDSeGP5bKXWq1tgWBIlTjoW_7gqO6/view?usp=sharing
Bundle: https://drive.google.com/file/d/1E09pKcJZWmVbodMxyW9dBJZTwoPq-MH1/view?usp=sharing

Restart jujud-unit seems to unblock the unit.
Is there a more effective solution?

[Canonical Juju QA Bot (juju-qa-bot)]

This bug has not been updated in 2 years, so we're marking it Low importance. If you believe this is incorrect, please update the importance.