error: private-address/public-address not set (1.25.5)

[James Page]

We've seen this quite a bit over the last few weeks since 1.25.5 was released; deployments will fail as units cannot get either a private-address or a public-address at some point during hook execution - for example:

unit-cinder-0[3537]: 2016-05-18 10:26:07 INFO unit.cinder/0.install logger.go:40 error: private-address not set
unit-cinder-0[3537]: 2016-05-18 10:26:07 INFO unit.cinder/0.install logger.go:40 Traceback (most recent call last):
unit-cinder-0[3537]: 2016-05-18 10:26:07 INFO unit.cinder/0.install logger.go:40 File "/var/lib/juju/agents/unit-cinder-0/charm/hooks/install.real", line 579, in <module>
unit-cinder-0[3537]: 2016-05-18 10:26:07 INFO unit.cinder/0.install logger.go:40 assess_status(CONFIGS)
unit-cinder-0[3537]: 2016-05-18 10:26:07 INFO unit.cinder/0.install logger.go:40 File "/var/lib/juju/agents/unit-cinder-0/charm/hooks/cinder_utils.py", line 886, in assess_status
unit-cinder-0[3537]: 2016-05-18 10:26:07 INFO unit.cinder/0.install logger.go:40 assess_status_func(configs)()
unit-cinder-0[3537]: 2016-05-18 10:26:07 INFO unit.cinder/0.install logger.go:40 File "/var/lib/juju/agents/unit-cinder-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1530, in _assess_status_func
unit-cinder-0[3537]: 2016-05-18 10:26:07 INFO unit.cinder/0.install logger.go:40 state, message = _determine_os_workload_status(*args, **kwargs)
unit-cinder-0[3537]: 2016-05-18 10:26:07 INFO unit.cinder/0.install logger.go:40 File "/var/lib/juju/agents/unit-cinder-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 930, in _determine_os_workload_status
unit-cinder-0[3537]: 2016-05-18 10:26:07 INFO unit.cinder/0.install logger.go:40 configs, required_interfaces)
unit-cinder-0[3537]: 2016-05-18 10:26:07 INFO unit.cinder/0.install logger.go:40 File "/var/lib/juju/agents/unit-cinder-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 984, in _ows_check_generic_interfaces
unit-cinder-0[3537]: 2016-05-18 10:26:07 INFO unit.cinder/0.install logger.go:40 required_interfaces)
unit-cinder-0[3537]: 2016-05-18 10:26:07 INFO unit.cinder/0.install logger.go:40 File "/var/lib/juju/agents/unit-cinder-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1265, in incomplete_relation_data
unit-cinder-0[3537]: 2016-05-18 10:26:07 INFO unit.cinder/0.install logger.go:40 complete_ctxts = configs.complete_contexts()
unit-cinder-0[3537]: 2016-05-18 10:26:07 INFO unit.cinder/0.install logger.go:40 File "/var/lib/juju/agents/unit-cinder-0/charm/hooks/charmhelpers/contrib/openstack/templating.py", line 295, in complete_contexts
unit-cinder-0[3537]: 2016-05-18 10:26:07 INFO unit.cinder/0.install logger.go:40 for i in six.itervalues(self.templates)]
unit-cinder-0[3537]: 2016-05-18 10:26:07 INFO unit.cinder/0.install logger.go:40 File "/var/lib/juju/agents/unit-cinder-0/charm/hooks/charmhelpers/contrib/openstack/templating.py", line 120, in complete_contexts
unit-cinder-0[3537]: 2016-05-18 10:26:07 INFO unit.cinder/0.install logger.go:40 self.context()
unit-cinder-0[3537]: 2016-05-18 10:26:07 INFO unit.cinder/0.install logger.go:40 File "/var/lib/juju/agents/unit-cinder-0/charm/hooks/charmhelpers/contrib/openstack/templating.py", line 105, in context
unit-cinder-0[3537]: 2016-05-18 10:26:07 INFO unit.cinder/0.install logger.go:40 _ctxt = context()
unit-cinder-0[3537]: 2016-05-18 10:26:07 INFO unit.cinder/0.install logger.go:40 File "/var/lib/juju/agents/unit-cinder-0/charm/hooks/charmhelpers/contrib/openstack/context.py", line 592, in __call__
unit-cinder-0[3537]: 2016-05-18 10:26:07 INFO unit.cinder/0.install logger.go:40 addr = get_host_ip(unit_get('private-address'))
unit-cinder-0[3537]: 2016-05-18 10:26:07 INFO unit.cinder/0.install logger.go:40 File "/var/lib/juju/agents/unit-cinder-0/charm/hooks/charmhelpers/core/hookenv.py", line 74, in wrapper
unit-cinder-0[3537]: 2016-05-18 10:26:07 INFO unit.cinder/0.install logger.go:40 res = func(*args, **kwargs)
unit-cinder-0[3537]: 2016-05-18 10:26:07 INFO unit.cinder/0.install logger.go:40 File "/var/lib/juju/agents/unit-cinder-0/charm/hooks/charmhelpers/core/hookenv.py", line 624, in unit_get
unit-cinder-0[3537]: 2016-05-18 10:26:07 INFO unit.cinder/0.install logger.go:40 return json.loads(subprocess.check_output(_args).decode('UTF-8'))
unit-cinder-0[3537]: 2016-05-18 10:26:07 INFO unit.cinder/0.install logger.go:40 File "/usr/lib/python2.7/subprocess.py", line 573, in check_output
unit-cinder-0[3537]: 2016-05-18 10:26:07 INFO unit.cinder/0.install logger.go:40 raise CalledProcessError(retcode, cmd, output=output)
unit-cinder-0[3537]: 2016-05-18 10:26:07 INFO unit.cinder/0.install logger.go:40 subprocess.CalledProcessError: Command '['unit-get', '--format=json', 'private-address']' returned non-zero exit status 1

This does not happen consistently, but we're hitting it multiple times a day atm with just a light development load.

[James Page (james-page)]

juju status on the install is odd as well:

https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline/openstack/charm-ceph/317910/1/2016-05-18_08-48-05/test_charm_amulet_smoke/juju-stat-tabular-collect.txt

[Edward Hope-Morley (hopem)]

We are also seeing this a lot with Juju 1.25.5 and its a pain since there does not appear to be a way to resolve it even if the unit is rebooted and definitely has networking correctly setup. It's possibly related to https://bugs.launchpad.net/juju-core/+bug/1484105 which has been around for a very long time now.

[James Page (james-page)]

Full index from failed test run:

https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline/openstack/charm-ceph/317910/1/2016-05-18_08-48-05/index.html

[Dimiter Naydenov (dimitern)]

On IRC I've suggested to enable trace logging and get the machine-0.log, to have more context analysing the issues as it's quite intermittent and hard to reproduce otherwise.

[Ryan Beisner (1chb1n)]

We caught it here:

https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_amulet_full/openstack/charm-ceph-osd/318611/1/2016-05-19_14-00-50/index.html

2016-05-19 14:52:30 DEBUG worker.uniter.jujuc server.go:173 hook context id "cinder/0-install-2328604234966462243"; dir "/var/lib/juju/agents/unit-cinder-0/charm"
2016-05-19 14:52:30 INFO install error: private-address not set
2016-05-19 14:52:30 INFO install Traceback (most recent call last):
2016-05-19 14:52:30 INFO install File "/var/lib/juju/agents/unit-cinder-0/charm/hooks/install.real", line 579, in <module>
2016-05-19 14:52:30 INFO install assess_status(CONFIGS)
2016-05-19 14:52:30 INFO install File "/var/lib/juju/agents/unit-cinder-0/charm/hooks/cinder_utils.py", line 886, in assess_status
2016-05-19 14:52:30 INFO install assess_status_func(configs)()
2016-05-19 14:52:30 INFO install File "/var/lib/juju/agents/unit-cinder-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1530, in _assess_status_func
2016-05-19 14:52:30 INFO install state, message = _determine_os_workload_status(*args, **kwargs)
2016-05-19 14:52:30 INFO install File "/var/lib/juju/agents/unit-cinder-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 930, in _determine_os_workload_status
2016-05-19 14:52:30 INFO install configs, required_interfaces)
2016-05-19 14:52:30 INFO install File "/var/lib/juju/agents/unit-cinder-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 984, in _ows_check_generic_interfaces
2016-05-19 14:52:30 INFO install required_interfaces)
2016-05-19 14:52:30 INFO install File "/var/lib/juju/agents/unit-cinder-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1265, in incomplete_relation_data
2016-05-19 14:52:30 INFO install complete_ctxts = configs.complete_contexts()
2016-05-19 14:52:30 INFO install File "/var/lib/juju/agents/unit-cinder-0/charm/hooks/charmhelpers/contrib/openstack/templating.py", line 295, in complete_contexts
2016-05-19 14:52:30 INFO install for i in six.itervalues(self.templates)]
2016-05-19 14:52:30 INFO install File "/var/lib/juju/agents/unit-cinder-0/charm/hooks/charmhelpers/contrib/openstack/templating.py", line 120, in complete_contexts
2016-05-19 14:52:30 INFO install self.context()
2016-05-19 14:52:30 INFO install File "/var/lib/juju/agents/unit-cinder-0/charm/hooks/charmhelpers/contrib/openstack/templating.py", line 105, in context
2016-05-19 14:52:30 INFO install _ctxt = context()
2016-05-19 14:52:30 INFO install File "/var/lib/juju/agents/unit-cinder-0/charm/hooks/charmhelpers/contrib/openstack/context.py", line 592, in __call__
2016-05-19 14:52:30 INFO install addr = get_host_ip(unit_get('private-address'))
2016-05-19 14:52:30 INFO install File "/var/lib/juju/agents/unit-cinder-0/charm/hooks/charmhelpers/core/hookenv.py", line 74, in wrapper
2016-05-19 14:52:30 INFO install res = func(*args, **kwargs)
2016-05-19 14:52:30 INFO install File "/var/lib/juju/agents/unit-cinder-0/charm/hooks/charmhelpers/core/hookenv.py", line 624, in unit_get
2016-05-19 14:52:30 INFO install return json.loads(subprocess.check_output(_args).decode('UTF-8'))
2016-...

[Ryan Beisner (1chb1n)]

FYI, machine 0 log with TRACE during ^ that deploy is at:
https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_amulet_full/openstack/charm-ceph-osd/318611/1/2016-05-19_14-00-50/test_charm_amulet_full/logs/0-var-log.tar.bz2