keystone charm has config-change hook failed

Bug #1592129 reported by Narinder Gupta on 2016-06-13
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack keystone charm
Medium
Unassigned
keystone (Juju Charms Collection)
Medium
Unassigned

Bug Description

deploy the openstack bundle on mitaka with xenial with juju 1.25.5 and faced the config-hook failed.

logs have been placed at
udo pastebinit unit-keystone-0.log
sudo: unable to resolve host juju-machine-1-lxc-4
http://paste.ubuntu.com/17302238/

pastebinit keystone.log
http://paste.ubuntu.com/17302289/

root@juju-machine-1-lxc-4:/var/log/apache2# pastebinit error.log
http://paste.ubuntu.com/17302326/

David Ames (thedac) on 2016-06-13
affects: charms → keystone (Juju Charms Collection)
Narinder Gupta (narindergupta) wrote :

charm location and bundles has been placed as well.

jenkins@intelpod5jumphost:~/joid/ci$ pastebinit fetch-charms.sh
http://paste.ubuntu.com/17302391/
jenkins@intelpod5jumphost:~/joid/ci$ pastebinit bundles.yaml
http://paste.ubuntu.com/17302397/
jenkins@intelpod5jumphost:~/joid/ci$

David Ames (thedac) wrote :
Download full text (6.8 KiB)

Race condition on restart_pid_check. Apache should be down but is not:

2016-06-13 18:00:45 INFO config-changed Traceback (most recent call last):
2016-06-13 18:00:45 INFO config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/config-changed", line 707, in <module>
2016-06-13 18:00:45 INFO config-changed main()
2016-06-13 18:00:45 INFO config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/config-changed", line 700, in main
2016-06-13 18:00:45 INFO config-changed hooks.execute(sys.argv)
2016-06-13 18:00:45 INFO config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/core/hookenv.py", line 717, in execute
2016-06-13 18:00:45 INFO config-changed self._hooks[hook_name]()
2016-06-13 18:00:45 INFO config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1574, in wrapped_f
2016-06-13 18:00:45 INFO config-changed restart_functions)
2016-06-13 18:00:45 INFO config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/core/host.py", line 486, in restart_on_change_helper
2016-06-13 18:00:45 INFO config-changed restart_functions[service_name](service_name)
2016-06-13 18:00:45 INFO config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 386, in restart_pid_check
2016-06-13 18:00:45 INFO config-changed check_pids_gone(ptable_string)
2016-06-13 18:00:45 INFO config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/core/decorators.py", line 42, in _retry_on_exception_inner_2
2016-06-13 18:00:45 INFO config-changed return f(*args, **kwargs)
2016-06-13 18:00:45 INFO config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 381, in check_pids_gone
2016-06-13 18:00:45 INFO config-changed assert(subprocess.call(["pgrep", svc_string]) == 1)
2016-06-13 18:00:45 INFO config-changed AssertionError

2016-06-13 18:21:09 INFO config-changed 16846
2016-06-13 18:21:09 INFO config-changed 16847
2016-06-13 18:21:09 INFO config-changed Traceback (most recent call last):
2016-06-13 18:21:09 INFO config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/config-changed", line 707, in <module>
2016-06-13 18:21:09 INFO config-changed main()
2016-06-13 18:21:09 INFO config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/config-changed", line 700, in main
2016-06-13 18:21:09 INFO config-changed hooks.execute(sys.argv)
2016-06-13 18:21:09 INFO config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/core/hookenv.py", line 717, in execute
2016-06-13 18:21:09 INFO config-changed self._hooks[hook_name]()
2016-06-13 18:21:09 INFO config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1574, in wrapped_f
2016-06-13 18:21:09 INFO config-changed restart_functions)
2016-06-13 18:21:09 INFO config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/core/host.py", line 475, in restart_on_change_helper
2016-06-13 18:21:09 INFO config-changed ...

Read more...

Changed in keystone (Juju Charms Collection):
status: New → Triaged
importance: Undecided → Critical
milestone: none → 16.07
David Ames (thedac) wrote :

apache2 is failing to stop. I think the key to this bug is the line in the log:

2016-06-13 17:59:53 INFO config-changed Job for apache2.service failed because the control process exited with error code. See "systemctl status apache2.service" and "journalctl -xe" for details.

I have yet to recreate this in the lab.

Narinder, if you see it again can you plese run the following commands on the keystone node and paste the results here:

systemctl status apache2.service
journalctl -xe

James Page (james-page) on 2016-07-14
Changed in keystone (Juju Charms Collection):
importance: Critical → Medium
Liam Young (gnuoy) on 2016-07-29
Changed in keystone (Juju Charms Collection):
milestone: 16.07 → 16.10
James Page (james-page) on 2016-10-14
Changed in keystone (Juju Charms Collection):
milestone: 16.10 → 17.01
Kevin Metz (pertinent) wrote :

I encountered this same issue. Editing the /etc/hosts file and adding the hostname to 127.0.0.1 resolved this. Ultimately it was an issue on the server with the /etc/resolve.conf file. Once this was fixed I rebooted the containers and everything seems working correctly now

James Page (james-page) on 2017-02-23
Changed in charm-keystone:
importance: Undecided → Medium
status: New → Triaged
Changed in keystone (Juju Charms Collection):
status: Triaged → Invalid
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers