Pike deployments failing when charm tries to perform server-list

Bug #1848307 reported by Liam Young on 2019-10-16
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Designate Charm
High
Unassigned

Bug Description

Deployments of designate in an HA configuration are failing with designate stuck in the following state:

00:17:14 designate/0* maintenance executing 15 172.17.107.54 9001/tcp Installation complete - awaiting next status
00:17:14 designate-hacluster/0* active idle 172.17.107.54 Unit is ready and clustered
00:17:14 designate/1 maintenance executing 16 172.17.107.33 9001/tcp Installation complete - awaiting next status
00:17:14 designate-hacluster/2 active idle 172.17.107.33 Unit is ready and clustered
00:17:14 designate/2 maintenance executing 17 172.17.107.5 9001/tcp Installation complete - awaiting next status
00:17:14 designate-hacluster/1 active idle 172.17.107.5 Unit is ready and clustered

Looking at the looks the hooks are stuck looping over the following error:

unit-designate-0: 07:38:34 DEBUG unit.designate/0.certificates-relation-changed for server in get_servers():
unit-designate-0: 07:38:34 DEBUG unit.designate/0.certificates-relation-changed File "reactive/designate_utils.py", line 121, in get_servers
unit-designate-0: 07:38:34 DEBUG unit.designate/0.certificates-relation-changed out, err = run_command(cmd)
unit-designate-0: 07:38:34 DEBUG unit.designate/0.certificates-relation-changed File "reactive/designate_utils.py", line 34, in run_command
unit-designate-0: 07:38:34 DEBUG unit.designate/0.certificates-relation-changed cmd, p.returncode, out, err))
unit-designate-0: 07:38:34 DEBUG unit.designate/0.certificates-relation-changed RuntimeError: ['designate', 'server-list', '-f', 'value'] failed, status code 1 stdout b'' stderr b'/usr/lib/python2.7/dist-packages/designateclient/cli/base.py:38: DeprecationWarning: The "designate" CLI is being deprecated in favour of the "openstack" CLI plugin. All designate API v2 commands are implemented there. When the v1 API is removed this CLI will stop functioning\n DeprecationWarning)\nERROR: Bad Request (HTTP 400)\n'
unit-designate-0: 07:38:34 INFO unit.designate/0.juju-log certificates:109: Retrying 'ensure_api_responding' 35 more times (delay=30)
unit-designate-0: 07:37:45 WARNING unit.designate/0.juju-log certificates:109: Checking API service is responding
unit-designate-0: 07:37:46 DEBUG unit.designate/0.certificates-relation-changed Traceback (most recent call last):
unit-designate-0: 07:37:46 DEBUG unit.designate/0.certificates-relation-changed File "reactive/designate_utils.py", line 169, in <module>
unit-designate-0: 07:37:46 DEBUG unit.designate/0.certificates-relation-changed commands[args.command]()
unit-designate-0: 07:37:46 DEBUG unit.designate/0.certificates-relation-changed File "reactive/designate_utils.py", line 137, in display_servers
unit-designate-0: 07:37:46 DEBUG unit.designate/0.certificates-relation-changed for server in get_servers():
unit-designate-0: 07:37:46 DEBUG unit.designate/0.certificates-relation-changed File "reactive/designate_utils.py", line 121, in get_servers
unit-designate-0: 07:37:46 DEBUG unit.designate/0.certificates-relation-changed out, err = run_command(cmd)
unit-designate-0: 07:37:46 DEBUG unit.designate/0.certificates-relation-changed File "reactive/designate_utils.py", line 34, in run_command
unit-designate-0: 07:37:46 DEBUG unit.designate/0.certificates-relation-changed cmd, p.returncode, out, err))
unit-designate-0: 07:37:46 DEBUG unit.designate/0.certificates-relation-changed RuntimeError: ['designate', 'server-list', '-f', 'value'] failed, status code 1 stdout b'' stderr b'/usr/lib/python2.7/dist-packages/designateclient/cli/base.py:38: DeprecationWarning: The "designate" CLI is being deprecated in favour of the "openstack" CLI plugin. All designate API v2 commands are implemented there. When the v1 API is removed this CLI will stop functioning\n DeprecationWarning)\nERROR: Bad Request (HTTP 400)\n'
unit-designate-0: 07:37:46 INFO unit.designate/0.juju-log certificates:109: Retrying 'ensure_api_responding' 37 more times (delay=20)
unit-designate-0: 07:38:06 WARNING unit.designate/0.juju-log certificates:109: Checking API service is responding
unit-designate-0: 07:38:07 DEBUG unit.designate/0.certificates-relation-changed Traceback (most recent call last):
unit-designate-0: 07:38:07 DEBUG unit.designate/0.certificates-relation-changed File "reactive/designate_utils.py", line 169, in <module>
unit-designate-0: 07:38:07 DEBUG unit.designate/0.certificates-relation-changed commands[args.command]()
unit-designate-0: 07:38:07 DEBUG unit.designate/0.certificates-relation-changed File "reactive/designate_utils.py", line 137, in display_servers
unit-designate-0: 07:38:07 DEBUG unit.designate/0.certificates-relation-changed for server in get_servers():
unit-designate-0: 07:38:07 DEBUG unit.designate/0.certificates-relation-changed File "reactive/designate_utils.py", line 121, in get_servers
unit-designate-0: 07:38:07 DEBUG unit.designate/0.certificates-relation-changed out, err = run_command(cmd)
unit-designate-0: 07:38:07 DEBUG unit.designate/0.certificates-relation-changed File "reactive/designate_utils.py", line 34, in run_command
unit-designate-0: 07:38:07 DEBUG unit.designate/0.certificates-relation-changed cmd, p.returncode, out, err))
unit-designate-0: 07:38:07 DEBUG unit.designate/0.certificates-relation-changed RuntimeError: ['designate', 'server-list', '-f', 'value'] failed, status code 1 stdout b'' stderr b'/usr/lib/python2.7/dist-packages/designateclient/cli/base.py:38: DeprecationWarning: The "designate" CLI is being deprecated in favour of the "openstack" CLI plugin. All designate API v2 commands are implemented there. When the v1 API is removed this CLI will stop functioning\n DeprecationWarning)\nERROR: Bad Request (HTTP 400)\n'
unit-designate-0: 07:38:07 INFO unit.designate/0.juju-log certificates:109: Retrying 'ensure_api_responding' 36 more times (delay=25)
unit-designate-0: 07:38:32 WARNING unit.designate/0.juju-log certificates:109: Checking API service is responding
unit-designate-0: 07:38:34 DEBUG unit.designate/0.certificates-relation-changed Traceback (most recent call last):
^C

Liam Young (gnuoy) wrote :

Looking into this further this appears to happen after designate has received a certificate via the certificates relation.

Changed in charm-designate:
status: New → Triaged
importance: Undecided → High
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers