[19.04][Queens -> Rocky] Upgrading to Rocky resulted in "Services not running that should be: designate-producer"
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Designate |
Fix Released
|
Undecided
|
Corey Bryant | ||
OpenStack Charms Deployment Guide |
Fix Released
|
High
|
Unassigned | ||
OpenStack Designate Charm |
Fix Released
|
Undecided
|
Chris MacNaughton | ||
Ubuntu Cloud Archive |
Fix Released
|
High
|
Unassigned | ||
Rocky |
Won't Fix
|
High
|
Unassigned | ||
Stein |
Won't Fix
|
High
|
Unassigned | ||
Train |
Fix Released
|
High
|
Unassigned | ||
memcached-charm |
Fix Released
|
High
|
Unassigned | ||
designate (Ubuntu) |
Fix Released
|
High
|
Unassigned | ||
Disco |
Won't Fix
|
High
|
Unassigned | ||
Eoan |
Fix Released
|
High
|
Unassigned |
Bug Description
Designate has to use Worker and Producer as of Rocky as the support for zone manager and pool manager was up for removal in Rocky. This was addressed in https:/
During a Queens to Rocky upgrade I ran into an issue with designate-producer being down (see the details below).
I was able to successfully start designate-producer by hand. It looks like there were several attempts to restart the service by systemd itself ("Scheduled restart job, restart counter is at 5.") which eventually failed when the failcount became higher than 5. Debugging mode was disabled so I did not see anything in the producer log.
designate/0* blocked idle 2/lxd/2 10.232.46.153 9001/tcp Services not running that should be: designate-producer
hacluster-
systemctl list-unit-files | grep designate-
designate-
designate-
designate-
designate-
designate-
designate-
designate-
ubuntu@
https:/
ubuntu@
172877 bash /lib/systemd/
172878 bash /lib/systemd/
172884 /var/lib/
172886 /var/lib/
392388 /usr/bin/python3.6 /usr/bin/
392399 /usr/bin/python3.6 /usr/bin/
392405 /usr/bin/python3.6 /usr/bin/
392411 /usr/bin/python3.6 /usr/bin/
392415 /usr/bin/python3.6 /usr/bin/
392423 /usr/bin/python3.6 /usr/bin/
392481 /usr/bin/python3.6 /usr/bin/
392482 /usr/bin/python3.6 /usr/bin/
392483 /usr/bin/python3.6 /usr/bin/
392484 /usr/bin/python3.6 /usr/bin/
----------
designate/0 unit log:
2019-05-09 00:20:38 DEBUG openstack-upgrade Setting up designate-producer (1:7.0.
2019-05-09 00:20:38 DEBUG openstack-upgrade Created symlink /etc/systemd/
# journalctl -u designate-producer
journalctl -u designate-producer | grep start
May 09 00:20:40 juju-eeda89-2-lxd-2 systemd[1]: designate-
May 09 00:20:40 juju-eeda89-2-lxd-2 systemd[1]: designate-
May 09 00:20:41 juju-eeda89-2-lxd-2 systemd[1]: designate-
May 09 00:20:41 juju-eeda89-2-lxd-2 systemd[1]: designate-
May 09 00:20:43 juju-eeda89-2-lxd-2 systemd[1]: designate-
May 09 00:20:43 juju-eeda89-2-lxd-2 systemd[1]: designate-
May 09 00:20:44 juju-eeda89-2-lxd-2 systemd[1]: designate-
May 09 00:20:44 juju-eeda89-2-lxd-2 systemd[1]: designate-
May 09 00:20:46 juju-eeda89-2-lxd-2 systemd[1]: designate-
May 09 00:20:46 juju-eeda89-2-lxd-2 systemd[1]: designate-
May 09 00:20:46 juju-eeda89-2-lxd-2 systemd[1]: Failed to start OpenStack Designate DNSaaS producer.
designate/0 unit log (continued):
2019-05-09 00:20:58 INFO juju-log Purging ['designate-
2019-05-09 00:20:59 DEBUG openstack-upgrade Removing designate-
2019-05-09 00:21:15 DEBUG openstack-upgrade Removing designate-
2019-05-09 00:21:31 DEBUG openstack-upgrade Removing python-designate (1:7.0.
2019-05-09 00:21:31 DEBUG openstack-upgrade update-
-------------------
grep producer /var/log/
2019-05-09 00:18:43 DEBUG openstack-upgrade update-
2019-05-09 00:19:58 INFO juju-log Installing ['designate-agent', 'designate-api', 'designate-
2019-05-09 00:19:58 DEBUG openstack-upgrade designate-producer designate-worker python3-amqp python3-anyjson
2019-05-09 00:20:09 DEBUG openstack-upgrade Get:112 http://
2019-05-09 00:20:20 DEBUG openstack-upgrade Selecting previously unselected package designate-producer.
2019-05-09 00:20:20 DEBUG openstack-upgrade Preparing to unpack .../105-
2019-05-09 00:20:20 DEBUG openstack-upgrade Unpacking designate-producer (1:7.0.
2019-05-09 00:20:38 DEBUG openstack-upgrade Setting up designate-producer (1:7.0.
2019-05-09 00:20:38 DEBUG openstack-upgrade Created symlink /etc/systemd/
2019-05-09 00:21:31 DEBUG openstack-upgrade update-
Related branches
- Alvaro Uria (community): Approve
-
Diff: 101 lines (+46/-1)4 files modifiedhooks/cache-relation-changed (+1/-0)
hooks/memcached_hooks.py (+10/-1)
templates/memcached.conf (+4/-0)
unit_tests/test_memcached_hooks.py (+31/-0)
description: | updated |
Changed in charm-designate: | |
assignee: | nobody → Liam Young (gnuoy) |
tags: | added: py3 |
Changed in charm-designate: | |
milestone: | 19.07 → none |
Changed in designate (Ubuntu Eoan): | |
status: | New → Fix Released |
importance: | Undecided → High |
Changed in designate (Ubuntu Disco): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in designate (Ubuntu): | |
status: | Triaged → Fix Committed |
status: | Fix Committed → Fix Released |
Changed in cloud-archive: | |
status: | New → Fix Released |
importance: | Undecided → High |
Changed in charm-designate: | |
assignee: | nobody → Chris MacNaughton (chris.macnaughton) |
Changed in charm-memcached: | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in charm-memcached: | |
milestone: | none → 20.08 |
status: | Triaged → Fix Committed |
Changed in charm-designate: | |
milestone: | none → 20.08 |
Changed in charm-designate: | |
status: | Fix Committed → Fix Released |
Changed in charm-deployment-guide: | |
status: | Fix Committed → Fix Released |
TRIAGE:
Seems we have a race such that designate-producer is not ready during the upgrade.
Guarantee designate-producer has everything it needs or is restarted at the end of the upgrade process.