During a Queens to Rocky upgrade I ran into an issue with designate-producer being down (see the details below).
I was able to successfully start designate-producer by hand. It looks like there were several attempts to restart the service by systemd itself ("Scheduled restart job, restart counter is at 5.") which eventually failed when the failcount became higher than 5. Debugging mode was disabled so I did not see anything in the producer log.
designate/0* blocked idle 2/lxd/2 10.232.46.153 9001/tcp Services not running that should be: designate-producer
hacluster-designate/0* active idle 10.232.46.153 Unit is ready and clustered
2019-05-09 00:20:38 DEBUG openstack-upgrade Setting up designate-producer (1:7.0.0-0ubuntu1~cloud0) ...
2019-05-09 00:20:38 DEBUG openstack-upgrade Created symlink /etc/systemd/system/multi-user.target.wants/designate-producer.service → /lib/systemd/system/designate-producer.service.
# journalctl -u designate-producer
journalctl -u designate-producer | grep start
May 09 00:20:40 juju-eeda89-2-lxd-2 systemd[1]: designate-producer.service: Service hold-off time over, scheduling restart.
May 09 00:20:40 juju-eeda89-2-lxd-2 systemd[1]: designate-producer.service: Scheduled restart job, restart counter is at 1.
May 09 00:20:41 juju-eeda89-2-lxd-2 systemd[1]: designate-producer.service: Service hold-off time over, scheduling restart.
May 09 00:20:41 juju-eeda89-2-lxd-2 systemd[1]: designate-producer.service: Scheduled restart job, restart counter is at 2.
May 09 00:20:43 juju-eeda89-2-lxd-2 systemd[1]: designate-producer.service: Service hold-off time over, scheduling restart.
May 09 00:20:43 juju-eeda89-2-lxd-2 systemd[1]: designate-producer.service: Scheduled restart job, restart counter is at 3.
May 09 00:20:44 juju-eeda89-2-lxd-2 systemd[1]: designate-producer.service: Service hold-off time over, scheduling restart.
May 09 00:20:44 juju-eeda89-2-lxd-2 systemd[1]: designate-producer.service: Scheduled restart job, restart counter is at 4.
May 09 00:20:46 juju-eeda89-2-lxd-2 systemd[1]: designate-producer.service: Service hold-off time over, scheduling restart.
May 09 00:20:46 juju-eeda89-2-lxd-2 systemd[1]: designate-producer.service: Scheduled restart job, restart counter is at 5.
May 09 00:20:46 juju-eeda89-2-lxd-2 systemd[1]: Failed to start OpenStack Designate DNSaaS producer.
designate/0 unit log (continued):
2019-05-09 00:20:58 INFO juju-log Purging ['designate-pool-manager', 'designate-zone-manager', 'python-designate', 'python-memcache']
2019-05-09 00:21:31 DEBUG openstack-upgrade update-alternatives: using /usr/bin/python3-designate-producer to provide /usr/bin/designate-producer (designate-producer) in auto mode
-------------------
grep producer /var/log/juju/unit-designate-0.log
2019-05-09 00:18:43 DEBUG openstack-upgrade update-alternatives: using /usr/bin/python2-designate-producer to provide /usr/bin/designate-producer (designate-producer) in auto mode
2019-05-09 00:19:58 INFO juju-log Installing ['designate-agent', 'designate-api', 'designate-central', 'designate-common', 'designate-mdns', 'designate-worker', 'designate-sink', 'designate-producer', 'bind9utils', 'python3-designate', 'python-apt', 'memcached', 'python3-memcache', 'haproxy', 'apache2'] with options: ['--option', 'Dpkg::Options::=--force-confnew', '--option', 'Dpkg::Options::=--force-confdef']
2019-05-09 00:19:58 DEBUG openstack-upgrade designate-producer designate-worker python3-amqp python3-anyjson
2019-05-09 00:20:09 DEBUG openstack-upgrade Get:112 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-updates/rocky/main amd64 designate-producer all 1:7.0.0-0ubuntu1~cloud0 [10.6 kB]
2019-05-09 00:20:20 DEBUG openstack-upgrade Selecting previously unselected package designate-producer.
2019-05-09 00:20:20 DEBUG openstack-upgrade Preparing to unpack .../105-designate-producer_1%3a7.0.0-0ubuntu1~cloud0_all.deb ...
2019-05-09 00:20:20 DEBUG openstack-upgrade Unpacking designate-producer (1:7.0.0-0ubuntu1~cloud0) ...
2019-05-09 00:20:38 DEBUG openstack-upgrade Setting up designate-producer (1:7.0.0-0ubuntu1~cloud0) ...
2019-05-09 00:20:38 DEBUG openstack-upgrade Created symlink /etc/systemd/system/multi-user.target.wants/designate-producer.service → /lib/systemd/system/designate-producer.service.
2019-05-09 00:21:31 DEBUG openstack-upgrade update-alternatives: using /usr/bin/python3-designate-producer to provide /usr/bin/designate-producer (designate-producer) in auto mode
Designate has to use Worker and Producer as of Rocky as the support for zone manager and pool manager was up for removal in Rocky. This was addressed in https:/ /bugs.launchpad .net/charm- designate/ +bug/1773190
During a Queens to Rocky upgrade I ran into an issue with designate-producer being down (see the details below).
I was able to successfully start designate-producer by hand. It looks like there were several attempts to restart the service by systemd itself ("Scheduled restart job, restart counter is at 5.") which eventually failed when the failcount became higher than 5. Debugging mode was disabled so I did not see anything in the producer log.
designate/0* blocked idle 2/lxd/2 10.232.46.153 9001/tcp Services not running that should be: designate-producer designate/ 0* active idle 10.232.46.153 Unit is ready and clustered
hacluster-
systemctl list-unit-files | grep designate- agent.service enabled api.service enabled central. service enabled mdns.service enabled producer. service enabled sink.service enabled worker. service enabled
designate-
designate-
designate-
designate-
designate-
designate-
designate-
ubuntu@ juju-eeda89- 2-lxd-2: ~$ journalctl -u designate- producer. service /paste. ubuntu. com/p/WRXYvfynn d/
https:/
ubuntu@ juju-eeda89- 2-lxd-2: ~$ pgrep -af designate system/ jujud-unit- hacluster- designate- 0/exec- start.sh system/ jujud-unit- designate- 0/exec- start.sh juju/tools/ unit-designate- 0/jujud unit --data-dir /var/lib/juju --unit-name designate/0 --debug juju/tools/ unit-hacluster- designate- 0/jujud unit --data-dir /var/lib/juju --unit-name hacluster- designate/ 0 --debug designate- mdns --config- file=/etc/ designate/ designate. conf --log-file= /var/log/ designate/ designate- mdns.log designate- agent --config- file=/etc/ designate/ designate. conf --log-file= /var/log/ designate/ designate- agent.log designate- worker --config- file=/etc/ designate/ designate. conf --log-file= /var/log/ designate/ designate- worker. log designate- central --config- file=/etc/ designate/ designate. conf --log-file= /var/log/ designate/ designate- central. log designate- sink --config- file=/etc/ designate/ designate. conf --log-file= /var/log/ designate/ designate- sink.log designate- api --config- file=/etc/ designate/ designate. conf --log-file= /var/log/ designate/ designate- api.log designate- central --config- file=/etc/ designate/ designate. conf --log-file= /var/log/ designate/ designate- central. log designate- central --config- file=/etc/ designate/ designate. conf --log-file= /var/log/ designate/ designate- central. log designate- central --config- file=/etc/ designate/ designate. conf --log-file= /var/log/ designate/ designate- central. log designate- central --config- file=/etc/ designate/ designate. conf --log-file= /var/log/ designate/ designate- central. log
172877 bash /lib/systemd/
172878 bash /lib/systemd/
172884 /var/lib/
172886 /var/lib/
392388 /usr/bin/python3.6 /usr/bin/
392399 /usr/bin/python3.6 /usr/bin/
392405 /usr/bin/python3.6 /usr/bin/
392411 /usr/bin/python3.6 /usr/bin/
392415 /usr/bin/python3.6 /usr/bin/
392423 /usr/bin/python3.6 /usr/bin/
392481 /usr/bin/python3.6 /usr/bin/
392482 /usr/bin/python3.6 /usr/bin/
392483 /usr/bin/python3.6 /usr/bin/
392484 /usr/bin/python3.6 /usr/bin/
----------
designate/0 unit log:
2019-05-09 00:20:38 DEBUG openstack-upgrade Setting up designate-producer (1:7.0. 0-0ubuntu1~ cloud0) ... system/ multi-user. target. wants/designate -producer. service → /lib/systemd/ system/ designate- producer. service.
2019-05-09 00:20:38 DEBUG openstack-upgrade Created symlink /etc/systemd/
# journalctl -u designate-producer producer. service: Service hold-off time over, scheduling restart. producer. service: Scheduled restart job, restart counter is at 1. producer. service: Service hold-off time over, scheduling restart. producer. service: Scheduled restart job, restart counter is at 2. producer. service: Service hold-off time over, scheduling restart. producer. service: Scheduled restart job, restart counter is at 3. producer. service: Service hold-off time over, scheduling restart. producer. service: Scheduled restart job, restart counter is at 4. producer. service: Service hold-off time over, scheduling restart. producer. service: Scheduled restart job, restart counter is at 5.
journalctl -u designate-producer | grep start
May 09 00:20:40 juju-eeda89-2-lxd-2 systemd[1]: designate-
May 09 00:20:40 juju-eeda89-2-lxd-2 systemd[1]: designate-
May 09 00:20:41 juju-eeda89-2-lxd-2 systemd[1]: designate-
May 09 00:20:41 juju-eeda89-2-lxd-2 systemd[1]: designate-
May 09 00:20:43 juju-eeda89-2-lxd-2 systemd[1]: designate-
May 09 00:20:43 juju-eeda89-2-lxd-2 systemd[1]: designate-
May 09 00:20:44 juju-eeda89-2-lxd-2 systemd[1]: designate-
May 09 00:20:44 juju-eeda89-2-lxd-2 systemd[1]: designate-
May 09 00:20:46 juju-eeda89-2-lxd-2 systemd[1]: designate-
May 09 00:20:46 juju-eeda89-2-lxd-2 systemd[1]: designate-
May 09 00:20:46 juju-eeda89-2-lxd-2 systemd[1]: Failed to start OpenStack Designate DNSaaS producer.
designate/0 unit log (continued):
2019-05-09 00:20:58 INFO juju-log Purging ['designate- pool-manager' , 'designate- zone-manager' , 'python-designate', 'python-memcache']
2019-05-09 00:20:59 DEBUG openstack-upgrade Removing designate- pool-manager (1:7.0. 0-0ubuntu1~ cloud0) ... zone-manager (1:7.0. 0-0ubuntu1~ cloud0) ... 0-0ubuntu1~ cloud0) ...
2019-05-09 00:21:15 DEBUG openstack-upgrade Removing designate-
2019-05-09 00:21:31 DEBUG openstack-upgrade Removing python-designate (1:7.0.
2019-05-09 00:21:31 DEBUG openstack-upgrade update- alternatives: using /usr/bin/ python3- designate- producer to provide /usr/bin/ designate- producer (designate- producer) in auto mode
-------------------
grep producer /var/log/ juju/unit- designate- 0.log alternatives: using /usr/bin/ python2- designate- producer to provide /usr/bin/ designate- producer (designate- producer) in auto mode central' , 'designate-common', 'designate-mdns', 'designate-worker', 'designate-sink', 'designate- producer' , 'bind9utils', 'python3- designate' , 'python-apt', 'memcached', 'python3-memcache', 'haproxy', 'apache2'] with options: ['--option', 'Dpkg:: Options: :=--force- confnew' , '--option', 'Dpkg:: Options: :=--force- confdef' ] ubuntu- cloud.archive. canonical. com/ubuntu bionic- updates/ rocky/main amd64 designate-producer all 1:7.0.0- 0ubuntu1~ cloud0 [10.6 kB] designate- producer_ 1%3a7.0. 0-0ubuntu1~ cloud0_ all.deb ... 0-0ubuntu1~ cloud0) ... 0-0ubuntu1~ cloud0) ... system/ multi-user. target. wants/designate -producer. service → /lib/systemd/ system/ designate- producer. service. alternatives: using /usr/bin/ python3- designate- producer to provide /usr/bin/ designate- producer (designate- producer) in auto mode
2019-05-09 00:18:43 DEBUG openstack-upgrade update-
2019-05-09 00:19:58 INFO juju-log Installing ['designate-agent', 'designate-api', 'designate-
2019-05-09 00:19:58 DEBUG openstack-upgrade designate-producer designate-worker python3-amqp python3-anyjson
2019-05-09 00:20:09 DEBUG openstack-upgrade Get:112 http://
2019-05-09 00:20:20 DEBUG openstack-upgrade Selecting previously unselected package designate-producer.
2019-05-09 00:20:20 DEBUG openstack-upgrade Preparing to unpack .../105-
2019-05-09 00:20:20 DEBUG openstack-upgrade Unpacking designate-producer (1:7.0.
2019-05-09 00:20:38 DEBUG openstack-upgrade Setting up designate-producer (1:7.0.
2019-05-09 00:20:38 DEBUG openstack-upgrade Created symlink /etc/systemd/
2019-05-09 00:21:31 DEBUG openstack-upgrade update-
------- ------- ------- -----
I was able to start designate-producer