| 2019-05-10 09:37:36 |
Dmitrii Shcherbakov |
bug |
|
|
added bug |
| 2019-05-10 09:37:51 |
Dmitrii Shcherbakov |
description |
Designate has to use Worker and Producer as of Rocky as the support for zone manager and pool manager was up for removal in Rocky. This was addressed in https://bugs.launchpad.net/charm-designate/+bug/1773190
During a Queens to Rocky upgrade I ran into an issue with designate-producer being down (see the details below).
I was able to successfully start designate-producer by hand. It looks like there were several attempts to restart the service by systemd itself ("Scheduled restart job, restart counter is at 5.") which eventually failed when the failcount became higher than 5. Debugging mode was disabled so I did not see anything in the producer log.
designate/0* blocked idle 2/lxd/2 10.232.46.153 9001/tcp Services not running that should be: designate-producer
hacluster-designate/0* active idle 10.232.46.153 Unit is ready and clustered
systemctl list-unit-files | grep designate-
designate-agent.service enabled
designate-api.service enabled
designate-central.service enabled
designate-mdns.service enabled
designate-producer.service enabled
designate-sink.service enabled
designate-worker.service enabled
ubuntu@juju-eeda89-2-lxd-2:~$ journalctl -u designate-producer.service
https://paste.ubuntu.com/p/WRXYvfynnd/
ubuntu@juju-eeda89-2-lxd-2:~$ pgrep -af designate
172877 bash /lib/systemd/system/jujud-unit-hacluster-designate-0/exec-start.sh
172878 bash /lib/systemd/system/jujud-unit-designate-0/exec-start.sh
172884 /var/lib/juju/tools/unit-designate-0/jujud unit --data-dir /var/lib/juju --unit-name designate/0 --debug
172886 /var/lib/juju/tools/unit-hacluster-designate-0/jujud unit --data-dir /var/lib/juju --unit-name hacluster-designate/0 --debug
392388 /usr/bin/python3.6 /usr/bin/designate-mdns --config-file=/etc/designate/designate.conf --log-file=/var/log/designate/designate-mdns.log
392399 /usr/bin/python3.6 /usr/bin/designate-agent --config-file=/etc/designate/designate.conf --log-file=/var/log/designate/designate-agent.log
392405 /usr/bin/python3.6 /usr/bin/designate-worker --config-file=/etc/designate/designate.conf --log-file=/var/log/designate/designate-worker.log
392411 /usr/bin/python3.6 /usr/bin/designate-central --config-file=/etc/designate/designate.conf --log-file=/var/log/designate/designate-central.log
392415 /usr/bin/python3.6 /usr/bin/designate-sink --config-file=/etc/designate/designate.conf --log-file=/var/log/designate/designate-sink.log
392423 /usr/bin/python3.6 /usr/bin/designate-api --config-file=/etc/designate/designate.conf --log-file=/var/log/designate/designate-api.log
392481 /usr/bin/python3.6 /usr/bin/designate-central --config-file=/etc/designate/designate.conf --log-file=/var/log/designate/designate-central.log
392482 /usr/bin/python3.6 /usr/bin/designate-central --config-file=/etc/designate/designate.conf --log-file=/var/log/designate/designate-central.log
392483 /usr/bin/python3.6 /usr/bin/designate-central --config-file=/etc/designate/designate.conf --log-file=/var/log/designate/designate-central.log
392484 /usr/bin/python3.6 /usr/bin/designate-central --config-file=/etc/designate/designate.conf --log-file=/var/log/designate/designate-central.log
----------
designate/0 unit log:
2019-05-09 00:20:38 DEBUG openstack-upgrade Setting up designate-producer (1:7.0.0-0ubuntu1~cloud0) ...
2019-05-09 00:20:38 DEBUG openstack-upgrade Created symlink /etc/systemd/system/multi-user.target.wants/designate-producer.service → /lib/systemd/system/designate-producer.service.
# journalctl -u designate-producer
journalctl -u designate-producer | grep start
May 09 00:20:40 juju-eeda89-2-lxd-2 systemd[1]: designate-producer.service: Service hold-off time over, scheduling restart.
May 09 00:20:40 juju-eeda89-2-lxd-2 systemd[1]: designate-producer.service: Scheduled restart job, restart counter is at 1.
May 09 00:20:41 juju-eeda89-2-lxd-2 systemd[1]: designate-producer.service: Service hold-off time over, scheduling restart.
May 09 00:20:41 juju-eeda89-2-lxd-2 systemd[1]: designate-producer.service: Scheduled restart job, restart counter is at 2.
May 09 00:20:43 juju-eeda89-2-lxd-2 systemd[1]: designate-producer.service: Service hold-off time over, scheduling restart.
May 09 00:20:43 juju-eeda89-2-lxd-2 systemd[1]: designate-producer.service: Scheduled restart job, restart counter is at 3.
May 09 00:20:44 juju-eeda89-2-lxd-2 systemd[1]: designate-producer.service: Service hold-off time over, scheduling restart.
May 09 00:20:44 juju-eeda89-2-lxd-2 systemd[1]: designate-producer.service: Scheduled restart job, restart counter is at 4.
May 09 00:20:46 juju-eeda89-2-lxd-2 systemd[1]: designate-producer.service: Service hold-off time over, scheduling restart.
May 09 00:20:46 juju-eeda89-2-lxd-2 systemd[1]: designate-producer.service: Scheduled restart job, restart counter is at 5.
May 09 00:20:46 juju-eeda89-2-lxd-2 systemd[1]: Failed to start OpenStack Designate DNSaaS producer.
designate/0 unit log (continued):
2019-05-09 00:20:58 INFO juju-log Purging ['designate-pool-manager', 'designate-zone-manager', 'python-designate', 'python-memcache']
2019-05-09 00:20:59 DEBUG openstack-upgrade Removing designate-pool-manager (1:7.0.0-0ubuntu1~cloud0) ...
2019-05-09 00:21:15 DEBUG openstack-upgrade Removing designate-zone-manager (1:7.0.0-0ubuntu1~cloud0) ...
2019-05-09 00:21:31 DEBUG openstack-upgrade Removing python-designate (1:7.0.0-0ubuntu1~cloud0) ...
2019-05-09 00:21:31 DEBUG openstack-upgrade update-alternatives: using /usr/bin/python3-designate-producer to provide /usr/bin/designate-producer (designate-producer) in auto mode
-------------------
grep producer /var/log/juju/unit-designate-0.log
2019-05-09 00:18:43 DEBUG openstack-upgrade update-alternatives: using /usr/bin/python2-designate-producer to provide /usr/bin/designate-producer (designate-producer) in auto mode
2019-05-09 00:19:58 INFO juju-log Installing ['designate-agent', 'designate-api', 'designate-central', 'designate-common', 'designate-mdns', 'designate-worker', 'designate-sink', 'designate-producer', 'bind9utils', 'python3-designate', 'python-apt', 'memcached', 'python3-memcache', 'haproxy', 'apache2'] with options: ['--option', 'Dpkg::Options::=--force-confnew', '--option', 'Dpkg::Options::=--force-confdef']
2019-05-09 00:19:58 DEBUG openstack-upgrade designate-producer designate-worker python3-amqp python3-anyjson
2019-05-09 00:20:09 DEBUG openstack-upgrade Get:112 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-updates/rocky/main amd64 designate-producer all 1:7.0.0-0ubuntu1~cloud0 [10.6 kB]
2019-05-09 00:20:20 DEBUG openstack-upgrade Selecting previously unselected package designate-producer.
2019-05-09 00:20:20 DEBUG openstack-upgrade Preparing to unpack .../105-designate-producer_1%3a7.0.0-0ubuntu1~cloud0_all.deb ...
2019-05-09 00:20:20 DEBUG openstack-upgrade Unpacking designate-producer (1:7.0.0-0ubuntu1~cloud0) ...
2019-05-09 00:20:38 DEBUG openstack-upgrade Setting up designate-producer (1:7.0.0-0ubuntu1~cloud0) ...
2019-05-09 00:20:38 DEBUG openstack-upgrade Created symlink /etc/systemd/system/multi-user.target.wants/designate-producer.service → /lib/systemd/system/designate-producer.service.
2019-05-09 00:21:31 DEBUG openstack-upgrade update-alternatives: using /usr/bin/python3-designate-producer to provide /usr/bin/designate-producer (designate-producer) in auto mode
--------------------------
I was able to start designate-producer |
Designate has to use Worker and Producer as of Rocky as the support for zone manager and pool manager was up for removal in Rocky. This was addressed in https://bugs.launchpad.net/charm-designate/+bug/1773190
During a Queens to Rocky upgrade I ran into an issue with designate-producer being down (see the details below).
I was able to successfully start designate-producer by hand. It looks like there were several attempts to restart the service by systemd itself ("Scheduled restart job, restart counter is at 5.") which eventually failed when the failcount became higher than 5. Debugging mode was disabled so I did not see anything in the producer log.
designate/0* blocked idle 2/lxd/2 10.232.46.153 9001/tcp Services not running that should be: designate-producer
hacluster-designate/0* active idle 10.232.46.153 Unit is ready and clustered
systemctl list-unit-files | grep designate-
designate-agent.service enabled
designate-api.service enabled
designate-central.service enabled
designate-mdns.service enabled
designate-producer.service enabled
designate-sink.service enabled
designate-worker.service enabled
ubuntu@juju-eeda89-2-lxd-2:~$ journalctl -u designate-producer.service
https://paste.ubuntu.com/p/WRXYvfynnd/
ubuntu@juju-eeda89-2-lxd-2:~$ pgrep -af designate
172877 bash /lib/systemd/system/jujud-unit-hacluster-designate-0/exec-start.sh
172878 bash /lib/systemd/system/jujud-unit-designate-0/exec-start.sh
172884 /var/lib/juju/tools/unit-designate-0/jujud unit --data-dir /var/lib/juju --unit-name designate/0 --debug
172886 /var/lib/juju/tools/unit-hacluster-designate-0/jujud unit --data-dir /var/lib/juju --unit-name hacluster-designate/0 --debug
392388 /usr/bin/python3.6 /usr/bin/designate-mdns --config-file=/etc/designate/designate.conf --log-file=/var/log/designate/designate-mdns.log
392399 /usr/bin/python3.6 /usr/bin/designate-agent --config-file=/etc/designate/designate.conf --log-file=/var/log/designate/designate-agent.log
392405 /usr/bin/python3.6 /usr/bin/designate-worker --config-file=/etc/designate/designate.conf --log-file=/var/log/designate/designate-worker.log
392411 /usr/bin/python3.6 /usr/bin/designate-central --config-file=/etc/designate/designate.conf --log-file=/var/log/designate/designate-central.log
392415 /usr/bin/python3.6 /usr/bin/designate-sink --config-file=/etc/designate/designate.conf --log-file=/var/log/designate/designate-sink.log
392423 /usr/bin/python3.6 /usr/bin/designate-api --config-file=/etc/designate/designate.conf --log-file=/var/log/designate/designate-api.log
392481 /usr/bin/python3.6 /usr/bin/designate-central --config-file=/etc/designate/designate.conf --log-file=/var/log/designate/designate-central.log
392482 /usr/bin/python3.6 /usr/bin/designate-central --config-file=/etc/designate/designate.conf --log-file=/var/log/designate/designate-central.log
392483 /usr/bin/python3.6 /usr/bin/designate-central --config-file=/etc/designate/designate.conf --log-file=/var/log/designate/designate-central.log
392484 /usr/bin/python3.6 /usr/bin/designate-central --config-file=/etc/designate/designate.conf --log-file=/var/log/designate/designate-central.log
----------
designate/0 unit log:
2019-05-09 00:20:38 DEBUG openstack-upgrade Setting up designate-producer (1:7.0.0-0ubuntu1~cloud0) ...
2019-05-09 00:20:38 DEBUG openstack-upgrade Created symlink /etc/systemd/system/multi-user.target.wants/designate-producer.service → /lib/systemd/system/designate-producer.service.
# journalctl -u designate-producer
journalctl -u designate-producer | grep start
May 09 00:20:40 juju-eeda89-2-lxd-2 systemd[1]: designate-producer.service: Service hold-off time over, scheduling restart.
May 09 00:20:40 juju-eeda89-2-lxd-2 systemd[1]: designate-producer.service: Scheduled restart job, restart counter is at 1.
May 09 00:20:41 juju-eeda89-2-lxd-2 systemd[1]: designate-producer.service: Service hold-off time over, scheduling restart.
May 09 00:20:41 juju-eeda89-2-lxd-2 systemd[1]: designate-producer.service: Scheduled restart job, restart counter is at 2.
May 09 00:20:43 juju-eeda89-2-lxd-2 systemd[1]: designate-producer.service: Service hold-off time over, scheduling restart.
May 09 00:20:43 juju-eeda89-2-lxd-2 systemd[1]: designate-producer.service: Scheduled restart job, restart counter is at 3.
May 09 00:20:44 juju-eeda89-2-lxd-2 systemd[1]: designate-producer.service: Service hold-off time over, scheduling restart.
May 09 00:20:44 juju-eeda89-2-lxd-2 systemd[1]: designate-producer.service: Scheduled restart job, restart counter is at 4.
May 09 00:20:46 juju-eeda89-2-lxd-2 systemd[1]: designate-producer.service: Service hold-off time over, scheduling restart.
May 09 00:20:46 juju-eeda89-2-lxd-2 systemd[1]: designate-producer.service: Scheduled restart job, restart counter is at 5.
May 09 00:20:46 juju-eeda89-2-lxd-2 systemd[1]: Failed to start OpenStack Designate DNSaaS producer.
designate/0 unit log (continued):
2019-05-09 00:20:58 INFO juju-log Purging ['designate-pool-manager', 'designate-zone-manager', 'python-designate', 'python-memcache']
2019-05-09 00:20:59 DEBUG openstack-upgrade Removing designate-pool-manager (1:7.0.0-0ubuntu1~cloud0) ...
2019-05-09 00:21:15 DEBUG openstack-upgrade Removing designate-zone-manager (1:7.0.0-0ubuntu1~cloud0) ...
2019-05-09 00:21:31 DEBUG openstack-upgrade Removing python-designate (1:7.0.0-0ubuntu1~cloud0) ...
2019-05-09 00:21:31 DEBUG openstack-upgrade update-alternatives: using /usr/bin/python3-designate-producer to provide /usr/bin/designate-producer (designate-producer) in auto mode
-------------------
grep producer /var/log/juju/unit-designate-0.log
2019-05-09 00:18:43 DEBUG openstack-upgrade update-alternatives: using /usr/bin/python2-designate-producer to provide /usr/bin/designate-producer (designate-producer) in auto mode
2019-05-09 00:19:58 INFO juju-log Installing ['designate-agent', 'designate-api', 'designate-central', 'designate-common', 'designate-mdns', 'designate-worker', 'designate-sink', 'designate-producer', 'bind9utils', 'python3-designate', 'python-apt', 'memcached', 'python3-memcache', 'haproxy', 'apache2'] with options: ['--option', 'Dpkg::Options::=--force-confnew', '--option', 'Dpkg::Options::=--force-confdef']
2019-05-09 00:19:58 DEBUG openstack-upgrade designate-producer designate-worker python3-amqp python3-anyjson
2019-05-09 00:20:09 DEBUG openstack-upgrade Get:112 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-updates/rocky/main amd64 designate-producer all 1:7.0.0-0ubuntu1~cloud0 [10.6 kB]
2019-05-09 00:20:20 DEBUG openstack-upgrade Selecting previously unselected package designate-producer.
2019-05-09 00:20:20 DEBUG openstack-upgrade Preparing to unpack .../105-designate-producer_1%3a7.0.0-0ubuntu1~cloud0_all.deb ...
2019-05-09 00:20:20 DEBUG openstack-upgrade Unpacking designate-producer (1:7.0.0-0ubuntu1~cloud0) ...
2019-05-09 00:20:38 DEBUG openstack-upgrade Setting up designate-producer (1:7.0.0-0ubuntu1~cloud0) ...
2019-05-09 00:20:38 DEBUG openstack-upgrade Created symlink /etc/systemd/system/multi-user.target.wants/designate-producer.service → /lib/systemd/system/designate-producer.service.
2019-05-09 00:21:31 DEBUG openstack-upgrade update-alternatives: using /usr/bin/python3-designate-producer to provide /usr/bin/designate-producer (designate-producer) in auto mode |
|
| 2019-05-15 17:42:49 |
David Ames |
charm-designate: status |
New |
Triaged |
|
| 2019-05-15 17:42:52 |
David Ames |
charm-designate: importance |
Undecided |
Critical |
|
| 2019-05-15 17:42:57 |
David Ames |
charm-designate: milestone |
|
19.07 |
|
| 2019-07-01 10:19:18 |
James Page |
charm-designate: assignee |
|
Liam Young (gnuoy) |
|
| 2019-07-02 15:28:20 |
Liam Young |
bug task added |
|
designate (Ubuntu) |
|
| 2019-07-02 15:28:39 |
Liam Young |
charm-designate: status |
Triaged |
Invalid |
|
| 2019-07-02 15:28:44 |
Liam Young |
charm-designate: assignee |
Liam Young (gnuoy) |
|
|
| 2019-07-04 16:44:15 |
Dmitrii Shcherbakov |
attachment added |
|
/var/log/ from designate/1 (broken) https://bugs.launchpad.net/charm-designate/+bug/1828534/+attachment/5275081/+files/var-log-designate-1.tar.gz |
|
| 2019-07-04 16:45:16 |
Dmitrii Shcherbakov |
attachment added |
|
/var/log/ from designate/0 (healthy) https://bugs.launchpad.net/charm-designate/+bug/1828534/+attachment/5275082/+files/var-log-designate-0.tar.gz |
|
| 2019-07-24 20:58:37 |
Corey Bryant |
designate (Ubuntu): status |
New |
Triaged |
|
| 2019-07-24 20:58:40 |
Corey Bryant |
designate (Ubuntu): importance |
Undecided |
High |
|
| 2019-07-24 20:59:53 |
Corey Bryant |
charm-designate: importance |
Critical |
Undecided |
|
| 2019-07-29 19:41:55 |
Corey Bryant |
bug task added |
|
designate |
|
| 2019-07-29 20:17:28 |
OpenStack Infra |
designate: status |
New |
In Progress |
|
| 2019-07-29 20:17:28 |
OpenStack Infra |
designate: assignee |
|
Corey Bryant (corey.bryant) |
|
| 2019-07-29 20:19:57 |
Corey Bryant |
tags |
cpe-onsite |
cpe-onsite py3 |
|
| 2019-08-13 10:55:46 |
Frode Nordahl |
charm-designate: milestone |
19.07 |
|
|
| 2019-08-26 04:46:51 |
OpenStack Infra |
designate: status |
In Progress |
Fix Released |
|
| 2019-12-04 19:19:48 |
Corey Bryant |
nominated for series |
|
Ubuntu Eoan |
|
| 2019-12-04 19:19:48 |
Corey Bryant |
bug task added |
|
designate (Ubuntu Eoan) |
|
| 2019-12-04 19:19:48 |
Corey Bryant |
nominated for series |
|
Ubuntu Disco |
|
| 2019-12-04 19:19:48 |
Corey Bryant |
bug task added |
|
designate (Ubuntu Disco) |
|
| 2019-12-04 19:20:00 |
Corey Bryant |
designate (Ubuntu Eoan): status |
New |
Fix Released |
|
| 2019-12-04 19:20:03 |
Corey Bryant |
designate (Ubuntu Eoan): importance |
Undecided |
High |
|
| 2019-12-04 19:20:07 |
Corey Bryant |
designate (Ubuntu Disco): importance |
Undecided |
High |
|
| 2019-12-04 19:20:09 |
Corey Bryant |
designate (Ubuntu Disco): status |
New |
Triaged |
|
| 2019-12-04 19:20:23 |
Corey Bryant |
bug task added |
|
cloud-archive |
|
| 2019-12-04 19:20:37 |
Corey Bryant |
nominated for series |
|
cloud-archive/stein |
|
| 2019-12-04 19:20:37 |
Corey Bryant |
bug task added |
|
cloud-archive/stein |
|
| 2019-12-04 19:20:37 |
Corey Bryant |
nominated for series |
|
cloud-archive/rocky |
|
| 2019-12-04 19:20:37 |
Corey Bryant |
bug task added |
|
cloud-archive/rocky |
|
| 2019-12-04 19:20:37 |
Corey Bryant |
nominated for series |
|
cloud-archive/train |
|
| 2019-12-04 19:20:37 |
Corey Bryant |
bug task added |
|
cloud-archive/train |
|
| 2019-12-04 19:20:49 |
Corey Bryant |
designate (Ubuntu): status |
Triaged |
Fix Committed |
|
| 2019-12-04 19:20:53 |
Corey Bryant |
designate (Ubuntu): status |
Fix Committed |
Fix Released |
|
| 2019-12-04 19:21:00 |
Corey Bryant |
cloud-archive/train: status |
New |
Fix Released |
|
| 2019-12-04 19:21:03 |
Corey Bryant |
cloud-archive/train: importance |
Undecided |
High |
|
| 2019-12-04 19:21:07 |
Corey Bryant |
cloud-archive/rocky: importance |
Undecided |
High |
|
| 2019-12-04 19:21:09 |
Corey Bryant |
cloud-archive/stein: importance |
Undecided |
High |
|
| 2019-12-04 19:21:12 |
Corey Bryant |
cloud-archive/stein: status |
New |
Triaged |
|
| 2019-12-04 19:21:15 |
Corey Bryant |
cloud-archive/rocky: status |
New |
Triaged |
|
| 2019-12-04 19:21:20 |
Corey Bryant |
cloud-archive: status |
New |
Fix Released |
|
| 2019-12-04 19:21:23 |
Corey Bryant |
cloud-archive: importance |
Undecided |
High |
|
| 2019-12-05 14:43:38 |
Corey Bryant |
designate (Ubuntu Disco): status |
Triaged |
Won't Fix |
|
| 2019-12-05 14:43:41 |
Corey Bryant |
cloud-archive/stein: status |
Triaged |
Won't Fix |
|
| 2019-12-05 14:43:48 |
Corey Bryant |
cloud-archive/rocky: status |
Triaged |
Won't Fix |
|
| 2019-12-10 20:50:20 |
Dincer Celik |
bug |
|
|
added subscriber Dincer Celik |
| 2020-05-12 14:50:36 |
Corey Bryant |
bug task added |
|
charm-deployment-guide |
|
| 2020-05-12 14:50:59 |
Corey Bryant |
charm-deployment-guide: importance |
Undecided |
High |
|
| 2020-05-12 14:50:59 |
Corey Bryant |
charm-deployment-guide: status |
New |
Triaged |
|
| 2020-05-19 16:28:57 |
Drew Freiberger |
bug |
|
|
added subscriber Canonical IS BootStack |
| 2020-05-19 16:29:28 |
Drew Freiberger |
bug task added |
|
charm-memcached |
|
| 2020-05-19 16:46:24 |
Corey Bryant |
charm-deployment-guide: status |
Triaged |
Fix Committed |
|
| 2020-06-03 12:04:04 |
Chris MacNaughton |
charm-designate: assignee |
|
Chris MacNaughton (chris.macnaughton) |
|
| 2020-06-04 14:05:16 |
Diko Parvanov |
charm-memcached: importance |
Undecided |
High |
|
| 2020-06-04 14:05:16 |
Diko Parvanov |
charm-memcached: status |
New |
Triaged |
|
| 2020-06-10 09:26:33 |
Alvaro Uria |
merge proposal linked |
|
https://code.launchpad.net/~chris.macnaughton/charm-memcached/+git/charm-memcached/+merge/384961 |
|
| 2020-06-10 09:28:34 |
Alvaro Uria |
charm-memcached: status |
Triaged |
Fix Committed |
|
| 2020-06-10 09:28:34 |
Alvaro Uria |
charm-memcached: milestone |
|
20.08 |
|
| 2020-06-13 21:35:11 |
OpenStack Infra |
charm-designate: status |
In Progress |
Fix Committed |
|
| 2020-06-23 19:31:51 |
Andrea Ieri |
charm-memcached: status |
Fix Committed |
Fix Released |
|
| 2020-08-03 14:03:31 |
James Page |
charm-designate: milestone |
|
20.08 |
|
| 2020-08-14 15:40:55 |
Alex Kavanagh |
charm-designate: status |
Fix Committed |
Fix Released |
|
| 2020-08-18 11:24:40 |
Chris MacNaughton |
charm-deployment-guide: status |
Fix Committed |
Fix Released |
|
