Intermittent deployment failures with unit in state 'Services not running that should be: neutron-bgp-dragent'

[Frode Nordahl]

The root cause is most likely that systemd is attempting to start the service before all required relations are available and the charm has written out the configuration.

Jul 22 09:12:06 juju-a3c504-zaza-5209eceb5893-3 systemd[1]: neutron-bgp-dragent.service: Main process exited, code=exited, status=1/FAILURE
Jul 22 09:12:06 juju-a3c504-zaza-5209eceb5893-3 systemd[1]: neutron-bgp-dragent.service: Failed with result 'exit-code'.
Jul 22 09:12:06 juju-a3c504-zaza-5209eceb5893-3 systemd[1]: neutron-bgp-dragent.service: Service hold-off time over, scheduling restart.
Jul 22 09:12:06 juju-a3c504-zaza-5209eceb5893-3 systemd[1]: neutron-bgp-dragent.service: Scheduled restart job, restart counter is at 20.
Jul 22 09:12:06 juju-a3c504-zaza-5209eceb5893-3 systemd[1]: Stopped OpenStack Neutron dynamic routing agent.
Jul 22 09:12:06 juju-a3c504-zaza-5209eceb5893-3 systemd[1]: Started OpenStack Neutron dynamic routing agent.
Jul 22 09:12:06 juju-a3c504-zaza-5209eceb5893-3 systemd[1]: Stopping OpenStack Neutron dynamic routing agent...
Jul 22 09:12:06 juju-a3c504-zaza-5209eceb5893-3 systemd[1]: Stopped OpenStack Neutron dynamic routing agent.
Jul 22 09:12:06 juju-a3c504-zaza-5209eceb5893-3 systemd[1]: neutron-bgp-dragent.service: Start request repeated too quickly.
Jul 22 09:12:06 juju-a3c504-zaza-5209eceb5893-3 systemd[1]: neutron-bgp-dragent.service: Failed with result 'start-limit-hit'.
Jul 22 09:12:06 juju-a3c504-zaza-5209eceb5893-3 systemd[1]: Failed to start OpenStack Neutron dynamic routing agent.

2019-07-22 09:12:05.534 16088 INFO neutron.common.config [-] Logging enabled!
2019-07-22 09:12:05.535 16088 INFO neutron.common.config [-] /usr/bin/neutron-bgp-dragent version 13.0.3
2019-07-22 09:12:05.912 16088 CRITICAL neutron [-] Unhandled error: AttributeError: 'NoneType' object has no attribute 'rpartition'
2019-07-22 09:12:05.912 16088 ERROR neutron Traceback (most recent call last):
2019-07-22 09:12:05.912 16088 ERROR neutron File "/usr/bin/neutron-bgp-dragent", line 10, in <module>
2019-07-22 09:12:05.912 16088 ERROR neutron sys.exit(main())
2019-07-22 09:12:05.912 16088 ERROR neutron File "/usr/lib/python3/dist-packages/neutron_dynamic_routing/cmd/eventlet/agents/bgp_dragent.py", line 20, in main
2019-07-22 09:12:05.912 16088 ERROR neutron bgp_dragent.main()
2019-07-22 09:12:05.912 16088 ERROR neutron File "/usr/lib/python3/dist-packages/neutron_dynamic_routing/services/bgp/agent/entry.py", line 45, in main
2019-07-22 09:12:05.912 16088 ERROR neutron manager='neutron_dynamic_routing.services.bgp.agent.bgp_dragent.'
2019-07-22 09:12:05.912 16088 ERROR neutron File "/usr/lib/python3/dist-packages/neutron/service.py", line 389, in create
2019-07-22 09:12:05.912 16088 ERROR neutron periodic_fuzzy_delay=periodic_fuzzy_delay)
2019-07-22 09:12:05.912 16088 ERROR neutron File "/usr/lib/python3/dist-packages/neutron/service.py", line 321, in __init__
2019-07-22 09:12:05.912 16088 ERROR neutron self.manager = manager_class(host=host, *args, **kwargs)
2019-07-22 09:12:05.912 16088 ERROR neutron File "/usr/lib/python3/dist-packages/neutron_dynamic_routing/services/bgp/agent/bgp_dragent.py", line 664, in __init__
2019-07-22 09:12:05.912 16088 ERROR neutron self).__init__(host, conf)
2019-07-22 09:12:05.912 16088 ERROR neutron File "/usr/lib/python3/dist-packages/neutron_dynamic_routing/services/bgp/agent/bgp_dragent.py", line 59, in __init__
2019-07-22 09:12:05.912 16088 ERROR neutron self.initialize_driver(conf)
2019-07-22 09:12:05.912 16088 ERROR neutron File "/usr/lib/python3/dist-packages/osprofiler/profiler.py", line 158, in wrapper
2019-07-22 09:12:05.912 16088 ERROR neutron result = f(*args, **kwargs)
2019-07-22 09:12:05.912 16088 ERROR neutron File "/usr/lib/python3/dist-packages/neutron_dynamic_routing/services/bgp/agent/bgp_dragent.py", line 73, in initialize_driver
2019-07-22 09:12:05.912 16088 ERROR neutron self.conf))
2019-07-22 09:12:05.912 16088 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_utils/importutils.py", line 44, in import_object
2019-07-22 09:12:05.912 16088 ERROR neutron return import_class(import_str)(*args, **kwargs)
2019-07-22 09:12:05.912 16088 ERROR neutron File "/usr/lib/python3/dist-packages/oslo_utils/importutils.py", line 29, in import_class
2019-07-22 09:12:05.912 16088 ERROR neutron mod_str, _sep, class_str = import_str.rpartition('.')
2019-07-22 09:12:05.912 16088 ERROR neutron AttributeError: 'NoneType' object has no attribute 'rpartition'
2019-07-22 09:12:05.912 16088 ERROR neutron

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/672010

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/672011

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/672010
Committed: https://git.openstack.org/cgit/openstack/charm-neutron-dynamic-routing/commit/?id=21cf26b29cfb335fc155f572ce17c18ea0ee0b88
Submitter: Zuul
Branch: master

commit 21cf26b29cfb335fc155f572ce17c18ea0ee0b88
Author: Frode Nordahl <email address hidden>
Date: Mon Jul 22 12:41:45 2019 +0200

    Update charm to use bus to get charm instance

    Add ``py3`` target to tox.ini for developer friendliness

    Update unit tests to use ``charms.openstack`` unit test helers.

    Change-Id: I4752d8e776491f934cd5c1232166933a9ba17746
    Partial-Bug: #1837379

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/672011
Committed: https://git.openstack.org/cgit/openstack/charm-neutron-dynamic-routing/commit/?id=871bcaaeb31efe21a21e4a8134c570d80d2cc313
Submitter: Zuul
Branch: master

commit 871bcaaeb31efe21a21e4a8134c570d80d2cc313
Author: Frode Nordahl <email address hidden>
Date: Mon Jul 22 12:48:20 2019 +0200

    Hold services down until config rendered

    Change-Id: I1dc570a26bfa97f5612c7b82f06181b2f4e6a665
    Closes-Bug: #1837379