Unit stays blocked: Services not running that should be: ironic-neutron-agent

[Bas de Bruijne]

In the testrun:
https://solutions.qa.canonical.com/testruns/testRun/5f9f81e0-5383-4347-95bc-dec7b540c180

The Neutron API Ironic Plugin stays blocked:
------------------------------
neutron-api/0 waiting Incomplete relations: database
  filebeat/34 active Filebeat ready.
  hacluster-neutron-api/1 active Unit is ready and clustered
  landscape-client/39 maintenance Need computer-title and juju-info to proceed
  logrotated/38 active Unit is ready.
  neutron-api-mysql-router/1 waiting 'shared-db' incomplete, Waiting for proxied DB creation from cluster
  neutron-ironic-agent/1 blocked Services not running that should be: ironic-neutron-agent
  nrpe/38 active Ready (source version/commit cs-nrpe-...)
  public-policy-routing/21 active Unit is ready
  telegraf/36 active Monitoring neutron-api/0 (source version/commit 26e531a)
------------------------------

This is the first time we've seen this happening, so I expect that there is a race condition.
I can't find anything standing out in the logs.

FCE console output:
https://oil-jenkins.canonical.com/job/fce_build/1272//console
Crashdumps etc:
https://oil-jenkins.canonical.com/artifacts/5f9f81e0-5383-4347-95bc-dec7b540c180/index.html

[Muhammad Ahmad (ahmadfsbd)]

Experienced the same bug while working with ironic-conductor (master branch), the bundles use this charm and fail on functional tests of jammy-yoga only. To re-confirm, I deployed bundles from charm-neutron-api-plugin (master branch):

https://opendev.org/openstack/charm-neutron-api-plugin-ironic

Again, the functional tests failed only with the jammy-yoga bundle with the same error.

############# juju status ##############
neutron-api/0* waiting idle 17 10.5.1.39 9696/tcp Incomplete relations: identity, database
  neutron-api-mysql-router/0* error idle 10.5.1.39 hook failed: "install"
  neutron-ironic-agent/0* blocked idle 10.5.1.39 Services not running that should be: ironic-neutron-agent

############ juju debug-log #############

tracer: ++ queue handler reactive/neutron_ironic_handlers.py:30:setup_endpoint
tracer: ++ queue handler reactive/neutron_ironic_handlers.py:45:configure_principal
unit-neutron-ironic-agent-0: 11:17:49 INFO unit.neutron-ironic-agent/0.juju-log identity-credentials:18: Invoking reactive handler: reactive/neutron_ironic_handlers.py:21:render_stuff
unit-neutron-ironic-agent-0: 11:17:49 ERROR unit.neutron-ironic-agent/0.juju-log identity-credentials:18: Could not load template ironic_neutron_agent.ini from None.
unit-neutron-ironic-agent-0: 11:17:49 DEBUG unit.neutron-ironic-agent/0.juju-log identity-credentials:18: Writing file /etc/neutron/plugins/ml2/ironic_neutron_agent.ini root:neutron 640
unit-neutron-ironic-agent-0: 11:17:49 WARNING unit.neutron-ironic-agent/0.identity-credentials-relation-changed Failed to stop ironic-neutron-agent.service: Unit ironic-neutron-agent.service not loaded.
unit-neutron-ironic-agent-0: 11:17:49 WARNING unit.neutron-ironic-agent/0.identity-credentials-relation-changed Failed to start ironic-neutron-agent.service: Unit ironic-neutron-agent.service not found.

[Liam Young (gnuoy)]

This seems to be reproducible with:

juju deploy --channel edge --series jammy neutron-api
juju deploy --channel edge --series jammy ch:neutron-api-plugin-ironic neutron-ironic-agent
juju add-relation neutron-api neutron-ironic-agent

wait for units to finishing executing hooks

juju deploy --channel 8.0/stable --series jammy mysql-router
juju add-relation neutron-api mysql-router

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-neutron-api-plugin-ironic/+/853160

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-neutron-api-plugin-ironic/+/853160
Committed: https://opendev.org/openstack/charm-neutron-api-plugin-ironic/commit/9fb3530b0c9c065afb12ac7f3b46ec959b1cb55b
Submitter: "Zuul (22348)"
Branch: master

commit 9fb3530b0c9c065afb12ac7f3b46ec959b1cb55b
Author: Liam Young <email address hidden>
Date: Mon Aug 15 11:32:23 2022 +0000

    Charm use virtualenv

    The charm is polluting the system it is installed on causing
    bug #1970230. All reactive charms should use a venv wherever
    possible and I belive it is just an oversight that this one
    does not.

    Closes-Bug: #1970230
    Change-Id: I3bb162c5a10735c6c4402069719d13c5c2037bf7

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/charm-neutron-api-plugin-ironic/+/853353

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/charm-neutron-api-plugin-ironic/+/853354

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/charm-neutron-api-plugin-ironic/+/853355

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/charm-neutron-api-plugin-ironic/+/853356

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/charm-neutron-api-plugin-ironic/+/853357

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/charm-neutron-api-plugin-ironic/+/853358

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-neutron-api-plugin-ironic/+/853356
Committed: https://opendev.org/openstack/charm-neutron-api-plugin-ironic/commit/a7f4d0a8170ec4a16d3bb1d38ffc9b8c5c3b29f8
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit a7f4d0a8170ec4a16d3bb1d38ffc9b8c5c3b29f8
Author: Liam Young <email address hidden>
Date: Mon Aug 15 11:32:23 2022 +0000

    Charm use virtualenv

    The charm is polluting the system it is installed on causing
    bug #1970230. All reactive charms should use a venv wherever
    possible and I belive it is just an oversight that this one
    does not.

    Closes-Bug: #1970230
    Change-Id: I3bb162c5a10735c6c4402069719d13c5c2037bf7
    (cherry picked from commit 9fb3530b0c9c065afb12ac7f3b46ec959b1cb55b)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-neutron-api-plugin-ironic/+/853357
Committed: https://opendev.org/openstack/charm-neutron-api-plugin-ironic/commit/7400e9076a191f0b576399108c6fb7198abe41ab
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 7400e9076a191f0b576399108c6fb7198abe41ab
Author: Liam Young <email address hidden>
Date: Mon Aug 15 11:32:23 2022 +0000

    Charm use virtualenv

    The charm is polluting the system it is installed on causing
    bug #1970230. All reactive charms should use a venv wherever
    possible and I belive it is just an oversight that this one
    does not.

    Closes-Bug: #1970230
    Change-Id: I3bb162c5a10735c6c4402069719d13c5c2037bf7
    (cherry picked from commit 9fb3530b0c9c065afb12ac7f3b46ec959b1cb55b)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-neutron-api-plugin-ironic/+/853355
Committed: https://opendev.org/openstack/charm-neutron-api-plugin-ironic/commit/4fba8db56ce7bf7a7455bf3e9f7beb54b5f2bcb1
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 4fba8db56ce7bf7a7455bf3e9f7beb54b5f2bcb1
Author: Liam Young <email address hidden>
Date: Mon Aug 15 11:32:23 2022 +0000

    Charm use virtualenv

    The charm is polluting the system it is installed on causing
    bug #1970230. All reactive charms should use a venv wherever
    possible and I belive it is just an oversight that this one
    does not.

    Closes-Bug: #1970230
    Change-Id: I3bb162c5a10735c6c4402069719d13c5c2037bf7
    (cherry picked from commit 9fb3530b0c9c065afb12ac7f3b46ec959b1cb55b)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-neutron-api-plugin-ironic/+/853354
Committed: https://opendev.org/openstack/charm-neutron-api-plugin-ironic/commit/c68a50b2ab091deed2909ffef3fc64d1cd3b4fdb
Submitter: "Zuul (22348)"
Branch: stable/xena

commit c68a50b2ab091deed2909ffef3fc64d1cd3b4fdb
Author: Liam Young <email address hidden>
Date: Mon Aug 15 11:32:23 2022 +0000

    Charm use virtualenv

    The charm is polluting the system it is installed on causing
    bug #1970230. All reactive charms should use a venv wherever
    possible and I belive it is just an oversight that this one
    does not.

    Closes-Bug: #1970230
    Change-Id: I3bb162c5a10735c6c4402069719d13c5c2037bf7
    (cherry picked from commit 9fb3530b0c9c065afb12ac7f3b46ec959b1cb55b)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-neutron-api-plugin-ironic/+/853353
Committed: https://opendev.org/openstack/charm-neutron-api-plugin-ironic/commit/babf5249ef0a8161bc13cb3e21d6c154815bff36
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit babf5249ef0a8161bc13cb3e21d6c154815bff36
Author: Liam Young <email address hidden>
Date: Mon Aug 15 11:32:23 2022 +0000

    Charm use virtualenv

    The charm is polluting the system it is installed on causing
    bug #1970230. All reactive charms should use a venv wherever
    possible and I belive it is just an oversight that this one
    does not.

    Closes-Bug: #1970230
    Change-Id: I3bb162c5a10735c6c4402069719d13c5c2037bf7
    (cherry picked from commit 9fb3530b0c9c065afb12ac7f3b46ec959b1cb55b)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-neutron-api-plugin-ironic/+/853358
Committed: https://opendev.org/openstack/charm-neutron-api-plugin-ironic/commit/6bb2f86eaa0865bd5223077717327f5e615bf6b0
Submitter: "Zuul (22348)"
Branch: stable/train

commit 6bb2f86eaa0865bd5223077717327f5e615bf6b0
Author: Liam Young <email address hidden>
Date: Mon Aug 15 11:32:23 2022 +0000

    Charm use virtualenv

    The charm is polluting the system it is installed on causing
    bug #1970230. All reactive charms should use a venv wherever
    possible and I belive it is just an oversight that this one
    does not.

    Closes-Bug: #1970230
    Change-Id: I3bb162c5a10735c6c4402069719d13c5c2037bf7
    (cherry picked from commit 9fb3530b0c9c065afb12ac7f3b46ec959b1cb55b)