nova-api-metadata nagios checks not removed after upgrade to rocky

[Junien F]

Hi,

Using bionic, 18.11 charms. I recently upgraded my cloud from queens to rocky. In queens, the nova-api-metadata service lives on the neutron-gateway units. In rocky, it lives on the nova-cloud-controller units.

After the upgrade, the neutron-gateway units still have a /etc/nagios/nrpe.d/check_nova-api-metadata.cfg file, as well as /var/lib/nagios/export/service__<nagios_context>-neutron-gateway-0_check_nova-api-metadata.cfg file. These should be removed during the upgrade.

Thanks

[Drew Freiberger (afreiberger)]

Experienced this on bionic-distro to bionic-rocky upgrade today. The service is masked, it appears we should somehow update the nrpe check within the remove_legacy_nova_metadata() function.

[Andrea Ieri (aieri)]

For future travelers: a successful workaround is to manually inject an updated monitors value that doesn't list the nova-api-metadata check. This is described in bug 1870438; that example is about keystone, but the process is the same.

[Andrea Ieri (aieri)]

Here's a more step by step explanation of the workaround:

$ juju run -u neutron-gateway/0 -- relation-ids nrpe-external-master
nrpe-external-master:253

$ juju run -u neutron-gateway/0 -- relation-list -r253
nrpe-host/22

$ juju run -u nrpe-host/22 -- relation-get -r253 - neutron-gateway/0
egress-subnets: <REDACTED>
ingress-address: <REDACTED>
monitors: |
  monitors:
    remote:
      nrpe:
        netns: {command: check_netns}
        neutron-dhcp-agent: {command: check_neutron-dhcp-agent}
        neutron-l3-agent: {command: check_neutron-l3-agent}
        neutron-lbaasv2-agent: {command: check_neutron-lbaasv2-agent}
        neutron-metadata-agent: {command: check_neutron-metadata-agent}
        neutron-metering-agent: {command: check_neutron-metering-agent}
        neutron-openvswitch-agent: {command: check_neutron-openvswitch-agent}
        nova-api-metadata: {command: check_nova-api-metadata}
primary: "True"
private-address: <REDACTED>

# Now save the monitors section and fix it

$ cat monitors.lp1812813.out
monitors:
  remote:
    nrpe:
      netns: {command: check_netns}
      neutron-dhcp-agent: {command: check_neutron-dhcp-agent}
      neutron-l3-agent: {command: check_neutron-l3-agent}
      neutron-lbaasv2-agent: {command: check_neutron-lbaasv2-agent}
      neutron-metadata-agent: {command: check_neutron-metadata-agent}
      neutron-metering-agent: {command: check_neutron-metering-agent}
      neutron-openvswitch-agent: {command: check_neutron-openvswitch-agent}

# Finally set the amended relation data:

$ juju run -u neutron-gateway/0 -- relation-set -r253 monitors="$(cat monitors.lp1812813.out)"

[Pedro Guimarães (pguimaraes)]

Hi, I faced this issue recently. My conclusion is that quantum-network-service relation should be outdated from Rocky onward.

We can see from:
https://github.com/openstack/charm-neutron-gateway/blob/7e68feed95eb9489184914cf218265d4a30558e2/hooks/neutron_hooks.py#L173

That, if we are running Rocky or later + a secret has been shared from nova-cc to neutron-gw, neutron-gw charm will disable and mask nova-api-metadata service.

However, on nova-cc side, that will always happen on quantum-network-service. We can check that a -joined relation runs:
https://github.com/openstack/charm-nova-cloud-controller/blob/65ed6620c7594cac914d21c2203742f63dc43d17/hooks/nova_cc_hooks.py#L978

Which leads to always sharing a secret if running Rocky or later:
https://github.com/openstack/charm-nova-cloud-controller/blob/86bb59ec3e85b354e65c2f6db0d864239600d5df/hooks/nova_cc_utils.py#L1914
https://github.com/openstack/charm-nova-cloud-controller/blob/86bb59ec3e85b354e65c2f6db0d864239600d5df/hooks/nova_cc_utils.py#L1851

It effectively means that nova-api-metadata will be installed, but then disconfigured and masked.

However, the nrpe-check remains, as well as the broken service: https://pastebin.canonical.com/p/txjFcNX7Vd/

My suggestion is to deprecate this relation and make it optional for neutron-gw & nova-cc.

[Pedro Guimarães (pguimaraes)]

The least damaging change we can do here is to add "nova-api-metadata" to deprecated services:
https://github.com/openstack/charm-neutron-gateway/blob/a03fe36fa65b710b6cd8059b870c44204f3e3856/hooks/neutron_utils.py#L1124

And add the following:

def deprecated_services():
    ''' Returns a list of deprecated services with this charm '''
    cmp_release = CompareOpenStackReleases(os_release('neutron-common'))
    if disable_nova_metadata(): <---------------
        return ['nova-api-metadata'] <---------------
    if cmp_release >= 'train':
        return ['neutron-lbaasv2-agent']

    return []

And make sure that NRPE list is updated whenever quantum-network-services relation changes.
Add update_nrpe_config() here:
https://github.com/openstack/charm-neutron-gateway/blob/7e68feed95eb9489184914cf218265d4a30558e2/hooks/neutron_hooks.py#L271

*******************************************************************

The solution above fixes NRPE checks. However, we keep having an unnecessary package installed and a masked service available.

I tried deploying without neutron-gw / nova-cc relation.
Neutron-gw blocks waiting for nova-api-metadata service to come up, but that service will stay unconfigured without the relation.

The best approach is to ensure the package is never installed in the first place.
That can be done here: https://github.com/openstack/charm-neutron-gateway/blob/7e68feed95eb9489184914cf218265d4a30558e2/hooks/neutron_hooks.py#L114

If this returns True for >= Rocky, this check will remove nova-related packages at install time:
https://github.com/openstack/charm-neutron-gateway/blob/a03fe36fa65b710b6cd8059b870c44204f3e3856/hooks/neutron_utils.py#L285

Now, that will return False, since disable_nova_metadata depends on a specific parameter passed by nova-cc relation, which only comes up later on. Therefore, this is a check that always get False.

Looking deeper into neutron-gateway:
https://github.com/openstack/charm-neutron-gateway/blob/a03fe36fa65b710b6cd8059b870c44204f3e3856/hooks/neutron_utils.py#L937

It should return in such a way that:

def disable_nova_metadata(cmp_os_source=None):
    """Check whether nova mnetadata service should be disabled."""
    if not cmp_os_source:
        cmp_os_source = CompareOpenStackReleases(os_release('neutron-common'))
    if cmp_os_source >= 'rocky':
        return True

Since, as mentioned on the previous comment, nova-cc will always certify that as True for Rocky onward.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/732094
Committed: https://git.openstack.org/cgit/openstack/charm-neutron-gateway/commit/?id=775c06f8ff513017454597724a2c52089caad95c
Submitter: Zuul
Branch: master

commit 775c06f8ff513017454597724a2c52089caad95c
Author: Pedro Guimaraes <email address hidden>
Date: Fri May 29 22:28:43 2020 +0200

    Deprecate nova-api-metadata nrpe check

    This change adds nova-api-metadata to the list of
    deprecated services for NRPE checks. That will be
    happen if disable_nova_metadata() is set to True.

    Added an update to NRPE checks to quantum-network
    -service changed hook, since its change defines
    if nova-api-metadata should be deprecated.

    Change-Id: I6b1174561f3dad3ec5334f1e38e45c068a0107a8
    Closes-Bug: #1812813

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/20.05
Review: https://review.opendev.org/732501

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/732501
Committed: https://git.openstack.org/cgit/openstack/charm-neutron-gateway/commit/?id=3dd6d68c2ceaef1b7deab010aaadd4f236d24abc
Submitter: Zuul
Branch: stable/20.05

commit 3dd6d68c2ceaef1b7deab010aaadd4f236d24abc
Author: Pedro Guimaraes <email address hidden>
Date: Fri May 29 22:28:43 2020 +0200

    Deprecate nova-api-metadata nrpe check

    This change adds nova-api-metadata to the list of
    deprecated services for NRPE checks. That will be
    happen if disable_nova_metadata() is set to True.

    Added an update to NRPE checks to quantum-network
    -service changed hook, since its change defines
    if nova-api-metadata should be deprecated.

    Change-Id: I6b1174561f3dad3ec5334f1e38e45c068a0107a8
    Closes-Bug: #1812813
    (cherry picked from commit 775c06f8ff513017454597724a2c52089caad95c)