When removing a compute service with ovn-chassis subordinate, agents are not deleted or deletable

[Drew Freiberger]

I have run a juju remove-unit nova-compute-kvm-sriov/X on a cloud with ovn-chassis subordinate (with SRIOV enabled) running on a cloud.

After this, when I run "openstack network agent list|grep XXX", I see 3 dead agents (SRIOV special case, typically it'd be just two agents.)

One is the SRIOV NIC Agent. I was able to delete this NIC Agent with "openstack network agent delete <uuid>", however, the "OVN Controller Gateway agent" and "OVN Metadata agent" cannot be deleted via neutron api.

$ os network agent list|grep XXX
| myhostname.fqdn | OVN Controller Gateway agent | myhostname.fqdn | | XXX | UP | ovn-controller |
| 2d51ab41-bf32-4328-a51a-ff710198c559 | OVN Metadata agent | myhostname.fqdn | | XXX | UP | networking-ovn-metadata-agent |

$ openstack network agent delete 2d51ab41-bf32-4328-a51a-ff710198c559
Failed to delete network agent with ID '2d51ab41-bf32-4328-a51a-ff710198c559': ResourceNotFound: 404: Client Error for url: https://neutron.fqdn:9696/v2.0/agents/2d51ab41-bf32-4328-a51a-ff710198c559, Agent 2d51ab41-bf32-4328-a51a-ff710198c559 could not be found.
1 of 1 network agents failed to delete.

Gamma-LAB MODEL(openstack) jujumanage@inf1az1cz202904rt:~$ os network agent delete myhostname.fqdn
Failed to delete network agent with ID 'myhostname.fqdn': ResourceNotFound: 404: Client Error for url: https://neutron.fqdn:9696/v2.0/agents/cz20430n2j.mgt.tra.stg.tlc.gamma.cloud, Agent myhostname.fqdn could not be found.
1 of 1 network agents failed to delete.

See bugzilla bug: https://bugzilla.redhat.com/show_bug.cgi?id=1849166 which is directly related.

It would be useful to have a well defined process for clearing these dead OVN agent entries so monitoring of dead network agents is accurate.