[OVN] neutron_ovn_metadata_agent retrying on UUID errors infinitely

[Ivan Zhang]

*Descrption:
  ** We observed neutron_ovn_metadata_agent was stuck when connection ovn_sb_db is reset
  ** After investigation, we found that MetadataAgent.register_metadata_agent keeps retrying infinitely due to an unrecoverable error in this line: chassis_id = uuid.UUID(self._get_own_chassis_name()). This block the thread for RowEventHandler.notify_loop.
  ** The neutron is deployed by kolla ansible, where chassis name returned here is the hostname of the compute nodes, and it is not recognized as a valid UUID.

* Version:
  ** OpenStack Xena
  ** Ubuntu focal 20.04
  ** Kolla Ansible

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hello Ivan:

From OVS documentation [1], you can see "system-ID" is type UUID. You can set "random" to generate, using "uuidgen" [2], a random generated UUID number.

If OVS DB API fails to check the type of the value set in "system-id", this is a problem in the OVS code. It shouldn't be able to set any possible string in Open_vSwitch:external_ids:system-id.

I'll make the OVN agent code more robust, writing an ERROR message in the logs if the reported issue happens. But need to change the chassis system-id assignation of your deployment tool, providing a valid UUID value.

Regards.

[1]https://docs.openvswitch.org/en/latest/ref/ovs-ctl.8/
[2]https://www.apt-browse.org/browse/ubuntu/xenial/main/amd64/openvswitch-switch/2.5.0-0ubuntu1/file/usr/share/openvswitch/scripts/ovs-ctl

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/819634

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/819634
Committed: https://opendev.org/openstack/neutron/commit/68f3e21034461ffa1525d2680b20ba2ecbb53dd4
Submitter: "Zuul (22348)"
Branch: master

commit 68f3e21034461ffa1525d2680b20ba2ecbb53dd4
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Nov 29 12:05:42 2021 +0000

    [OVN] Chassis name (OVS system-id) must be a UUID formatted string

    The OVS system-id must be a UUID formated string. If not, the OVN
    metadata agent will log the error and exit.

    Closes-Bug: #1952550
    Change-Id: Iad2b07f6e40dcbf690889d3b69bc00bb2ed0c05c

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/821828

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/821829

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/821830

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/neutron/+/821831

[Felipe Reyes (freyes)]

Hello Everyone,

I got into this issue and I was testing the patch in a lab environment when I found some conflicting documentation between OVS and OVN.

This patch enforces that the system-id must be a UUID string, this is consistent with OVS documentation[0]:

"""
--system-id=<uuid> or --system-id=random

This specifies a unique system identifier to store into external-ids:system-id in the database’s Open_vSwitch table.[...]
"""

Now when we go to OVN RBAC documentation[1] we get the following:

"""
Suppose there are three machines in your deployment. machine_1 runs chassis_1 and has IP address machine_1-ip.[...]

NOTE: chassis_1 must be the same string as external_ids:system-id in the Open_vSwitch table (the chassis name) of machine_1. Same applies for chassis_2.
"""

One of the commits added testing for this functionality[2] and system-id is set to the same value of hostname[3]

If OVN RBAC starts using a UUID for the system-id (instead of a hostname), the certificates would need to be generated using that UUID which would be strange from the operations point of view.

Best,

[0] https://docs.openvswitch.org/en/latest/ref/ovs-ctl.8/#options
[1] https://docs.ovn.org/en/latest/tutorials/ovn-rbac.html
[2] https://github.com/ovn-org/ovn/commit/c948d6bb05b4d8d34db7a88590eddb4c6de2b3c4
[3] https://github.com/ovn-org/ovn/blob/main/tests/ovn-macros.at#L322-L323

[Frode Nordahl (fnordahl)]

Adding to #8, apart from Neutron breaking the OVN RBAC use case with this enforcement, the Open_vSwitch table external_ids column is a string key/string value type map [4]. For these maps the schema documentation may contain type hints, but this is also not the case for this key [5].

4: https://github.com/openvswitch/ovs/blob/38b42aa93fed2f2eeadb5da3feb4e9ab5a4d6122/vswitchd/vswitch.ovsschema#L27-L29
5: https://github.com/openvswitch/ovs/blob/38b42aa93fed2f2eeadb5da3feb4e9ab5a4d6122/vswitchd/vswitch.xml#L70-L75

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hello Frode:

OVS provides support for other host types like XenServer. In OpenStack we dropped this support.

The OVS system-id (not the xs-system-uuid) must be a UUID type identifier [1]. The Neutron OVN backend, specifically the OVN metadata agent, will check value format before starting. We cannot accept a random string as you are expecting. This enforcement is not breaking any OVN RBAC because that is enforced in OVS, not OVN.

Regards.

[1]https://www.apt-browse.org/browse/ubuntu/xenial/main/amd64/openvswitch-switch/2.5.0-0ubuntu1/file/usr/share/openvswitch/scripts/ovs-ctl

[Frode Nordahl (fnordahl)]

Rodolfo,

This is not enforced in OVS either as I pointed out in #9. The system-id is a string value in the OVS database schema. [4][5]

It is used by OVN for the chassis name [6][7], which is also a string value in the OVN database scheme.

Why does Neutron need to impose this check when it clearly breaks multiple existing users? What does this enforcement gain us?

How would you feel if you had to issue SSL certificates with UUID's as CN's? Would you feel that was a useful thing of a system have you do?

6: https://github.com/ovn-org/ovn/blob/8fd4cecf3034848d222e023a68a03e9a8eadf621/ovn-sb.ovsschema#L29
7: https://github.com/ovn-org/ovn/blob/8fd4cecf3034848d222e023a68a03e9a8eadf621/ovn-sb.xml#L241-L250

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/822328

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/822328
Committed: https://opendev.org/openstack/neutron/commit/79037c951637dc06d47b6d354776d116a1d2a9ad
Submitter: "Zuul (22348)"
Branch: master

commit 79037c951637dc06d47b6d354776d116a1d2a9ad
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Dec 20 14:14:20 2021 +0000

    [OVN] Accept OVS system-id as non UUID formatted string

    Accept OVS system-id non UUID formatted strings. The OVN metadata
    agent will generate a unique UUID from the OVS system-id. If this
    string is a UUID, this value will be used. If not, the OVN metadata
    agent will generate a UUID based on the provided string.

    This patch amends [1].

    [1]https://review.opendev.org/c/openstack/neutron/+/819634

    Closes-Bug: #1952550

    Change-Id: I42a8a767a6ef9454419b26f80339394759644faf

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/822667

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Rodolfo Alonso <email address hidden>" on branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/821828

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Rodolfo Alonso <email address hidden>" on branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/neutron/+/821831

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Rodolfo Alonso <email address hidden>" on branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/821830

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Rodolfo Alonso <email address hidden>" on branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/821829

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/822668

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/822669

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/neutron/+/822691

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/822668
Committed: https://opendev.org/openstack/neutron/commit/b07eeb2789abc79c0fa86db0bdf7bc111ea725ac
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit b07eeb2789abc79c0fa86db0bdf7bc111ea725ac
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Dec 20 14:14:20 2021 +0000

    [OVN] Accept OVS system-id as non UUID formatted string

    Accept OVS system-id non UUID formatted strings. The OVN metadata
    agent will generate a unique UUID from the OVS system-id. If this
    string is a UUID, this value will be used. If not, the OVN metadata
    agent will generate a UUID based on the provided string.

    This patch amends [1].

    [1]https://review.opendev.org/c/openstack/neutron/+/819634

    Closes-Bug: #1952550

    Conflicts:
            neutron/agent/ovn/metadata/agent.py
            neutron/tests/unit/agent/ovn/metadata/test_agent.py

    Change-Id: I42a8a767a6ef9454419b26f80339394759644faf
    (cherry picked from commit 79037c951637dc06d47b6d354776d116a1d2a9ad)
    (cherry picked from commit 6da4432fed255f3bcf3831f5d0520ab389ce36e5)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/822667
Committed: https://opendev.org/openstack/neutron/commit/6da4432fed255f3bcf3831f5d0520ab389ce36e5
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 6da4432fed255f3bcf3831f5d0520ab389ce36e5
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Dec 20 14:14:20 2021 +0000

    [OVN] Accept OVS system-id as non UUID formatted string

    Accept OVS system-id non UUID formatted strings. The OVN metadata
    agent will generate a unique UUID from the OVS system-id. If this
    string is a UUID, this value will be used. If not, the OVN metadata
    agent will generate a UUID based on the provided string.

    This patch amends [1].

    [1]https://review.opendev.org/c/openstack/neutron/+/819634

    Closes-Bug: #1952550

    Conflicts:
            neutron/agent/ovn/metadata/agent.py

    Change-Id: I42a8a767a6ef9454419b26f80339394759644faf
    (cherry picked from commit 79037c951637dc06d47b6d354776d116a1d2a9ad)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/822669
Committed: https://opendev.org/openstack/neutron/commit/4bf531448eac151b53b05cae0a5ca51060ca38c3
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 4bf531448eac151b53b05cae0a5ca51060ca38c3
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Dec 20 14:14:20 2021 +0000

    [OVN] Accept OVS system-id as non UUID formatted string

    Accept OVS system-id non UUID formatted strings. The OVN metadata
    agent will generate a unique UUID from the OVS system-id. If this
    string is a UUID, this value will be used. If not, the OVN metadata
    agent will generate a UUID based on the provided string.

    This patch amends [1].

    [1]https://review.opendev.org/c/openstack/neutron/+/819634

    Closes-Bug: #1952550

    Conflicts:
            neutron/agent/ovn/metadata/agent.py
            neutron/tests/unit/agent/ovn/metadata/test_agent.py

    Change-Id: I42a8a767a6ef9454419b26f80339394759644faf
    (cherry picked from commit 79037c951637dc06d47b6d354776d116a1d2a9ad)
    (cherry picked from commit 6da4432fed255f3bcf3831f5d0520ab389ce36e5)
    (cherry picked from commit b07eeb2789abc79c0fa86db0bdf7bc111ea725ac)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/822691
Committed: https://opendev.org/openstack/neutron/commit/162b02195c41d93f3ef8af77dafd460877d1bb2c
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 162b02195c41d93f3ef8af77dafd460877d1bb2c
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Dec 20 14:14:20 2021 +0000

    [OVN] Accept OVS system-id as non UUID formatted string

    Accept OVS system-id non UUID formatted strings. The OVN metadata
    agent will generate a unique UUID from the OVS system-id. If this
    string is a UUID, this value will be used. If not, the OVN metadata
    agent will generate a UUID based on the provided string.

    This patch amends [1].

    [1]https://review.opendev.org/c/openstack/neutron/+/819634

    Closes-Bug: #1952550

    Conflicts:
            neutron/agent/ovn/metadata/agent.py
            neutron/tests/unit/agent/ovn/metadata/test_agent.py

    Change-Id: I42a8a767a6ef9454419b26f80339394759644faf
    (cherry picked from commit 79037c951637dc06d47b6d354776d116a1d2a9ad)
    (cherry picked from commit 6da4432fed255f3bcf3831f5d0520ab389ce36e5)
    (cherry picked from commit b07eeb2789abc79c0fa86db0bdf7bc111ea725ac)
    (cherry picked from commit 4bf531448eac151b53b05cae0a5ca51060ca38c3)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 19.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 17.3.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 18.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 20.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron ussuri-eol release.