[OVN]Metadata get warn logs after boot instance server about "MetadataServiceReadyWaitTimeoutException"

Bug #1946588 reported by huanghailun
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
High
huanghailun

Bug Description

The operation:
1. Create virtual machines in batches
2. View neutron-server logs

Warning logs as:
2021-10-10 19:18:08.384 78 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [req-18282ec4-8752-4c7b-8cb8-3f54a02377ba - - - - -] Metadata service is not ready for port 5466933e-a22e-4288-b40c-ddc01fe23cfa, check neutron-ovn-metadata-agent status/logs.: neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.MetadataServiceReadyWaitTimeoutException

This is due to:
The value about neutron-metadata-proxy-networks of Chassis's external_ids are network_id, and the datapath field of Port_Binding table is the uuid of Datapath_Binding tables.

When call function set_port_status_up, it check self._wait_for_metadata_provisioned_if_needed(port_id)

```
            ...
            try:
                n_utils.wait_until_true(
                    lambda: datapath in
                    self._sb_ovn.get_chassis_metadata_networks(chassis),
                    timeout=METADATA_READY_WAIT_TIMEOUT,
                    exception=MetadataServiceReadyWaitTimeoutException)
            except MetadataServiceReadyWaitTimeoutException:
                # If we reach this point it means that metadata agent didn't
                # provision the datapath for this port on its chassis. Either
                # the agent is not running or it crashed. We'll complete the
                # provisioning block though.
                LOG.warning("Metadata service is not ready for port %s, check"
                            " neutron-ovn-metadata-agent status/logs.",
                            port_id)
            ...
```

Get datapath:
    def get_logical_port_chassis_and_datapath(self, name):
        for port in self._tables['Port_Binding'].rows.values():
            if port.logical_port == name:
                datapath = str(port.datapath.uuid)
                chassis = port.chassis[0].name if port.chassis else None
                return chassis, datapath

Get neutron-metadata-proxy-networks:
    def get_chassis_metadata_networks(self, chassis_name):
        """Return a list with the metadata networks the chassis is hosting."""
        try:
            chassis = self.lookup('Chassis', chassis_name)
        except idlutils.RowNotFound:
            LOG.warning("Couldn't find Chassis named %s in OVN while looking "
                        "for metadata networks", chassis_name)
            return []
        proxy_networks = chassis.external_ids.get(
            'neutron-metadata-proxy-networks', None)
        return proxy_networks.split(',') if proxy_networks else []

I fix it by this way:
    def get_logical_port_chassis_and_datapath(self, name):
        for port in self._tables['Port_Binding'].rows.values():
            if port.logical_port == name:
                network_id = utils.get_network_name_from_datapath(
                    port.datapath)
                chassis = port.chassis[0].name if port.chassis else None
                return chassis, network_id

Version: stable/wallaby

huanghailun (lnun)
tags: added: metadata ovn
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/813376

Revision history for this message
Rodolfo Alonso (rodolfo-alonso-hernandez) wrote :

Hello:

This error could be affecting from Ussuri to Wallaby. In Xena this code was removed in [1].

Regards.

[1]https://github.com/openstack/neutron/commit/36ba1cc3196148850263c8a8c8058e6f7ced3958#diff-f2c894a40a002f3c9b2591bd0619f65b9582f0b9001a6564284479b0acd1f170

Changed in neutron:
importance: Undecided → High
assignee: nobody → hh (lnun)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/813376
Committed: https://opendev.org/openstack/neutron/commit/f5ea47b09507e7bd620ba3bcd1af771238d79d6d
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit f5ea47b09507e7bd620ba3bcd1af771238d79d6d
Author: hailun.huang <email address hidden>
Date: Mon Oct 11 11:25:45 2021 +0800

    [stable only][ovn]Update get datapath id to network from Port_Binding

    The value about neutron-metadata-proxy-networks of Chassis's
     external_ids are network_id, and the datapath field of
     Port_Binding table is the uuid of Datapath_Binding tables.
    When call function set_port_status_up after boot server instance,
     it check datapath in neutron-metadata-proxy-networks of
     Chassis's external_ids, the datapath id is not match with
     network id, so change get datapath id from Port_Binding
     to get network id.
    Since [1], this is not needed in Xena.
    [1]https://review.opendev.org/c/openstack/neutron/+/791997

    Closes-Bug: #1946588
    Change-Id: Ib3c1bea9b805dc7d9967f66b3b0136efb3e8e08d

tags: added: in-stable-wallaby
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/victoria)

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/823591

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/neutron/+/823592

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/ussuri)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/823592
Committed: https://opendev.org/openstack/neutron/commit/135e618c22326d6a23b47662d16443a43db840bb
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 135e618c22326d6a23b47662d16443a43db840bb
Author: hailun.huang <email address hidden>
Date: Mon Oct 11 11:25:45 2021 +0800

    [stable only][ovn]Update get datapath id to network from Port_Binding

    The value about neutron-metadata-proxy-networks of Chassis's
     external_ids are network_id, and the datapath field of
     Port_Binding table is the uuid of Datapath_Binding tables.
    When call function set_port_status_up after boot server instance,
     it check datapath in neutron-metadata-proxy-networks of
     Chassis's external_ids, the datapath id is not match with
     network id, so change get datapath id from Port_Binding
     to get network id.
    Since [1], this is not needed in Xena.
    [1]https://review.opendev.org/c/openstack/neutron/+/791997

    Closes-Bug: #1946588
    Change-Id: Ib3c1bea9b805dc7d9967f66b3b0136efb3e8e08d
    (cherry picked from commit f5ea47b09507e7bd620ba3bcd1af771238d79d6d)
    (cherry picked from commit cff314f92e158c8142dcd238a072fe2a7ca04d87)

tags: added: in-stable-ussuri
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/victoria)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/823591
Committed: https://opendev.org/openstack/neutron/commit/cff314f92e158c8142dcd238a072fe2a7ca04d87
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit cff314f92e158c8142dcd238a072fe2a7ca04d87
Author: hailun.huang <email address hidden>
Date: Mon Oct 11 11:25:45 2021 +0800

    [stable only][ovn]Update get datapath id to network from Port_Binding

    The value about neutron-metadata-proxy-networks of Chassis's
     external_ids are network_id, and the datapath field of
     Port_Binding table is the uuid of Datapath_Binding tables.
    When call function set_port_status_up after boot server instance,
     it check datapath in neutron-metadata-proxy-networks of
     Chassis's external_ids, the datapath id is not match with
     network id, so change get datapath id from Port_Binding
     to get network id.
    Since [1], this is not needed in Xena.
    [1]https://review.opendev.org/c/openstack/neutron/+/791997

    Closes-Bug: #1946588
    Change-Id: Ib3c1bea9b805dc7d9967f66b3b0136efb3e8e08d
    (cherry picked from commit f5ea47b09507e7bd620ba3bcd1af771238d79d6d)

tags: added: in-stable-victoria
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 17.3.0

This issue was fixed in the openstack/neutron 17.3.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 18.2.0

This issue was fixed in the openstack/neutron 18.2.0 release.

Changed in neutron:
status: New → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron ussuri-eol

This issue was fixed in the openstack/neutron ussuri-eol release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.