Unexpected number of DHCP interfaces for metadata proxy

[Maximilian Stinsky]

We upgraded neutron from ussuri to victoria in our lab environment and are seeing a strange error message that I dont really understand.

The dhcp agent is logging the following for every network: Unexpected number of DHCP interfaces for metadata proxy, expected 1, got 3

This seems to be related to the new metadata v6 implementation. [1]
But as we are running with dhcp_agents_per_network = 3 it is expected that we have 3 dhcp ports that are admin state up which have the dhcp device owner.

So far I dont understand what would be the right way to eliminate the error.

[1] https://github.com/openstack/neutron/blob/17.4.1/neutron/agent/dhcp/agent.py#L799

[Miguel Lavalle (minsel)]

I think I can replicate this with master branch code in a devstack with two nodes:

1) Show nodes in system

$ openstack host list
+-----------+-----------+----------+
| Host Name | Service | Zone |
+-----------+-----------+----------+
| central | scheduler | internal |
| central | conductor | internal |
| central | conductor | internal |
| central | compute | nova |
| worker | compute | nova |
+-----------+-----------+----------+

2) Configure more that 2 dhcp agents per network and re-start Neutron server:

vagrant@central:~$ cat /etc/neutron/neutron.conf | grep dhcp_agents_per_network
dhcp_agents_per_network = 2

3) Create network and subnets as non-admin:

$ export OS_CLOUD=ovs

$ openstack network create mi-red -c id -c name
+-------+--------------------------------------+
| Field | Value |
+-------+--------------------------------------+
| id | ea851d66-3a99-4a6c-97d7-31f105970bb5 |
| name | mi-red |
+-------+--------------------------------------+

$ openstack subnet create --ip-version 4 --subnet-pool dff62ef8-abc3-47fe-aa4f-3a357079ceed --network mi-red mi-red-subnet -c id -c name -c cidr
+-------+--------------------------------------+
| Field | Value |
+-------+--------------------------------------+
| cidr | 10.0.0.64/26 |
| id | 7083f53c-3c1c-440c-9e6a-acbb8a2942d0 |
| name | mi-red-subnet |
+-------+--------------------------------------+

$ openstack subnet create --ip-version 6 --subnet-pool bc427dc1-87e5-4c14-839c-7830e661706b --ipv6-ra-mode slaac --ipv6-address-mode slaac --network mi-red mi-red-subnet-ipv6 -c id -c name -c cidr
+-------+--------------------------------------+
| Field | Value |
+-------+--------------------------------------+
| cidr | fde8:d672:ad20:1::/64 |
| id | 5e2f17b0-9745-4624-87e8-41defce3497f |
| name | mi-red-subnet-ipv6 |
+-------+--------------------------------------+

4) Check how many dhcp ports with admin_state_up = true are there for the network:

$ export OS_CLOUD=ovs-admin

$ openstack network agent list --network ea851d66-3a99-4a6c-97d7-31f105970bb5 -c ID -c Alive -c Host
+--------------------------------------+---------+-------+
| ID | Host | Alive |
+--------------------------------------+---------+-------+
| 1d90cd00-5c31-49cc-9c97-d8133ec8ba2e | worker | :-) |
| 77866a27-3537-4725-a13a-d74bf681e384 | central | :-) |
+--------------------------------------+---------+-------+

$ openstack port list --device-owner network:dhcp -c ID -c 'Fixed IP Addresses' | grep 7083f53c-3c1c-440c-9e6a-acbb8a2942d0
| 2d82947a-72af-449d-a0f4-8fde2f261136 | ip_address='10.0.0.67', subnet_id='7083f53c-3c1c-440c-9e6a-acbb8a2942d0' |
| 6848ab9d-b814-4170-9fdc-7449fcebc26a | ip_address='10.0.0.66', subnet_id='7083f53c-3c1c-440c-9e6a-acbb8a2942d0' |

$ openstack port list --device-owner network:dhcp -c ID -c 'Fixed IP Addresses' | grep 7083f53c-3c1c-440c-9e6a-acbb8a2942d0...

[Maximilian Stinsky (mstinsky)]

Hi Miguel,

yes, the issue as you replicated it seems to be exactly what I am seeing in my lab environment.

[Maximilian Stinsky (mstinsky)]

Do you know of any side effects running into this error message?

So far I don't see anything breaking in my lab environment even with the error.
All tempest tests are successfull and haproxy also binds correctly on the new ipv6 metadata address.

[Bence Romsics (bence-romsics)]

Having written https://review.opendev.org/c/openstack/neutron/+/715482 I remember not knowing what conditions would lead to this else case (https://github.com/openstack/neutron/blob/37c2e6b708678d2e648f111d102b6394f4e704b0/neutron/agent/dhcp/agent.py#L802) to run. That's why I left an error log without throwing an exception.

Now re-reading the code I'm still lost a bit: Why would the dhcp agent care about ports hosted on another agent? That's why we have more than one ports in the result, don't we? A port hosted on another agent cannot have a local network interface (because that interface is on another host). At first look it seems to me we may need to filter the port list down to ports hosted by the current agent.

Other then a spurious error message the following could be tested to see if we have a functional error: On an isolated network (i.e. a network not connected to any neutron router) does the metadata service work? It may make sense to run this test multiple times because the list order here (https://github.com/openstack/neutron/blob/37c2e6b708678d2e648f111d102b6394f4e704b0/neutron/agent/dhcp/agent.py#L800) may be arbitrary and it may happen that when the first interface in the list is the right one, metadata works, but other times it does not.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/858835

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/858835
Committed: https://opendev.org/openstack/neutron/commit/41a7479d0f93ec2ff5019a1bfdc891dda5078d65
Submitter: "Zuul (22348)"
Branch: master

commit 41a7479d0f93ec2ff5019a1bfdc891dda5078d65
Author: Miguel Lavalle <email address hidden>
Date: Wed Sep 21 18:26:39 2022 -0500

    Handle several dhcp agents for metadata over ipv6

    [0] added the capability to the dhcp agent of handling metadata traffic
    over ipv6. However, the added code doesn't handle the case when a
    network is managed by more than one dhcp agent. This change fixes the
    issue.

    [0] https://review.opendev.org/c/openstack/neutron/+/715482

    Change-Id: Ie1e0739f2c7390031af196e72c678ee626d0f8b8
    Closes-Bug: #1989979

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/neutron/+/858898

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/858899

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/858900

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/858901

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/neutron/+/858902

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/858902
Committed: https://opendev.org/openstack/neutron/commit/611218db9089f2127a7ae48f3419fca3e542f88f
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 611218db9089f2127a7ae48f3419fca3e542f88f
Author: Miguel Lavalle <email address hidden>
Date: Wed Sep 21 18:26:39 2022 -0500

    Handle several dhcp agents for metadata over ipv6

    [0] added the capability to the dhcp agent of handling metadata traffic
    over ipv6. However, the added code doesn't handle the case when a
    network is managed by more than one dhcp agent. This change fixes the
    issue.

    [0] https://review.opendev.org/c/openstack/neutron/+/715482

    Change-Id: Ie1e0739f2c7390031af196e72c678ee626d0f8b8
    Closes-Bug: #1989979
    (cherry picked from commit 41a7479d0f93ec2ff5019a1bfdc891dda5078d65)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/858898
Committed: https://opendev.org/openstack/neutron/commit/ec9eabf8dd09a233e0a194e1c9a0a363f79a5b9e
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit ec9eabf8dd09a233e0a194e1c9a0a363f79a5b9e
Author: Miguel Lavalle <email address hidden>
Date: Wed Sep 21 18:26:39 2022 -0500

    Handle several dhcp agents for metadata over ipv6

    [0] added the capability to the dhcp agent of handling metadata traffic
    over ipv6. However, the added code doesn't handle the case when a
    network is managed by more than one dhcp agent. This change fixes the
    issue.

    [0] https://review.opendev.org/c/openstack/neutron/+/715482

    Change-Id: Ie1e0739f2c7390031af196e72c678ee626d0f8b8
    Closes-Bug: #1989979
    (cherry picked from commit 41a7479d0f93ec2ff5019a1bfdc891dda5078d65)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/858899
Committed: https://opendev.org/openstack/neutron/commit/fd7993cfe0a00f099ead9e3c9a3ee126e1f12488
Submitter: "Zuul (22348)"
Branch: stable/xena

commit fd7993cfe0a00f099ead9e3c9a3ee126e1f12488
Author: Miguel Lavalle <email address hidden>
Date: Wed Sep 21 18:26:39 2022 -0500

    Handle several dhcp agents for metadata over ipv6

    [0] added the capability to the dhcp agent of handling metadata traffic
    over ipv6. However, the added code doesn't handle the case when a
    network is managed by more than one dhcp agent. This change fixes the
    issue.

    [0] https://review.opendev.org/c/openstack/neutron/+/715482

    Change-Id: Ie1e0739f2c7390031af196e72c678ee626d0f8b8
    Closes-Bug: #1989979
    (cherry picked from commit 41a7479d0f93ec2ff5019a1bfdc891dda5078d65)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/858901
Committed: https://opendev.org/openstack/neutron/commit/76c194c97742a09adc343faf7a7368e327e90752
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 76c194c97742a09adc343faf7a7368e327e90752
Author: Miguel Lavalle <email address hidden>
Date: Wed Sep 21 18:26:39 2022 -0500

    Handle several dhcp agents for metadata over ipv6

    [0] added the capability to the dhcp agent of handling metadata traffic
    over ipv6. However, the added code doesn't handle the case when a
    network is managed by more than one dhcp agent. This change fixes the
    issue.

    [0] https://review.opendev.org/c/openstack/neutron/+/715482

    Change-Id: Ie1e0739f2c7390031af196e72c678ee626d0f8b8
    Closes-Bug: #1989979
    (cherry picked from commit 41a7479d0f93ec2ff5019a1bfdc891dda5078d65)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/858900
Committed: https://opendev.org/openstack/neutron/commit/1c0c03eaac0964e77ef52efa9eae46eae0c2c742
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 1c0c03eaac0964e77ef52efa9eae46eae0c2c742
Author: Miguel Lavalle <email address hidden>
Date: Wed Sep 21 18:26:39 2022 -0500

    Handle several dhcp agents for metadata over ipv6

    [0] added the capability to the dhcp agent of handling metadata traffic
    over ipv6. However, the added code doesn't handle the case when a
    network is managed by more than one dhcp agent. This change fixes the
    issue.

    [0] https://review.opendev.org/c/openstack/neutron/+/715482

    Change-Id: Ie1e0739f2c7390031af196e72c678ee626d0f8b8
    Closes-Bug: #1989979
    (cherry picked from commit 41a7479d0f93ec2ff5019a1bfdc891dda5078d65)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 21.0.0.0rc2 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 18.6.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 19.5.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 22.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 20.3.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron victoria-eom release.