Nova fails to live migrate instance with upper-case port MAC

[Dmitriy Rabotyagov]

Description
===========

When neutron port has MAC address defined in upper case and libvirt stores MAC in XML in lower case, migration is failed with KeyError:
```
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: 2021-09-30 10:31:38.028 3054313 ERROR nova.virt.libvirt.driver [req-911a4b70-5448-48a1-afa4-1bbd0b38737b - - - - -] [instance: 75f7
9d85-6505-486c-bc34-e78fd6350a77] Live Migration failure: '00:50:56:af:e1:73'
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: Traceback (most recent call last):
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: File "/openstack/venvs/nova-22.3.1/lib/python3.8/site-packages/eventlet/hubs/hub.py", line 461, in fire_timers
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: timer()
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: File "/openstack/venvs/nova-22.3.1/lib/python3.8/site-packages/eventlet/hubs/timer.py", line 59, in __call__
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: cb(*args, **kw)
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: File "/openstack/venvs/nova-22.3.1/lib/python3.8/site-packages/eventlet/event.py", line 175, in _do_send
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: waiter.switch(result)
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: File "/openstack/venvs/nova-22.3.1/lib/python3.8/site-packages/eventlet/greenthread.py", line 221, in main
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: result = function(*args, **kwargs)
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: File "/openstack/venvs/nova-22.3.1/lib/python3.8/site-packages/nova/utils.py", line 661, in context_wrapper
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: return func(*args, **kwargs)
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: File "/openstack/venvs/nova-22.3.1/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 9196, in _live_migration_operation
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: LOG.error("Live Migration failure: %s", e, instance=instance)
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: File "/openstack/venvs/nova-22.3.1/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: self.force_reraise()
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: File "/openstack/venvs/nova-22.3.1/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: six.reraise(self.type_, self.value, self.tb)
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: File "/openstack/venvs/nova-22.3.1/lib/python3.8/site-packages/six.py", line 703, in reraise
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: raise value
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: File "/openstack/venvs/nova-22.3.1/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 9152, in _live_migration_operation
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: new_xml_str = libvirt_migrate.get_updated_guest_xml(
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: File "/openstack/venvs/nova-22.3.1/lib/python3.8/site-packages/nova/virt/libvirt/migration.py", line 65, in get_updated_guest_xml
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: xml_doc = _update_vif_xml(xml_doc, migrate_data, get_vif_config)
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: File "/openstack/venvs/nova-22.3.1/lib/python3.8/site-packages/nova/virt/libvirt/migration.py", line 355, in _update_vif_xml
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: migrate_vif = migrate_vif_by_mac[mac_addr]
Sep 30 10:31:38 cc-compute08-dx1 nova-compute[3054313]: KeyError: '00:50:56:af:e1:73'
```

Environment
===========

Ubuntu 20.04
Libvirt 6.0.0-0ubuntu8.14
Nova 22.2.3.dev2 (sha 4ce01d6c49f81b6b2438549b01a89ea1b5956320)
Neutron with OpenVSwitch

[sean mooney (sean-k-mooney)]

adding neutron as i think neutron shoudl also be normaliasing the mac adress that users provide and alwasy storign it in lower case. a mac is technically a number not a string we just use hex encoding for human readablity so the caseing does not matter but it would be nice to at least consider moving this normalisation to the neutron api/db to avoid this problem.

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hello:

The MAC address sanitization was included in Neutron Xena release (19). I see you are using Nova 23 (Xena release too).

This support was added in:
- https://review.opendev.org/c/openstack/neutron/+/789831
- https://review.opendev.org/c/openstack/neutron-lib/+/788300

You need to update to the latests neutron-lib and Neutron Xena versions. To sanitize the existing MACs in the port records, you can execute "neutron-sanitize-port-mac-addresses" script. It will read all port MAC addresses and format as "xx:xx:xx:xx:xx:xx".

Regards.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/811947
Committed: https://opendev.org/openstack/nova/commit/6a15169ed9f16672c2cde1d7f27178bb7868c41f
Submitter: "Zuul (22348)"
Branch: master

commit 6a15169ed9f16672c2cde1d7f27178bb7868c41f
Author: Dmitriy Rabotyagov <email address hidden>
Date: Thu Sep 30 14:08:38 2021 +0300

    Ensure MAC addresses characters are in the same case

    Currently neutron can report ports to have MAC addresses
    in upper case when they're created like that. In the meanwhile
    libvirt configuration file always stores MAC in lower case
    which leads to KeyError while trying to retrieve migrate_vif.

    Closes-Bug: #1945646
    Change-Id: Ie3129ee395427337e9abcef2f938012608f643e1

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/nova/+/816882

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/816883

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/nova/+/816927

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/nova/+/817689

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/nova/+/817830

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/816882
Committed: https://opendev.org/openstack/nova/commit/63a6388f6a0265f84232731aba8aec1bff3c6d18
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 63a6388f6a0265f84232731aba8aec1bff3c6d18
Author: Dmitriy Rabotyagov <email address hidden>
Date: Thu Sep 30 14:08:38 2021 +0300

    Ensure MAC addresses characters are in the same case

    Currently neutron can report ports to have MAC addresses
    in upper case when they're created like that. In the meanwhile
    libvirt configuration file always stores MAC in lower case
    which leads to KeyError while trying to retrieve migrate_vif.

    Closes-Bug: #1945646
    Change-Id: Ie3129ee395427337e9abcef2f938012608f643e1
    (cherry picked from commit 6a15169ed9f16672c2cde1d7f27178bb7868c41f)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/816883
Committed: https://opendev.org/openstack/nova/commit/6c3d5de659e558e8f6ee353475b54ff3ca7240ee
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 6c3d5de659e558e8f6ee353475b54ff3ca7240ee
Author: Dmitriy Rabotyagov <email address hidden>
Date: Thu Sep 30 14:08:38 2021 +0300

    Ensure MAC addresses characters are in the same case

    Currently neutron can report ports to have MAC addresses
    in upper case when they're created like that. In the meanwhile
    libvirt configuration file always stores MAC in lower case
    which leads to KeyError while trying to retrieve migrate_vif.

    Closes-Bug: #1945646
    Change-Id: Ie3129ee395427337e9abcef2f938012608f643e1
    (cherry picked from commit 6a15169ed9f16672c2cde1d7f27178bb7868c41f)
    (cherry picked from commit 63a6388f6a0265f84232731aba8aec1bff3c6d18)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/816927
Committed: https://opendev.org/openstack/nova/commit/28d0059c1f52e51add31bff50f1f6e443c938792
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 28d0059c1f52e51add31bff50f1f6e443c938792
Author: Dmitriy Rabotyagov <email address hidden>
Date: Thu Sep 30 14:08:38 2021 +0300

    Ensure MAC addresses characters are in the same case

    Currently neutron can report ports to have MAC addresses
    in upper case when they're created like that. In the meanwhile
    libvirt configuration file always stores MAC in lower case
    which leads to KeyError while trying to retrieve migrate_vif.

    Conflicts:
      nova/tests/unit/virt/libvirt/test_migration.py

    Note: conflict is caused by not having six.text_type removal patch
    I779bd1446dc1f070fa5100ccccda7881fa508d79 in stable/victoria.
    Original assertion method was preserved to resolve this conflict.

    Closes-Bug: #1945646
    Change-Id: Ie3129ee395427337e9abcef2f938012608f643e1
    (cherry picked from commit 6a15169ed9f16672c2cde1d7f27178bb7868c41f)
    (cherry picked from commit 63a6388f6a0265f84232731aba8aec1bff3c6d18)
    (cherry picked from commit 6c3d5de659e558e8f6ee353475b54ff3ca7240ee)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/817689
Committed: https://opendev.org/openstack/nova/commit/184a3c976faed38907af148a533bc6e9faa410f5
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 184a3c976faed38907af148a533bc6e9faa410f5
Author: Dmitriy Rabotyagov <email address hidden>
Date: Thu Sep 30 14:08:38 2021 +0300

    Ensure MAC addresses characters are in the same case

    Currently neutron can report ports to have MAC addresses
    in upper case when they're created like that. In the meanwhile
    libvirt configuration file always stores MAC in lower case
    which leads to KeyError while trying to retrieve migrate_vif.

    Closes-Bug: #1945646
    Change-Id: Ie3129ee395427337e9abcef2f938012608f643e1
    (cherry picked from commit 6a15169ed9f16672c2cde1d7f27178bb7868c41f)
    (cherry picked from commit 63a6388f6a0265f84232731aba8aec1bff3c6d18)
    (cherry picked from commit 6c3d5de659e558e8f6ee353475b54ff3ca7240ee)
    (cherry picked from commit 28d0059c1f52e51add31bff50f1f6e443c938792)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/817830
Committed: https://opendev.org/openstack/nova/commit/a5da31ec1ea1d1c7b4df146857982699ebdc328e
Submitter: "Zuul (22348)"
Branch: stable/train

commit a5da31ec1ea1d1c7b4df146857982699ebdc328e
Author: Dmitriy Rabotyagov <email address hidden>
Date: Thu Sep 30 14:08:38 2021 +0300

    Ensure MAC addresses characters are in the same case

    Currently neutron can report ports to have MAC addresses
    in upper case when they're created like that. In the meanwhile
    libvirt configuration file always stores MAC in lower case
    which leads to KeyError while trying to retrieve migrate_vif.

    Closes-Bug: #1945646
    Change-Id: Ie3129ee395427337e9abcef2f938012608f643e1
    (cherry picked from commit 6a15169ed9f16672c2cde1d7f27178bb7868c41f)
    (cherry picked from commit 63a6388f6a0265f84232731aba8aec1bff3c6d18)
    (cherry picked from commit 6c3d5de659e558e8f6ee353475b54ff3ca7240ee)
    (cherry picked from commit 28d0059c1f52e51add31bff50f1f6e443c938792)
    (cherry picked from commit 184a3c976faed38907af148a533bc6e9faa410f5)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 22.4.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 23.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 24.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 25.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/c/openstack/nova/+/855553

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/stein
Review: https://review.opendev.org/c/openstack/nova/+/855553
Reason: This branch transitioned to End of Life for this project, open patches needs to be closed to be able to delete the branch.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova train-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova ussuri-eol release.