Nova fails to live migrate instance with dash separated MAC

[Dmitriy Rabotyagov]

Description
===========

In case network port was created with manually defined MAC address and dash ("-") was used as a separator instead of semi-column, attempt of server migration with that port will lead to failure

Steps to reproduce
==================

1. Manually create a port
openstack port create --mac-address "FA-16-3E-7B-D7-F4" --network f522fb2d-05b2-43e9-bcca-60d06620747d invalid_mac

2. Create a VM from that port:
openstack server create --flavor 1c1gb --image cirros --port 3642e1de-bada-4fc2-a010-ddde1589c154 --boot-from-volume 20 test_mac

3. Try to migrate server:
openstack server migrate --live 9e948ba7-4137-462a-91aa-7e9a64f52a9d

Expected result
===============

Instance is migrated

Actual result
=============

Live migration fails with stack trace

Environment
===========

Nova SHA: cf2abde7ee9437bbd3735d30b9b8e1dcb04a4bda
HV: Libvirt 8.0.0 + QEMU 4.2
Storage: Ceph
Networking: OVS

Logs
====

May 15 14:03:23 cc-compute12-dx1 nova-compute[855142]: 2024-05-15 14:03:23.478 855142 INFO nova.compute.manager [None req-3b021e43-ca4b-4c67-a094-6ce2152daeb6 5882947e723e4afdb15f5e2679f14c0e 4d866b39b4df4ab7a8d84c6a9d05bd3d - - default default] [instance: 9e948ba7-4137-462a-91aa-7e9a64f52a9d] Took 2.75 seconds for pre_live_migration on destination host cc-compute05-dx1.
May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]: 2024-05-15 14:03:24.803 855142 ERROR nova.virt.libvirt.driver [None req-3b021e43-ca4b-4c67-a094-6ce2152daeb6 5882947e723e4afdb15f5e2679f14c0e 4d866b39b4df4ab7a8d84c6a9d05bd3d - - default default] [instance: 9e948ba7-4137-462a-91aa-7e9a64f52a9d] Live Migration failure: 'fa:16:3e:7b:d7:f4'
May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]: Traceback (most recent call last):
May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]: File "/openstack/venvs/nova-27.3.0/lib/python3.8/site-packages/eventlet/hubs/hub.py", line 476, in fire_timers
May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]: timer()
May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]: File "/openstack/venvs/nova-27.3.0/lib/python3.8/site-packages/eventlet/hubs/timer.py", line 59, in __call__
May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]: cb(*args, **kw)
May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]: File "/openstack/venvs/nova-27.3.0/lib/python3.8/site-packages/eventlet/event.py", line 175, in _do_send
May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]: waiter.switch(result)
May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]: File "/openstack/venvs/nova-27.3.0/lib/python3.8/site-packages/eventlet/greenthread.py", line 221, in main
May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]: result = function(*args, **kwargs)
May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]: File "/openstack/venvs/nova-27.3.0/lib/python3.8/site-packages/nova/utils.py", line 654, in context_wrapper
May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]: return func(*args, **kwargs)
May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]: File "/openstack/venvs/nova-27.3.0/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 10252, in _live_migration_operation
May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]: LOG.error("Live Migration failure: %s", e, instance=instance)
May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]: File "/openstack/venvs/nova-27.3.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]: self.force_reraise()
May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]: File "/openstack/venvs/nova-27.3.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]: raise self.value
May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]: File "/openstack/venvs/nova-27.3.0/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 10208, in _live_migration_operation
May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]: new_xml_str = libvirt_migrate.get_updated_guest_xml(
May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]: File "/openstack/venvs/nova-27.3.0/lib/python3.8/site-packages/nova/virt/libvirt/migration.py", line 67, in get_updated_guest_xml
May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]: xml_doc = _update_vif_xml(xml_doc, migrate_data, get_vif_config)
May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]: File "/openstack/venvs/nova-27.3.0/lib/python3.8/site-packages/nova/virt/libvirt/migration.py", line 369, in _update_vif_xml
May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]: migrate_vif = migrate_vif_by_mac[mac_addr.lower()]
May 15 14:03:24 cc-compute12-dx1 nova-compute[855142]: KeyError: 'fa:16:3e:7b:d7:f4'
May 15 14:03:25 cc-compute12-dx1 nova-compute[855142]: 2024-05-15 14:03:25.288 855142 ERROR nova.virt.libvirt.driver [None req-3b021e43-ca4b-4c67-a094-6ce2152daeb6 5882947e723e4afdb15f5e2679f14c0e 4d866b39b4df4ab7a8d84c6a9d05bd3d - - default default] [instance: 9e948ba7-4137-462a-91aa-7e9a64f52a9d] Migration operation has aborted
May 15 14:03:25 cc-compute12-dx1 nova-compute[855142]: 2024-05-15 14:03:25.352 855142 INFO nova.compute.manager [None req-3b021e43-ca4b-4c67-a094-6ce2152daeb6 5882947e723e4afdb15f5e2679f14c0e 4d866b39b4df4ab7a8d84c6a9d05bd3d - - default default] [instance: 9e948ba7-4137-462a-91aa-7e9a64f52a9d] Swapping old allocation on dict_keys(['4e2aaa84-0561-4525-820f-0d2c1b3c7f86']) held by migration 75c70d12-41c8-4c67-994c-679e59787690 for instance
May 15 14:03:40 cc-compute12-dx1 nova-compute[855142]: 2024-05-15 14:03:40.929 855142 INFO nova.compute.resource_tracker [None req-017bb1d7-a8d5-4a2f-8df6-26280fa4cb89 - - - - - -] Instance 8eee2b1f-01cd-4b9e-b178-5eb9600cbb5f has allocations against this compute host but is not found in the database.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/919760

[sean mooney (sean-k-mooney)]

i honestly think this is a neuton bug and should be fixed by normalising the mac in neutron when the user provides one to be lowercase with : instead of -

this is the second time that someone has proposed a patch to modify nova to workaround this input normaliasiation bug in neutron so i would at least like to get input form the neutron folks as to why they are not normalising this in there api.

[Dmitriy Rabotyagov (noonedeadpunk)]

Well, neutron leverages netaddr.EUI to verify validity of Mac addresses, which absolutely legitimately allows to have EUI-48 to be separated with dashes instead of colons.

So reading a bit more on that, I am really not convinced that it is precisely a neutron bug.
It's a question of what nova should expect to receive from neutron and from libvirt and align on format.

As I think both delimiters are valid from viewpoint of EUI format.

[Brian Haley (brian-haley)]

I can't say I've ever seen this type of issue before, and never had anyone ask or propose for it, and as mentioned neutron just allows anything netaddr.EUI which both : and - are.

The other side of the discussion that I've experienced before is: "I made and API call and the returned data wasn't what I passed" (happened when I normalized ipv6-icmp). So I'm on the fence now about making such a change after almost 15 years. I'd have to float it at our next meeting to see what others think.

I'm tempted to just document around it - if both Nova and Neutron docs (or even the openstackclient help message) mentions that : should be used to avoid potential issues, then people that have written tooling to do things like this will have to change it.

[Dmitriy Rabotyagov (noonedeadpunk)]

So frankly, I think this is actually should be fixed in nova specifically.

So if I just open Wikipedia, it would say:
"The standard (IEEE 802) format for printing EUI-48 addresses in human-friendly form is six groups of two hexadecimal digits, separated by hyphens (-) in transmission order (e.g. 01-23-45-67-89-AB)." with reference to the doc:
https://standards.ieee.org/content/dam/ieee-standards/standards/web/documents/tutorials/eui.pdf

So it kinda weird even to document recommendation for usage of semicolon separated Mac address.

Then, from what it looks like, nova successfully pass hyphen separated Mac to libvirt, and then it libvirt that concerts/normalizes Mac to it's taste. Correct me if I'm wrong, please.

So then, it is even a driver specific in nova, as other drivers might be fine with having hyphen in mac.

Which leads me to the way, that it really has nothing to do with neutron more or less and a very nova-specific issue in the wiring of what libvirt wants to what users supply with.