| 2019-01-12 16:59:56 |
Brett Osteen |
description |
Description:
============
Booting a guest with a neutron port of type 'direct-physical' will cause nova to allocate a PCI passthrough device for the port. The MAC address of the PCI passthrough device in the guest is not a virtual MAC address (fa:16:...) but the MAC address of the physical device since the full device is allocated to the guest (compared to SR-IOV where a virtual MAC address is arbitrarily chosen for the port).
When resizing the guest (to another flavor), nova will allocate a new PCI device for the guest. After the resize, the guest will be bound to another PCI device which has a different MAC address. However the MAC address on the neutron port is not updated, causing DHCP to not work because the MAC address is unknown.
The same issue can be observed when migrating a guest to another host.
Steps to reproduce:
===================
1- Configure a compute with two NICs PCI passthrough devices
2- Create a neutron port of type 'direct-physical':
PORTID=`neutron port-create $NETID --binding:vnic_type direct-physical | grep "\ id\ " | awk '{ print $4 }'`
3- Boot a guest with the port-id:
nova boot guest --image=ubuntu --nic port-id=$PORTID --flavor=m1.medium
4- Note the MAC address of the neutron port:
[centos@IronPass-2 devstack]$ neutron port-show 657990ce-3446-40a2-bc33-8040a32cb72b
+-----------------------+-----------------------------------------------------------------------------------------------------------+
| Field | Value |
+-----------------------+-----------------------------------------------------------------------------------------------------------+
| admin_state_up | True |
| allowed_address_pairs | |
| binding:vnic_type | direct-physical |
| created_at | 2016-08-26T14:57:07 |
| description | |
| device_id | f1005ec2-1875-4345-af01-58afaee7e68d |
| device_owner | compute:None |
| extra_dhcp_opts | |
| fixed_ips | {"subnet_id": "8e6aaf8e-78e8-42d7-9b59-f96789728abc", "ip_address": "10.0.0.12"} |
| | {"subnet_id": "4210fe1c-b156-4ecb-a896-922c87f85c3f", "ip_address": "fdab:2588:f4:0:f816:3eff:fe37:acba"} |
| id | 657990ce-3446-40a2-bc33-8040a32cb72b |
| mac_address | 90:e2:ba:48:27:ed |
| name | sriov_port |
| network_id | dfdff739-a27b-4160-9e40-c4824e8a351d |
| port_security_enabled | True |
| revision | 57 |
| security_groups | 8c7b8173-f050-480d-968a-33e0b75332fc |
| status | ACTIVE |
| tenant_id | a11cb2aece1943c2a86ee0a55e1bd8f7 |
| updated_at | 2016-08-26T17:33:12 |
+-----------------------+-----------------------------------------------------------------------------------------------------------+
5- Log in the guest and get a DHCP address for the interface.
ip link add link eth1 name eth1.451 type vlan id 451
ip link set eth1.451 up
dhclient eth1.451
Note the MAC address of the interface:
4: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 90:e2:ba:48:27:ed brd ff:ff:ff:ff:ff:ff
inet6 fe80::92e2:baff:fe48:27ed/64 scope link
valid_lft forever preferred_lft forever
5: eth1.451@eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
link/ether 90:e2:ba:48:27:ed brd ff:ff:ff:ff:ff:ff
inet 10.0.0.12/24 brd 10.0.0.255 scope global eth1.451
valid_lft forever preferred_lft forever
inet6 fdab:2588:f4:0:92e2:baff:fe48:27ed/64 scope global dynamic
valid_lft 86396sec preferred_lft 14396sec
inet6 fe80::92e2:baff:fe48:27ed/64 scope link
valid_lft forever preferred_lft forever
6- Resize the guest
7- Note the MAC address of the neutron port (after resize):
[centos@IronPass-2 devstack]$ neutron port-show 657990ce-3446-40a2-bc33-8040a32cb72b
+-----------------------+-----------------------------------------------------------------------------------------------------------+
| Field | Value |
+-----------------------+-----------------------------------------------------------------------------------------------------------+
| admin_state_up | True |
| allowed_address_pairs | |
| binding:vnic_type | direct-physical |
| created_at | 2016-08-26T14:57:07 |
| description | |
| device_id | f1005ec2-1875-4345-af01-58afaee7e68d |
| device_owner | compute:None |
| extra_dhcp_opts | |
| fixed_ips | {"subnet_id": "8e6aaf8e-78e8-42d7-9b59-f96789728abc", "ip_address": "10.0.0.12"} |
| | {"subnet_id": "4210fe1c-b156-4ecb-a896-922c87f85c3f", "ip_address": "fdab:2588:f4:0:f816:3eff:fe37:acba"} |
| id | 657990ce-3446-40a2-bc33-8040a32cb72b |
| mac_address | 90:e2:ba:48:27:ed |
| name | sriov_port |
| network_id | dfdff739-a27b-4160-9e40-c4824e8a351d |
| port_security_enabled | True |
| revision | 59 |
| security_groups | 8c7b8173-f050-480d-968a-33e0b75332fc |
| status | ACTIVE |
| tenant_id | a11cb2aece1943c2a86ee0a55e1bd8f7 |
| updated_at | 2016-08-26T17:38:30 |
+-----------------------+-----------------------------------------------------------------------------------------------------------+
Notice that it's still hasn't changed: 90:e2:ba:48:27:ed
8- Log in the guest and get a DHCP address for the interface.
ip link add link eth1 name eth1.451 type vlan id 451
ip link set eth1.451 up
dhclient eth1.451
Note the MAC address of the interface has changed:
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 00:1e:67:51:36:71 brd ff:ff:ff:ff:ff:ff
inet6 fe80::21e:67ff:fe51:3671/64 scope link
valid_lft forever preferred_lft forever
5: eth2.451@eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
link/ether 00:1e:67:51:36:71 brd ff:ff:ff:ff:ff:ff
inet6 fe80::21e:67ff:fe51:3671/64 scope link
valid_lft forever preferred_lft forever
Also note that the DHCP client wasn't successful in getting a DHCP address.
9- From the dnsmasq host file, we can also see that it hasn't been updated:
cat /opt/stack/data/neutron/dhcp/dfdff739-a27b-4160-9e40-c4824e8a351d/host
fa:16:3e:e5:e0:c9,host-10-0-0-2.openstacklocal,10.0.0.2
90:e2:ba:48:27:ed,host-10-0-0-12.openstacklocal,10.0.0.12
fa:16:3e:99:c8:b9,host-10-0-0-5.openstacklocal,10.0.0.5 |
Description:
============
Booting a guest with a neutron port of type 'direct-physical' will cause nova to allocate a PCI passthrough device for the port. The MAC address of the PCI passthrough device in the guest is not a virtual MAC address (fa:16:...) but the MAC address of the physical device since the full device is allocated to the guest (compared to SR-IOV where a virtual MAC address is arbitrarily chosen for the port).
When resizing the guest (to another flavor), nova will allocate a new PCI device for the guest. After the resize, the guest will be bound to another PCI device which has a different MAC address. However the MAC address on the neutron port is not updated, causing DHCP to not work because the MAC address is unknown.
The same issue can be observed when migrating a guest to another host.
Steps to reproduce:
===================
1- Configure a compute with two NICs PCI passthrough devices
2- Create a neutron port of type 'direct-physical':
PORTID=`neutron port-create $NETID --binding:vnic_type direct-physical | grep "\ id\ " | awk '{ print $4 }'`
3- Boot a guest with the port-id:
nova boot guest --image=ubuntu --nic port-id=$PORTID --flavor=m1.medium
4- Note the MAC address of the neutron port:
[centos@IronPass-2 devstack]$ neutron port-show 657990ce-3446-40a2-bc33-8040a32cb72b
+-----------------------+-----------------------------------------------------------------------------------------------------------+
| Field | Value |
+-----------------------+-----------------------------------------------------------------------------------------------------------+
| admin_state_up | True |
| allowed_address_pairs | |
| binding:vnic_type | direct-physical |
| created_at | 2016-08-26T14:57:07 |
| description | |
| device_id | f1005ec2-1875-4345-af01-58afaee7e68d |
| device_owner | compute:None |
| extra_dhcp_opts | |
| fixed_ips | {"subnet_id": "8e6aaf8e-78e8-42d7-9b59-f96789728abc", "ip_address": "10.0.0.12"} |
| | {"subnet_id": "4210fe1c-b156-4ecb-a896-922c87f85c3f", "ip_address": "fdab:2588:f4:0:f816:3eff:fe37:acba"} |
| id | 657990ce-3446-40a2-bc33-8040a32cb72b |
| mac_address | 90:e2:ba:48:27:ed |
| name | sriov_port |
| network_id | dfdff739-a27b-4160-9e40-c4824e8a351d |
| port_security_enabled | True |
| revision | 57 |
| security_groups | 8c7b8173-f050-480d-968a-33e0b75332fc |
| status | ACTIVE |
| tenant_id | a11cb2aece1943c2a86ee0a55e1bd8f7 |
| updated_at | 2016-08-26T17:33:12 |
+-----------------------+-----------------------------------------------------------------------------------------------------------+
5- Log in the guest and get a DHCP address for the interface.
ip link add link eth1 name eth1.451 type vlan id 451
ip link set eth1.451 up
dhclient eth1.451
Note the MAC address of the interface:
4: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 90:e2:ba:48:27:ed brd ff:ff:ff:ff:ff:ff
inet6 fe80::92e2:baff:fe48:27ed/64 scope link
valid_lft forever preferred_lft forever
5: eth1.451@eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
link/ether 90:e2:ba:48:27:ed brd ff:ff:ff:ff:ff:ff
inet 10.0.0.12/24 brd 10.0.0.255 scope global eth1.451
valid_lft forever preferred_lft forever
inet6 fdab:2588:f4:0:92e2:baff:fe48:27ed/64 scope global dynamic
valid_lft 86396sec preferred_lft 14396sec
inet6 fe80::92e2:baff:fe48:27ed/64 scope link
valid_lft forever preferred_lft forever
6- Resize the guest
7- Note the MAC address of the neutron port (after resize):
[centos@IronPass-2 devstack]$ neutron port-show 657990ce-3446-40a2-bc33-8040a32cb72b
+-----------------------+-----------------------------------------------------------------------------------------------------------+
| Field | Value |
+-----------------------+-----------------------------------------------------------------------------------------------------------+
| admin_state_up | True |
| allowed_address_pairs | |
| binding:vnic_type | direct-physical |
| created_at | 2016-08-26T14:57:07 |
| description | |
| device_id | f1005ec2-1875-4345-af01-58afaee7e68d |
| device_owner | compute:None |
| extra_dhcp_opts | |
| fixed_ips | {"subnet_id": "8e6aaf8e-78e8-42d7-9b59-f96789728abc", "ip_address": "10.0.0.12"} |
| | {"subnet_id": "4210fe1c-b156-4ecb-a896-922c87f85c3f", "ip_address": "fdab:2588:f4:0:f816:3eff:fe37:acba"} |
| id | 657990ce-3446-40a2-bc33-8040a32cb72b |
| mac_address | 90:e2:ba:48:27:ed |
| name | sriov_port |
| network_id | dfdff739-a27b-4160-9e40-c4824e8a351d |
| port_security_enabled | True |
| revision | 59 |
| security_groups | 8c7b8173-f050-480d-968a-33e0b75332fc |
| status | ACTIVE |
| tenant_id | a11cb2aece1943c2a86ee0a55e1bd8f7 |
| updated_at | 2016-08-26T17:38:30 |
+-----------------------+-----------------------------------------------------------------------------------------------------------+
Notice that it's still hasn't changed: 90:e2:ba:48:27:ed
8- Log in the guest and get a DHCP address for the interface.
ip link add link eth1 name eth1.451 type vlan id 451
ip link set eth1.451 up
dhclient eth1.451
Note the MAC address of the interface has changed:
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 00:1e:67:51:36:71 brd ff:ff:ff:ff:ff:ff
inet6 fe80::21e:67ff:fe51:3671/64 scope link
valid_lft forever preferred_lft forever
5: eth2.451@eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
link/ether 00:1e:67:51:36:71 brd ff:ff:ff:ff:ff:ff
inet6 fe80::21e:67ff:fe51:3671/64 scope link
valid_lft forever preferred_lft forever
Also note that the DHCP client wasn't successful in getting a DHCP address.
9- From the dnsmasq host file, we can also see that it hasn't been updated:
cat /opt/stack/data/neutron/dhcp/dfdff739-a27b-4160-9e40-c4824e8a351d/host
fa:16:3e:e5:e0:c9,host-10-0-0-2.openstacklocal,10.0.0.2
90:e2:ba:48:27:ed,host-10-0-0-12.openstacklocal,10.0.0.12
fa:16:3e:99:c8:b9,host-10-0-0-5.openstacklocal,10.0.0.5
See also https://bugs.launchpad.net/nova/+bug/1617429 |
|
