Issue booting instance with normal port and macvtap agent

[James Denton]

OpenStack Release: Newton
Operating System: Ubuntu 16.04 LTS 4.4.0-45-generic
OpenStack Distro: OpenStack-Ansible 14.0.2

While working to test/implement macvtap functionality, I found it was not possible to boot an instance when using the macvtap mech driver and macvtap agent. My goal is/was to boot an instance with a 'normal' (non-PCI) port, and have a macvtap interface created on the compute node. What I found was that that booting the instance resulted in an ERROR state, with errors reported in neutron-server.log[1] and nova-compute.log[2]:

[1] Neutron Server: http://paste.openstack.org/show/596128/
[2] Nova Compute: http://paste.openstack.org/show/596129/

The Nova boot syntax can be seen here:

nova boot --image 'Cirros Test Image' --flavor 'm1.test' --nic port-id=5f08dcec-6689-4c1c-9f95-ffb69548c606 --availability-zone=ZONE-B TEST-1

An availability zone consisting of a single node was specified.

The interesting thing was observed (multiple times) in neutron-server.log during the boot process:

2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers [req-f531783a-8f29-423e-9764-026f4cd4693f 3e202337454b4813b0ce6ecff74d43d6 b009912cbdcd45a0b4ef34fb3d22e7e1 - - -] Mechanism driver macvtap failed in bind_port
2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers Traceback (most recent call last):
2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers File "/openstack/venvs/neutron-14.0.2/lib/python2.7/site-packages/neutron/plugins/ml2/managers.py", line 787, in _bind_port_level
2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers driver.obj.bind_port(context)
2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers File "/openstack/venvs/neutron-14.0.2/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/mech_agent.py", line 109, in bind_port
2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers agent):
2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers File "/openstack/venvs/neutron-14.0.2/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/macvtap/mech_driver/mech_macvtap.py", line 106, in try_to_bind_segment_for_agent
2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers if self._is_live_migration(context):
2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers File "/openstack/venvs/neutron-14.0.2/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/macvtap/mech_driver/mech_macvtap.py", line 68, in _is_live_migration
2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers port_profile = context.original.get(portbindings.PROFILE)
2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers AttributeError: 'NoneType' object has no attribute 'get'
2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers

The _is_live_migration function was implemented in [3]:

[3] https://review.openstack.org/#/c/361301/5/neutron/plugins/ml2/drivers/macvtap/mech_driver/mech_macvtap.py

context.original is set to None, which is resulting in the Traceback observed here. I have logged the object data at the time it hits the function in [4]:

[4] Object: http://paste.openstack.org/show/596130/

To work around this issue for the sake of booting the instance, I set 'port_profile' to None in mech_macvtap.py and restarted Neutron server. I was then able to boot the instance, and the agent created the VLAN and macvtap interfaces accordingly:

+--------------------------------------+------------------------------------+--------+------------+-------------+------------------------------------------------------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------------------------------------+--------+------------+-------------+------------------------------------------------------------------------------+
| 0022df55-6500-44c0-8ff5-4433c37157c1 | TEST-1 | ERROR | - | NOSTATE | |
| 69eda7de-00ad-4927-955f-9e5e1d9d6909 | TEST-1 | ACTIVE | - | Running | TEST=192.168.7.8 |

113: em50.62@em50: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 5c:b9:01:88:fd:a5 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5eb9:1ff:fe88:fda5/64 scope link
       valid_lft forever preferred_lft forever
115: <email address hidden>: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 500
    link/ether fa:16:3e:67:da:b8 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::f816:3eff:fe67:dab8/64 scope link
       valid_lft forever preferred_lft forever

root@817769-i812846:~# tcpdump -i macvtap0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on macvtap0, link-type EN10MB (Ethernet), capture size 262144 bytes
15:30:52.524485 IP6 :: > ff02::1:ff67:dab8: ICMP6, neighbor solicitation, who has fe80::f816:3eff:fe67:dab8, length 24
15:30:52.994199 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:67:da:b8 (oui Unknown), length 290
15:30:52.998185 IP6 :: > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
15:30:53.390218 IP6 :: > ff02::1:ff67:dab8: ICMP6, neighbor solicitation, who has fe80::f816:3eff:fe67:dab8, length 24
15:30:53.524385 IP6 fe80::f816:3eff:fe67:dab8 > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
15:30:53.524420 IP6 fe80::f816:3eff:fe67:dab8 > ip6-allrouters: ICMP6, router solicitation, length 16
15:30:54.008276 IP6 fe80::f816:3eff:fe67:dab8 > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
15:30:54.390184 IP6 fe80::f816:3eff:fe67:dab8 > ip6-allrouters: ICMP6, router solicitation, length 16
15:30:57.528300 IP6 fe80::f816:3eff:fe67:dab8 > ip6-allrouters: ICMP6, router solicitation, length 16
15:30:58.394100 IP6 fe80::f816:3eff:fe67:dab8 > ip6-allrouters: ICMP6, router solicitation, length 16
15:31:01.536286 IP6 fe80::f816:3eff:fe67:dab8 > ip6-allrouters: ICMP6, router solicitation, length 16
15:31:02.042017 IP6 fe80::f816:3eff:fe67:dab8 > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
15:31:02.402014 IP6 fe80::f816:3eff:fe67:dab8 > ip6-allrouters: ICMP6, router solicitation, length 16

Port details can be seen here:

+-----------------------+-------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-----------------------+-------------------------------------------------------------------------------------------------------------------------+
| admin_state_up | True |
| allowed_address_pairs | |
| binding:host_id | 817769-i812846.NewtonTest.com |
| binding:profile | {} |
| binding:vif_details | {"port_filter": false, "macvtap_mode": "bridge", "vlan": 62, "physical_interface": "em50", "macvtap_source": "em50.62"} |
| binding:vif_type | macvtap |
| binding:vnic_type | normal |
| created_at | 2017-01-23T20:18:35Z |
| description | |
| device_id | 69eda7de-00ad-4927-955f-9e5e1d9d6909 |
| device_owner | compute:ZONE-B |
| extra_dhcp_opts | |
| fixed_ips | {"subnet_id": "898171fa-903e-4549-bb82-fbb5f16c9bb1", "ip_address": "192.168.7.8"} |
| id | 5f08dcec-6689-4c1c-9f95-ffb69548c606 |
| mac_address | fa:16:3e:67:da:b8 |
| name | TEST |
| network_id | 508e4e86-0137-4774-8df5-bd6919080bea |
| port_security_enabled | True |
| project_id | a2bc030466504dea81191913f1071a10 |
| revision_number | 23 |
| security_groups | |
| status | ACTIVE |
| tenant_id | a2bc030466504dea81191913f1071a10 |
| updated_at | 2017-01-23T20:29:07Z |
+-----------------------+-------------------------------------------------------------------------------------------------------------------------+

Please let me know what additional information you need, including further troubleshooting or debug output.

James

[Luke Yildirim (luke.yildirim)]

+1

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/435144

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/435144
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=3dddfa56db69b504408c6894a814a89ea63b05f0
Submitter: Jenkins
Branch: master

commit 3dddfa56db69b504408c6894a814a89ea63b05f0
Author: Kevin Benton <email address hidden>
Date: Thu Feb 16 14:53:31 2017 -0800

    Macvtap: Check for no original port in is_live_migration

    Change 6865f4d9f22d5daa2f07ff9651c2280aed489c8c mistakenly
    assumed that None would not be present for the 'original'
    property on a PortContext. However, this is the default value
    for the original field in PortContext, which is what is used
    in the construction as part of the _create_port_db process
    in ML2.

    This resulted in binding failures for the macvtap mech drivers
    due to an attribute error in cases like brand new ports.

    This patch simply checks for None before trying to determine
    if it's a live migration (which it isn't in the case of port creation).

    Part of the issue is likely that the FakePortContext in the unit tests
    was defaulting to an empty dict which is the not the same behavior as
    the real PortContext.

    Change-Id: I6659235a70aa4528fd21911c04e651194591e449
    Closes-Bug: #1658802

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 11.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/470216

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/470217

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/470217
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=fe3a2139a872f432a5c4a509bc403ea5b6ac429e
Submitter: Jenkins
Branch: stable/newton

commit fe3a2139a872f432a5c4a509bc403ea5b6ac429e
Author: Kevin Benton <email address hidden>
Date: Thu Feb 16 14:53:31 2017 -0800

    Macvtap: Check for no original port in is_live_migration

    Change 6865f4d9f22d5daa2f07ff9651c2280aed489c8c mistakenly
    assumed that None would not be present for the 'original'
    property on a PortContext. However, this is the default value
    for the original field in PortContext, which is what is used
    in the construction as part of the _create_port_db process
    in ML2.

    This resulted in binding failures for the macvtap mech drivers
    due to an attribute error in cases like brand new ports.

    This patch simply checks for None before trying to determine
    if it's a live migration (which it isn't in the case of port creation).

    Part of the issue is likely that the FakePortContext in the unit tests
    was defaulting to an empty dict which is the not the same behavior as
    the real PortContext.

    Change-Id: I6659235a70aa4528fd21911c04e651194591e449
    Closes-Bug: #1658802
    (cherry picked from commit 3dddfa56db69b504408c6894a814a89ea63b05f0)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/470216
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=24fb21623eac04cf8abf8c04e0febb4c52e57540
Submitter: Jenkins
Branch: stable/ocata

commit 24fb21623eac04cf8abf8c04e0febb4c52e57540
Author: Kevin Benton <email address hidden>
Date: Thu Feb 16 14:53:31 2017 -0800

    Macvtap: Check for no original port in is_live_migration

    Change 6865f4d9f22d5daa2f07ff9651c2280aed489c8c mistakenly
    assumed that None would not be present for the 'original'
    property on a PortContext. However, this is the default value
    for the original field in PortContext, which is what is used
    in the construction as part of the _create_port_db process
    in ML2.

    This resulted in binding failures for the macvtap mech drivers
    due to an attribute error in cases like brand new ports.

    This patch simply checks for None before trying to determine
    if it's a live migration (which it isn't in the case of port creation).

    Part of the issue is likely that the FakePortContext in the unit tests
    was defaulting to an empty dict which is the not the same behavior as
    the real PortContext.

    Change-Id: I6659235a70aa4528fd21911c04e651194591e449
    Closes-Bug: #1658802
    (cherry picked from commit 3dddfa56db69b504408c6894a814a89ea63b05f0)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 9.4.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 10.0.3 release.