hyperv: Unable to get ports details for devices: AttributeError: 'NoneType' object has no attribute 'startswith'

[Matt Riedemann]

In a failed hyperv CI run I'm seeing this in the hyperv agent logs:

http://cloudbase-ci.com/nova/324720/5/Hyper-V_logs/192.168.3.143-compute01/neutron-hyperv-agent.log.gz

2018-04-06 02:43:29.230 588 91983184 MainThread INFO networking_hyperv.neutron.agent.layer2 [-] Hyper-V VM vNIC added: 5d31e08c-957c-45a5-a13d-fa114ea68b56
2018-04-06 02:43:29.230 588 91983184 MainThread INFO networking_hyperv.neutron.agent.layer2 [-] Hyper-V VM vNIC added: None
2018-04-06 02:43:29.246 588 91983184 MainThread INFO networking_hyperv.neutron.agent.layer2 [-] Hyper-V VM vNIC added: None
2018-04-06 02:43:29.246 588 91983184 MainThread INFO networking_hyperv.neutron.agent.layer2 [-] Hyper-V VM vNIC added: None
2018-04-06 02:43:29.262 588 91983184 MainThread INFO networking_hyperv.neutron.agent.layer2 [-] Hyper-V VM vNIC added: None
2018-04-06 02:43:30.496 588 35292864 MainThread DEBUG networking_hyperv.neutron.agent.layer2 [req-1c72ef49-f85f-4776-8219-ac410b3a00e6 - - - - -] Agent loop has new devices! _work c:\openstack\build\networking-hyperv\networking_hyperv\neutron\agent\layer2.py:427
2018-04-06 02:43:30.526 588 35292864 MainThread DEBUG networking_hyperv.neutron.agent.layer2 [req-1c72ef49-f85f-4776-8219-ac410b3a00e6 - - - - -] Unable to get ports details for devices set([u'5d31e08c-957c-45a5-a13d-fa114ea68b56', None]): 'NoneType' object has no attribute 'startswith'
Traceback (most recent call last):

  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
    res = self.dispatcher.dispatch(message)

  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)

  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
    result = func(ctxt, **new_args)

  File "/opt/stack/neutron/neutron/plugins/ml2/rpc.py", line 157, in get_devices_details_list
    for device in kwargs.pop('devices', [])

  File "/opt/stack/neutron/neutron/plugins/ml2/rpc.py", line 80, in get_device_details
    port_id = plugin._device_to_port_id(rpc_context, device)

  File "/opt/stack/neutron/neutron/plugins/ml2/plugin.py", line 1864, in _device_to_port_id
    if device.startswith(prefix):

AttributeError: 'NoneType' object has no attribute 'startswith'
 _treat_devices_added c:\openstack\build\networking-hyperv\networking_hyperv\neutron\agent\layer2.py:360

In this test run, the nova-compute service is also being reported as down, so the nova-scheduler is filtering it out and all server build requests fail. I don't know if the two are related, but thta's how I stumbled onto this error in the hyperv agent logs.

[Lucian Petrut (petrutlucian94)]

It looks like a new unnamed port was added, which is odd as port names are mandatory. I'll add a check. Thanks for reporting this bug.

The problem is that this seems to prevent other ports from being processed.

Most probably the nova service was disabled after too many consecutive failures.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/606066
Committed: https://git.openstack.org/cgit/openstack/os-win/commit/?id=42cfab1081dcb46b7dca898316fb095206b78666
Submitter: Zuul
Branch: master

commit 42cfab1081dcb46b7dca898316fb095206b78666
Author: Lucian Petrut <email address hidden>
Date: Fri Sep 28 16:08:27 2018 +0300

    Ignore events for unnamed VM vNICs

    VM vNIC names are mandatory. Still, we'll getting events for unnamed ports,
    breaking os-win consumers.

    We'll ignore those events, logging a warning.

    Change-Id: I58a408b33a590b9c1ca3d8f32fa3c4d027f43381
    Closes-Bug: #1761748

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/607549

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/607549
Committed: https://git.openstack.org/cgit/openstack/os-win/commit/?id=5ba1e81cd1f82859058178ebb9183c3e9ccf17d8
Submitter: Zuul
Branch: stable/rocky

commit 5ba1e81cd1f82859058178ebb9183c3e9ccf17d8
Author: Lucian Petrut <email address hidden>
Date: Fri Sep 28 16:08:27 2018 +0300

    Ignore events for unnamed VM vNICs

    VM vNIC names are mandatory. Still, we'll getting events for unnamed ports,
    breaking os-win consumers.

    We'll ignore those events, logging a warning.

    Change-Id: I58a408b33a590b9c1ca3d8f32fa3c4d027f43381
    Closes-Bug: #1761748
    (cherry picked from commit 42cfab1081dcb46b7dca898316fb095206b78666)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/607864

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/607864
Committed: https://git.openstack.org/cgit/openstack/os-win/commit/?id=65b06c243b169597d8bcb201c5a8ad5824fcddda
Submitter: Zuul
Branch: stable/queens

commit 65b06c243b169597d8bcb201c5a8ad5824fcddda
Author: Lucian Petrut <email address hidden>
Date: Fri Sep 28 16:08:27 2018 +0300

    Ignore events for unnamed VM vNICs

    VM vNIC names are mandatory. Still, we'll getting events for unnamed ports,
    breaking os-win consumers.

    We'll ignore those events, logging a warning.

    Change-Id: I58a408b33a590b9c1ca3d8f32fa3c4d027f43381
    Closes-Bug: #1761748
    (cherry picked from commit 42cfab1081dcb46b7dca898316fb095206b78666)
    (cherry picked from commit 5ba1e81cd1f82859058178ebb9183c3e9ccf17d8)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/os-win 4.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/os-win queens-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/os-win rocky-eol release.