OpenStack Compute (nova)

TypeError when sending notification during attach_interface

  1. Series ocata
  2. Bug #1737201
Bug #1737201 reported by Matt Riedemann
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Balazs Gibizer
Ocata
Fix Committed
Undecided
Unassigned
Pike
Fix Committed
Undecided
Unassigned

Bug Description

http://logs.openstack.org/50/524750/1/check/legacy-tempest-dsvm-neutron-full/eb8d805/logs/screen-n-api.txt.gz?level=TRACE#_Dec_04_13_34_20_635874

Dec 04 13:34:20.635874 ubuntu-xenial-inap-mtl01-0001196572 <email address hidden>[6808]: ERROR nova.api.openstack.extensions [None req-2d1b063f-1324-4498-af68-ce48c6d8e5a3 tempest-AttachInterfacesTestJSON-149718191 tempest-AttachInterfacesTestJSON-149718191] Unexpected exception in API method: TypeError: 'NoneType' object has no attribute '__getitem__'
Dec 04 13:34:20.636066 ubuntu-xenial-inap-mtl01-0001196572 <email address hidden>[6808]: Traceback (most recent call last):
Dec 04 13:34:20.636202 ubuntu-xenial-inap-mtl01-0001196572 <email address hidden>[6808]: File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
Dec 04 13:34:20.636336 ubuntu-xenial-inap-mtl01-0001196572 <email address hidden>[6808]: res = self.dispatcher.dispatch(message)
Dec 04 13:34:20.636474 ubuntu-xenial-inap-mtl01-0001196572 <email address hidden>[6808]: File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
Dec 04 13:34:20.636614 ubuntu-xenial-inap-mtl01-0001196572 <email address hidden>[6808]: return self._do_dispatch(endpoint, method, ctxt, args)
Dec 04 13:34:20.636745 ubuntu-xenial-inap-mtl01-0001196572 <email address hidden>[6808]: File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
Dec 04 13:34:20.636892 ubuntu-xenial-inap-mtl01-0001196572 <email address hidden>[6808]: result = func(ctxt, **new_args)
Dec 04 13:34:20.637049 ubuntu-xenial-inap-mtl01-0001196572 <email address hidden>[6808]: File "/opt/stack/new/nova/nova/exception_wrapper.py", line 76, in wrapped
Dec 04 13:34:20.637187 ubuntu-xenial-inap-mtl01-0001196572 <email address hidden>[6808]: function_name, call_dict, binary)
Dec 04 13:34:20.637317 ubuntu-xenial-inap-mtl01-0001196572 <email address hidden>[6808]: File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Dec 04 13:34:20.637442 ubuntu-xenial-inap-mtl01-0001196572 <email address hidden>[6808]: self.force_reraise()
Dec 04 13:34:20.637607 ubuntu-xenial-inap-mtl01-0001196572 <email address hidden>[6808]: File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Dec 04 13:34:20.637761 ubuntu-xenial-inap-mtl01-0001196572 <email address hidden>[6808]: six.reraise(self.type_, self.value, self.tb)
Dec 04 13:34:20.637895 ubuntu-xenial-inap-mtl01-0001196572 <email address hidden>[6808]: File "/opt/stack/new/nova/nova/exception_wrapper.py", line 67, in wrapped
Dec 04 13:34:20.638044 ubuntu-xenial-inap-mtl01-0001196572 <email address hidden>[6808]: return f(self, context, *args, **kw)
Dec 04 13:34:20.638183 ubuntu-xenial-inap-mtl01-0001196572 <email address hidden>[6808]: File "/opt/stack/new/nova/nova/compute/utils.py", line 930, in decorated_function
Dec 04 13:34:20.638306 ubuntu-xenial-inap-mtl01-0001196572 <email address hidden>[6808]: return function(self, context, *args, **kwargs)
Dec 04 13:34:20.638433 ubuntu-xenial-inap-mtl01-0001196572 <email address hidden>[6808]: File "/opt/stack/new/nova/nova/compute/manager.py", line 215, in decorated_function
Dec 04 13:34:20.638566 ubuntu-xenial-inap-mtl01-0001196572 <email address hidden>[6808]: kwargs['instance'], e, sys.exc_info())
Dec 04 13:34:20.638696 ubuntu-xenial-inap-mtl01-0001196572 <email address hidden>[6808]: File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Dec 04 13:34:20.638820 ubuntu-xenial-inap-mtl01-0001196572 <email address hidden>[6808]: self.force_reraise()
Dec 04 13:34:20.638953 ubuntu-xenial-inap-mtl01-0001196572 <email address hidden>[6808]: File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Dec 04 13:34:20.639076 ubuntu-xenial-inap-mtl01-0001196572 <email address hidden>[6808]: six.reraise(self.type_, self.value, self.tb)
Dec 04 13:34:20.639200 ubuntu-xenial-inap-mtl01-0001196572 <email address hidden>[6808]: File "/opt/stack/new/nova/nova/compute/manager.py", line 203, in decorated_function
Dec 04 13:34:20.639325 ubuntu-xenial-inap-mtl01-0001196572 <email address hidden>[6808]: return function(self, context, *args, **kwargs)
Dec 04 13:34:20.639462 ubuntu-xenial-inap-mtl01-0001196572 <email address hidden>[6808]: File "/opt/stack/new/nova/nova/compute/manager.py", line 5563, in attach_interface
Dec 04 13:34:20.639607 ubuntu-xenial-inap-mtl01-0001196572 <email address hidden>[6808]: phase=fields.NotificationPhase.END)
Dec 04 13:34:20.639735 ubuntu-xenial-inap-mtl01-0001196572 <email address hidden>[6808]: File "/opt/stack/new/nova/nova/rpc.py", line 225, in wrapped
Dec 04 13:34:20.639856 ubuntu-xenial-inap-mtl01-0001196572 <email address hidden>[6808]: return f(*args, **kwargs)
Dec 04 13:34:20.639991 ubuntu-xenial-inap-mtl01-0001196572 <email address hidden>[6808]: File "/opt/stack/new/nova/nova/compute/utils.py", line 370, in notify_about_instance_action
Dec 04 13:34:20.640566 ubuntu-xenial-inap-mtl01-0001196572 <email address hidden>[6808]: bdms=bdms)
Dec 04 13:34:20.640746 ubuntu-xenial-inap-mtl01-0001196572 <email address hidden>[6808]: File "/opt/stack/new/nova/nova/notifications/objects/instance.py", line 140, in __init__
Dec 04 13:34:20.640884 ubuntu-xenial-inap-mtl01-0001196572 <email address hidden>[6808]: bdms=bdms)
Dec 04 13:34:20.641010 ubuntu-xenial-inap-mtl01-0001196572 <email address hidden>[6808]: File "/opt/stack/new/nova/nova/notifications/objects/instance.py", line 114, in __init__
Dec 04 13:34:20.641135 ubuntu-xenial-inap-mtl01-0001196572 <email address hidden>[6808]: self.ip_addresses = IpPayload.from_network_info(network_info)
Dec 04 13:34:20.641272 ubuntu-xenial-inap-mtl01-0001196572 <email address hidden>[6808]: File "/opt/stack/new/nova/nova/notifications/objects/instance.py", line 280, in from_network_info
Dec 04 13:34:20.641413 ubuntu-xenial-inap-mtl01-0001196572 <email address hidden>[6808]: for ip in vif.fixed_ips():
Dec 04 13:34:20.641559 ubuntu-xenial-inap-mtl01-0001196572 <email address hidden>[6808]: File "/opt/stack/new/nova/nova/network/model.py", line 415, in fixed_ips
Dec 04 13:34:20.641683 ubuntu-xenial-inap-mtl01-0001196572 <email address hidden>[6808]: return [fixed_ip for subnet in self['network']['subnets']
Dec 04 13:34:20.641824 ubuntu-xenial-inap-mtl01-0001196572 <email address hidden>[6808]: TypeError: 'NoneType' object has no attribute '__getitem__'

The instance info cache is probably not up to date at this point, or we're just being sloppy about checking that fields are set.

Revision history for this message
Matt Riedemann (mriedem) wrote :

For some reason I'm only seeing this in one change right now, so maybe I just got unlucky but does seem like something we need to handle regardless:

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22return%20%5Bfixed_ip%20for%20subnet%20in%20self%5B'network'%5D%5B'subnets'%5D%5C%22%20AND%20tags%3A%5C%22screen-n-api.txt%5C%22&from=7d

Revision history for this message
Matt Riedemann (mriedem) wrote :

The code that's blowing up was added in Newton: https://review.openstack.org/#/c/325923/

Revision history for this message
Matt Riedemann (mriedem) wrote :

I'm not really sure how this can happen. The test creates a server and asserts it has an interface (port) and then it attaches another port but without requesting any specific network or port, it wants nova to allocate a port from the network available to the project, which there should be one otherwise allocate_for_instance would raise an exception and we wouldn't get to the notification code. I wonder if https://review.openstack.org/#/c/403568/ could be causing issues here where we store the info_cache on the instance in memory and maybe that is empty somehow because of this empty request, but as noted, we should be building the info cache from existing networks on the instance if none are requested.

Revision history for this message
Matt Riedemann (mriedem) wrote :

Note that the patch that this failed on is failing for some other weird issues with locked up guestfs and eventlet thread switching, so maybe that is doing something strange here.

Balazs Gibizer (balazs-gibizer)
Changed in nova:
assignee: nobody → Balazs Gibizer (balazs-gibizer)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/527920

Changed in nova:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/527920
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=553f2edde596348ca5447588c5a0b06f3b6be286
Submitter: Zuul
Branch: master

commit 553f2edde596348ca5447588c5a0b06f3b6be286
Author: Balazs Gibizer <email address hidden>
Date: Wed Dec 13 17:14:49 2017 +0100

    Fix possible TypeError in VIF.fixed_ips

    The VIF['network'] field can be initialized to None and therefore
    a later call to VIF.fixed_ips() could raise a TypeError. This problem
    was visible during AttachInterfacesTestJSON tempest test case when
    nova tried to emit instance.interfacae_attach notification.

    This patch checks makes sure that if VIF['network'] is None then
    VIF.fixed_ips() return an empty list instead of raising a TypeError.

    Change-Id: Ib285d874b19be5bc1dbcd1d2af32e461f67e34cb
    Closes-Bug: #1737201

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/531745

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/ocata)

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/531746

Balazs Gibizer (balazs-gibizer)
tags: added: pike-backport-potential
tags: added: ocata-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 17.0.0.0b3

This issue was fixed in the openstack/nova 17.0.0.0b3 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/pike)

Reviewed: https://review.openstack.org/531745
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=4eff70621146d0ddca19b8e22c76d8ee3d043aed
Submitter: Zuul
Branch: stable/pike

commit 4eff70621146d0ddca19b8e22c76d8ee3d043aed
Author: Balazs Gibizer <email address hidden>
Date: Wed Dec 13 17:14:49 2017 +0100

    Fix possible TypeError in VIF.fixed_ips

    The VIF['network'] field can be initialized to None and therefore
    a later call to VIF.fixed_ips() could raise a TypeError. This problem
    was visible during AttachInterfacesTestJSON tempest test case when
    nova tried to emit instance.interfacae_attach notification.

    This patch checks makes sure that if VIF['network'] is None then
    VIF.fixed_ips() return an empty list instead of raising a TypeError.

    Change-Id: Ib285d874b19be5bc1dbcd1d2af32e461f67e34cb
    Closes-Bug: #1737201
    (cherry picked from commit 553f2edde596348ca5447588c5a0b06f3b6be286)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 16.1.1

This issue was fixed in the openstack/nova 16.1.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/ocata)

Reviewed: https://review.openstack.org/531746
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=6a2882bb7bfabe0a694222c1e821fbe08fa26edf
Submitter: Zuul
Branch: stable/ocata

commit 6a2882bb7bfabe0a694222c1e821fbe08fa26edf
Author: Balazs Gibizer <email address hidden>
Date: Wed Dec 13 17:14:49 2017 +0100

    Fix possible TypeError in VIF.fixed_ips

    The VIF['network'] field can be initialized to None and therefore
    a later call to VIF.fixed_ips() could raise a TypeError. This problem
    was visible during AttachInterfacesTestJSON tempest test case when
    nova tried to emit instance.interfacae_attach notification.

    This patch checks makes sure that if VIF['network'] is None then
    VIF.fixed_ips() return an empty list instead of raising a TypeError.

    Change-Id: Ib285d874b19be5bc1dbcd1d2af32e461f67e34cb
    Closes-Bug: #1737201
    (cherry picked from commit 553f2edde596348ca5447588c5a0b06f3b6be286)
    (cherry picked from commit 4eff70621146d0ddca19b8e22c76d8ee3d043aed)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 15.1.1

This issue was fixed in the openstack/nova 15.1.1 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.