OpenStack Compute (nova)

object of type 'NoneType' has no len() from neutronclient in get_instance_nwinfo if neutron credentials are missing

Bug #1272600 reported by Clint Byrum
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Invalid
High
Unassigned
tripleo
Fix Released
Critical
Unassigned

Bug Description

To reproduce:

1. boot an instance
2. rebuild instance with a new image id
3. immediately restart nova-compute

-- instance is now stuck in REBUILD/rebuild_spawning state.

4. nova delete instance

-- instanec now says it is ACTIVE, but it cannot be deleted.

This traceback appears in logs:

2014-01-25 01:35:22,096.096 27134 ERROR nova.openstack.common.rpc.amqp [req-9f5bbb1e-4cab-4f60-9f28-57366b519533 2de96ce5da994575a08447c93bcafd53 4956c533154c476799c688eda7ed65ab] Exception during message handling
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_data
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp **args)
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/exception.py", line 90, in wrapped
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp payload)
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp six.reraise(self.type_, self.value, self.tb)
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/exception.py", line 73, in wrapped
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp return f(self, context, *args, **kw)
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 244, in decorated_function
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp pass
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp six.reraise(self.type_, self.value, self.tb)
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 230, in decorated_function
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 295, in decorated_function
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp function(self, context, *args, **kwargs)
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 272, in decorated_function
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp e, sys.exc_info())
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp six.reraise(self.type_, self.value, self.tb)
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 259, in decorated_function
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 2024, in terminate_instance
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp do_terminate_instance(instance, bdms)
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/openstack/common/lockutils.py", line 249, in inner
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp return f(*args, **kwargs)
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 2016, in do_terminate_instance
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp reservations=reservations)
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/hooks.py", line 105, in inner
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp rv = f(*args, **kwargs)
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 1994, in _delete_instance
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp user_id=user_id)
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp six.reraise(self.type_, self.value, self.tb)
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 1964, in _delete_instance
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp self._shutdown_instance(context, db_inst, bdms)
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 1874, in _shutdown_instance
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp network_info = self._get_instance_nw_info(context, instance)
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 891, in _get_instance_nw_info
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp instance)
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/network/api.py", line 50, in wrapper
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp res = f(self, context, *args, **kwargs)
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 439, in get_instance_nw_info
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp result = self._get_instance_nw_info(context, instance, networks)
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 446, in _get_instance_nw_info
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp nw_info = self._build_network_info_model(context, instance, networks)
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 998, in _build_network_info_model
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp data = client.list_ports(**search_opts)
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 108, in with_params
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp ret = self.function(instance, *args, **kwargs)
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 298, in list_ports
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp **_params)
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1197, in list
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp for r in self._pagination(collection, path, **params):
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1210, in _pagination
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp res = self.get(path, params=params)
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1183, in get
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp headers=headers, params=params)
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1168, in retry_request
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp headers=headers, params=params)
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1097, in do_request
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp self.httpclient.authenticate_and_fetch_endpoint_url()
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/neutronclient/client.py", line 172, in authenticate_and_fetch_endpoint_url
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp self.authenticate()
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/neutronclient/client.py", line 237, in authenticate
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp raise exceptions.Unauthorized(message=body)
2014-01-25 01:35:22,096.096 27134 TRACE nova.openstack.common.rpc.amqp Unauthorized: {"error": {"message": "object of type 'NoneType' has no len()", "code": 400, "title": "Bad Request"}}

Note that this was seen with a recent pull of master as of today, with the nova baremetal driver.

Tags: baremetal
Revision history for this message
Robert Collins (lifeless) wrote : Re: object of type 'NoneType' has no len() from neutronclient in get_instance_nwinfo

(Tripleo undercloud is down until we address this)

summary: - Deleting an instance that is in the REBUILD/rebuild_spawning state
- results in an undeletable instance
+ object of type 'NoneType' has no len() from neutronclient in
+ get_instance_nwinfo
Changed in tripleo:
status: New → Triaged
importance: Undecided → Critical
Revision history for this message
Robert Collins (lifeless) wrote :

The odd thing here is that it's *allocating* nwinfo on delete. I think we should add a way to hint that we don't want that - an instance that never got nwinfo shouldn't allocate nwinfo on delete only to delete it again.

Revision history for this message
Robert Collins (lifeless) wrote :

nvm, reading it wrong - it's not allocating.

Revision history for this message
Robert Collins (lifeless) wrote :

And I've confirmed - this happens on boot too.

Revision history for this message
Robert Collins (lifeless) wrote :

Neutron is trying to authenticate as follows:

{'auth': {'tenantName': None, 'passwordCredentials': {'username': None, 'password': None}}}

That seems guaranteed to fail.

Revision history for this message
Robert Collins (lifeless) wrote :

Ah - we've deleted the havana aliases. :(

commit ee50436938eb8122185b08681803d07f16dcd3c1

Changed in tripleo:
status: Triaged → Fix Released
Changed in nova:
status: New → Invalid
status: Invalid → Triaged
importance: Undecided → High
summary: object of type 'NoneType' has no len() from neutronclient in
- get_instance_nwinfo
+ get_instance_nwinfo if neutron credentials are missing
Revision history for this message
Robert Collins (lifeless) wrote :

So this isn't something we should need to diagnose: it should not even try to connect to Neutron without correct credentials.

Joe Gordon (jogo)
tags: added: baremetal
Revision history for this message
Kashyap Kopparam (kashyapk) wrote :

I face the same issue on a Havana setup. Can't delete instances or boot them.

Revision history for this message
Davanum Srinivas (DIMS) (dims-v) wrote :

is there something more we need to do on the Nova side?

Changed in nova:
status: Triaged → Incomplete
Sean Dague (sdague)
Changed in nova:
status: Incomplete → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.