OpenStack Compute (nova)

Unable to create an instance from remote machine using novaclient

Bug #1321508 reported by John Griffith
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Undecided
Unassigned

Bug Description

When attempting to boot an instance from a remote host using novaclient and API access file downloaded via dashboard I'm unable to create instances due to an error in attempting to retrieve networks. This is reproducible via devstack on both Precise and Trusty, and I've verified that using the same creds file on the actual compute host is succesful.

It looks like this might be related to the following commit:
https://github.com/openstack/nova/commit/869b435dca27e06f4160b781d86bba708475866a

Nova-API Trace:
```````````````````````````````````````````````````````````````````````````

etag: 51bc16b900bf0f814bb6c0c3dd8f0790
x-image-meta-is_public: True
x-image-meta-min_ram: 0
x-image-meta-owner: 8109ccec0fec4106ad3f005fd76130bb
x-image-meta-updated_at: 2014-05-21T00:19:30
content-type: text/html; charset=UTF-8
x-openstack-request-id: req-35660f52-3922-456b-b44c-6d65d8f696fa
x-image-meta-disk_format: qcow2
x-image-meta-name: Fedora-x86_64-20-20131211.1-sda
 from (pid=25312) log_http_response /opt/stack/python-glanceclient/glanceclient/common/http.py:153
2014-05-20 18:53:38.456 ERROR nova.api.openstack [req-44e54528-f86b-4771-a7a6-91f6af164058 demo demo] Caught error: No networks defined.
Traceback (most recent call last):

  File "/opt/stack/nova/nova/conductor/manager.py", line 597, in _object_dispatch
    return getattr(target, method)(context, *args, **kwargs)

  File "/opt/stack/nova/nova/objects/base.py", line 115, in wrapper
    result = fn(cls, context, *args, **kwargs)

  File "/opt/stack/nova/nova/objects/network.py", line 183, in get_by_uuids
    project_only)

  File "/opt/stack/nova/nova/db/api.py", line 1003, in network_get_all_by_uuids
    project_only=project_only)

  File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 164, in wrapper
    return f(*args, **kwargs)

  File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 2600, in network_get_all_by_uuids
    raise exception.NoNetworksFound()

NoNetworksFound: No networks defined.

Traceback (most recent call last):

  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
    incoming.message))

  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)

  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
    result = getattr(endpoint, method)(ctxt, **new_args)

  File "/opt/stack/nova/nova/network/manager.py", line 1351, in validate_networks
    self._get_networks_by_uuids(context, network_uuids)

  File "/opt/stack/nova/nova/network/manager.py", line 1377, in _get_networks_by_uuids
    context, network_uuids, project_only="allow_none")

  File "/opt/stack/nova/nova/objects/base.py", line 113, in wrapper
    args, kwargs)

  File "/opt/stack/nova/nova/conductor/rpcapi.py", line 355, in object_class_action
    objver=objver, args=args, kwargs=kwargs)

  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 150, in call
    wait_for_reply=True, timeout=timeout)

  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in _send
    timeout=timeout)

  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 386, in send
    return self._send(target, ctxt, message, wait_for_reply, timeout)

  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 379, in _send
    raise result

NoNetworksFound_Remote: No networks defined.
Traceback (most recent call last):

  File "/opt/stack/nova/nova/conductor/manager.py", line 597, in _object_dispatch
    return getattr(target, method)(context, *args, **kwargs)

  File "/opt/stack/nova/nova/objects/base.py", line 115, in wrapper
    result = fn(cls, context, *args, **kwargs)

  File "/opt/stack/nova/nova/objects/network.py", line 183, in get_by_uuids
    project_only)

  File "/opt/stack/nova/nova/db/api.py", line 1003, in network_get_all_by_uuids
    project_only=project_only)

  File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 164, in wrapper
    return f(*args, **kwargs)

  File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 2600, in network_get_all_by_uuids
    raise exception.NoNetworksFound()

NoNetworksFound: No networks defined.

2014-05-20 18:53:38.456 TRACE nova.api.openstack Traceback (most recent call last):
2014-05-20 18:53:38.456 TRACE nova.api.openstack File "/opt/stack/nova/nova/api/openstack/__init__.py", line 125, in __call__
2014-05-20 18:53:38.456 TRACE nova.api.openstack return req.get_response(self.application)
2014-05-20 18:53:38.456 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
2014-05-20 18:53:38.456 TRACE nova.api.openstack application, catch_exc_info=False)
2014-05-20 18:53:38.456 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
2014-05-20 18:53:38.456 TRACE nova.api.openstack app_iter = application(self.environ, start_response)
2014-05-20 18:53:38.456 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-05-20 18:53:38.456 TRACE nova.api.openstack return resp(environ, start_response)
2014-05-20 18:53:38.456 TRACE nova.api.openstack File "/opt/stack/python-keystoneclient/keystoneclient/middleware/auth_token.py", line 632, in __call__
2014-05-20 18:53:38.456 TRACE nova.api.openstack return self.app(env, start_response)
2014-05-20 18:53:38.456 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-05-20 18:53:38.456 TRACE nova.api.openstack return resp(environ, start_response)
2014-05-20 18:53:38.456 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-05-20 18:53:38.456 TRACE nova.api.openstack return resp(environ, start_response)
2014-05-20 18:53:38.456 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
2014-05-20 18:53:38.456 TRACE nova.api.openstack response = self.app(environ, start_response)
2014-05-20 18:53:38.456 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-05-20 18:53:38.456 TRACE nova.api.openstack return resp(environ, start_response)
2014-05-20 18:53:38.456 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2014-05-20 18:53:38.456 TRACE nova.api.openstack resp = self.call_func(req, *args, **self.kwargs)
2014-05-20 18:53:38.456 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2014-05-20 18:53:38.456 TRACE nova.api.openstack return self.func(req, *args, **kwargs)
2014-05-20 18:53:38.456 TRACE nova.api.openstack File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 917, in __call__
2014-05-20 18:53:38.456 TRACE nova.api.openstack content_type, body, accept)
2014-05-20 18:53:38.456 TRACE nova.api.openstack File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 983, in _process_stack
2014-05-20 18:53:38.456 TRACE nova.api.openstack action_result = self.dispatch(meth, request, action_args)
2014-05-20 18:53:38.456 TRACE nova.api.openstack File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 1067, in dispatch
2014-05-20 18:53:38.456 TRACE nova.api.openstack return method(req=request, **action_args)
2014-05-20 18:53:38.456 TRACE nova.api.openstack File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 956, in create
2014-05-20 18:53:38.456 TRACE nova.api.openstack legacy_bdm=legacy_bdm)
2014-05-20 18:53:38.456 TRACE nova.api.openstack File "/opt/stack/nova/nova/hooks.py", line 103, in inner
2014-05-20 18:53:38.456 TRACE nova.api.openstack rv = f(*args, **kwargs)
2014-05-20 18:53:38.456 TRACE nova.api.openstack File "/opt/stack/nova/nova/compute/api.py", line 1333, in create
2014-05-20 18:53:38.456 TRACE nova.api.openstack legacy_bdm=legacy_bdm)
2014-05-20 18:53:38.456 TRACE nova.api.openstack File "/opt/stack/nova/nova/compute/api.py", line 976, in _create_instance
2014-05-20 18:53:38.456 TRACE nova.api.openstack max_count)
2014-05-20 18:53:38.456 TRACE nova.api.openstack File "/opt/stack/nova/nova/compute/api.py", line 747, in _validate_and_build_base_options
2014-05-20 18:53:38.456 TRACE nova.api.openstack requested_networks, max_count)
2014-05-20 18:53:38.456 TRACE nova.api.openstack File "/opt/stack/nova/nova/compute/api.py", line 463, in _check_requested_networks
2014-05-20 18:53:38.456 TRACE nova.api.openstack max_count)
2014-05-20 18:53:38.456 TRACE nova.api.openstack File "/opt/stack/nova/nova/network/api.py", line 46, in wrapped
2014-05-20 18:53:38.456 TRACE nova.api.openstack return func(self, context, *args, **kwargs)
2014-05-20 18:53:38.456 TRACE nova.api.openstack File "/opt/stack/nova/nova/network/api.py", line 371, in validate_networks
2014-05-20 18:53:38.456 TRACE nova.api.openstack requested_networks)
2014-05-20 18:53:38.456 TRACE nova.api.openstack File "/opt/stack/nova/nova/network/rpcapi.py", line 230, in validate_networks
2014-05-20 18:53:38.456 TRACE nova.api.openstack return self.client.call(ctxt, 'validate_networks', networks=networks)
2014-05-20 18:53:38.456 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 361, in call
2014-05-20 18:53:38.456 TRACE nova.api.openstack return self.prepare().call(ctxt, method, **kwargs)
2014-05-20 18:53:38.456 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 150, in call
2014-05-20 18:53:38.456 TRACE nova.api.openstack wait_for_reply=True, timeout=timeout)
2014-05-20 18:53:38.456 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in _send
2014-05-20 18:53:38.456 TRACE nova.api.openstack timeout=timeout)
2014-05-20 18:53:38.456 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 386, in send
2014-05-20 18:53:38.456 TRACE nova.api.openstack return self._send(target, ctxt, message, wait_for_reply, timeout)
2014-05-20 18:53:38.456 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 379, in _send
2014-05-20 18:53:38.456 TRACE nova.api.openstack raise result
2014-05-20 18:53:38.456 TRACE nova.api.openstack NoNetworksFound_Remote: No networks defined.
2014-05-20 18:53:38.456 TRACE nova.api.openstack Traceback (most recent call last):
2014-05-20 18:53:38.456 TRACE nova.api.openstack
2014-05-20 18:53:38.456 TRACE nova.api.openstack File "/opt/stack/nova/nova/conductor/manager.py", line 597, in _object_dispatch
2014-05-20 18:53:38.456 TRACE nova.api.openstack return getattr(target, method)(context, *args, **kwargs)
2014-05-20 18:53:38.456 TRACE nova.api.openstack
2014-05-20 18:53:38.456 TRACE nova.api.openstack File "/opt/stack/nova/nova/objects/base.py", line 115, in wrapper
2014-05-20 18:53:38.456 TRACE nova.api.openstack result = fn(cls, context, *args, **kwargs)
2014-05-20 18:53:38.456 TRACE nova.api.openstack
2014-05-20 18:53:38.456 TRACE nova.api.openstack File "/opt/stack/nova/nova/objects/network.py", line 183, in get_by_uuids
2014-05-20 18:53:38.456 TRACE nova.api.openstack project_only)
2014-05-20 18:53:38.456 TRACE nova.api.openstack
2014-05-20 18:53:38.456 TRACE nova.api.openstack File "/opt/stack/nova/nova/db/api.py", line 1003, in network_get_all_by_uuids
2014-05-20 18:53:38.456 TRACE nova.api.openstack project_only=project_only)
2014-05-20 18:53:38.456 TRACE nova.api.openstack
2014-05-20 18:53:38.456 TRACE nova.api.openstack File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 164, in wrapper
2014-05-20 18:53:38.456 TRACE nova.api.openstack return f(*args, **kwargs)
2014-05-20 18:53:38.456 TRACE nova.api.openstack
2014-05-20 18:53:38.456 TRACE nova.api.openstack File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 2600, in network_get_all_by_uuids
2014-05-20 18:53:38.456 TRACE nova.api.openstack raise exception.NoNetworksFound()
2014-05-20 18:53:38.456 TRACE nova.api.openstack
2014-05-20 18:53:38.456 TRACE nova.api.openstack NoNetworksFound: No networks defined.
2014-05-20 18:53:38.456 TRACE nova.api.openstack
2014-05-20 18:53:38.456 TRACE nova.api.openstack Traceback (most recent call last):
2014-05-20 18:53:38.456 TRACE nova.api.openstack
2014-05-20 18:53:38.456 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
2014-05-20 18:53:38.456 TRACE nova.api.openstack incoming.message))
2014-05-20 18:53:38.456 TRACE nova.api.openstack
2014-05-20 18:53:38.456 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
2014-05-20 18:53:38.456 TRACE nova.api.openstack return self._do_dispatch(endpoint, method, ctxt, args)
2014-05-20 18:53:38.456 TRACE nova.api.openstack
2014-05-20 18:53:38.456 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
2014-05-20 18:53:38.456 TRACE nova.api.openstack result = getattr(endpoint, method)(ctxt, **new_args)
2014-05-20 18:53:38.456 TRACE nova.api.openstack result = getattr(endpoint, method)(ctxt, **new_args)
2014-05-20 18:53:38.456 TRACE nova.api.openstack
2014-05-20 18:53:38.456 TRACE nova.api.openstack File "/opt/stack/nova/nova/network/manager.py", line 1351, in validate_networks
2014-05-20 18:53:38.456 TRACE nova.api.openstack self._get_networks_by_uuids(context, network_uuids)
2014-05-20 18:53:38.456 TRACE nova.api.openstack
2014-05-20 18:53:38.456 TRACE nova.api.openstack File "/opt/stack/nova/nova/network/manager.py", line 1377, in _get_networks_by_uuids
2014-05-20 18:53:38.456 TRACE nova.api.openstack context, network_uuids, project_only="allow_none")
2014-05-20 18:53:38.456 TRACE nova.api.openstack
2014-05-20 18:53:38.456 TRACE nova.api.openstack File "/opt/stack/nova/nova/objects/base.py", line 113, in wrapper
2014-05-20 18:53:38.456 TRACE nova.api.openstack args, kwargs)
2014-05-20 18:53:38.456 TRACE nova.api.openstack
2014-05-20 18:53:38.456 TRACE nova.api.openstack File "/opt/stack/nova/nova/conductor/rpcapi.py", line 355, in object_class_action
2014-05-20 18:53:38.456 TRACE nova.api.openstack objver=objver, args=args, kwargs=kwargs)
2014-05-20 18:53:38.456 TRACE nova.api.openstack
2014-05-20 18:53:38.456 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 150, in call
2014-05-20 18:53:38.456 TRACE nova.api.openstack wait_for_reply=True, timeout=timeout)
2014-05-20 18:53:38.456 TRACE nova.api.openstack
2014-05-20 18:53:38.456 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in _send
2014-05-20 18:53:38.456 TRACE nova.api.openstack timeout=timeout)
2014-05-20 18:53:38.456 TRACE nova.api.openstack
2014-05-20 18:53:38.456 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 386, in send
2014-05-20 18:53:38.456 TRACE nova.api.openstack return self._send(target, ctxt, message, wait_for_reply, timeout)
2014-05-20 18:53:38.456 TRACE nova.api.openstack
2014-05-20 18:53:38.456 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 379, in _send
2014-05-20 18:53:38.456 TRACE nova.api.openstack raise result
2014-05-20 18:53:38.456 TRACE nova.api.openstack
2014-05-20 18:53:38.456 TRACE nova.api.openstack NoNetworksFound_Remote: No networks defined.
2014-05-20 18:53:38.456 TRACE nova.api.openstack Traceback (most recent call last):
2014-05-20 18:53:38.456 TRACE nova.api.openstack
2014-05-20 18:53:38.456 TRACE nova.api.openstack File "/opt/stack/nova/nova/conductor/manager.py", line 597, in _object_dispatch
2014-05-20 18:53:38.456 TRACE nova.api.openstack return getattr(target, method)(context, *args, **kwargs)
2014-05-20 18:53:38.456 TRACE nova.api.openstack
2014-05-20 18:53:38.456 TRACE nova.api.openstack File "/opt/stack/nova/nova/objects/base.py", line 115, in wrapper
2014-05-20 18:53:38.456 TRACE nova.api.openstack result = fn(cls, context, *args, **kwargs)
2014-05-20 18:53:38.456 TRACE nova.api.openstack
2014-05-20 18:53:38.456 TRACE nova.api.openstack File "/opt/stack/nova/nova/objects/network.py", line 183, in get_by_uuids
2014-05-20 18:53:38.456 TRACE nova.api.openstack project_only)
2014-05-20 18:53:38.456 TRACE nova.api.openstack
2014-05-20 18:53:38.456 TRACE nova.api.openstack File "/opt/stack/nova/nova/db/api.py", line 1003, in network_get_all_by_uuids
2014-05-20 18:53:38.456 TRACE nova.api.openstack project_only=project_only)
2014-05-20 18:53:38.456 TRACE nova.api.openstack
2014-05-20 18:53:38.456 TRACE nova.api.openstack File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 164, in wrapper
2014-05-20 18:53:38.456 TRACE nova.api.openstack return f(*args, **kwargs)
2014-05-20 18:53:38.456 TRACE nova.api.openstack
2014-05-20 18:53:38.456 TRACE nova.api.openstack File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 2600, in network_get_all_by_uuids
2014-05-20 18:53:38.456 TRACE nova.api.openstack raise exception.NoNetworksFound()
2014-05-20 18:53:38.456 TRACE nova.api.openstack
2014-05-20 18:53:38.456 TRACE nova.api.openstack NoNetworksFound: No networks defined.
2014-05-20 18:53:38.456 TRACE nova.api.openstack
2014-05-20 18:53:38.456 TRACE nova.api.openstack
2014-05-20 18:53:38.456 TRACE nova.api.openstack
2014-05-20 18:53:38.494 INFO nova.api.openstack [req-44e54528-f86b-4771-a7a6-91f6af164058 demo demo] http://172.16.140.247:8774/v2/3859a6e8a68f4e2db486a9fd31bbcc62/servers returned with HTTP 404
2014-05-20 18:53:38.495 DEBUG nova.api.openstack.wsgi [req-44e54528-f86b-4771-a7a6-91f6af164058 demo demo] Returning 404 to user: The resource could not be found. from (pid=25312) __call__ /opt/stack/nova/nova/api/openstack/wsgi.py:1209
2014-05-20 18:53:38.496 INFO nova.osapi_compute.wsgi.server [req-44e54528-f86b-4771-a7a6-91f6af164058 demo demo] 192.168.66.4 "POST /v2/3859a6e8a68f4e2db486a9fd31bbcc62/servers HTTP/1.1" status: 404 len: 272 time: 0.1439140

Tags: compute
Tracy Jones (tjones-i)
tags: added: compute
Revision history for this message
Christopher Yeoh (cyeoh-0) wrote :

Are you still able to replicate this bug? I've been unable to reproduce this on trusty with devstack

Changed in nova:
status: New → Incomplete
Revision history for this message
Sean Dague (sdague) wrote :

I'm assuming this is now addressed, haven't seen this in a while

Changed in nova:
status: Incomplete → Fix Released
Revision history for this message
John Griffith (john-griffith) wrote :

So I did some more poking at this after seeing it on ask.openstack again. There is still all sorts of strange behavior when I try and run this from a macbook;

First novaclient 2.23.2 (what's installed on my compute node) gives:
➜ ~ nova list
/usr/local/lib/python2.7/site-packages/novaclient/v1_1/__init__.py:30: UserWarning: Module novaclient.v1_1 is deprecated (taken as a basis for novaclient.v2). The preferable way to get client class or object you can find in novaclient.client module.
  warnings.warn("Module novaclient.v1_1 is deprecated (taken as a basis for "
ERROR (AttributeError): 'module' object has no attribute 'arg'
➜ ~

Forcing pip install version 2.15 fixed that; but the boot cmd still has this issue. Note that all other cmds seem to work, including net-list etc.

Jumped on my linux machine, and everything works fine. Seems to be something in osx env's. If I have a chance I'll try and gather more info/details. Currently trying a virtualenv to make sure everything is clean but seem to have same issue.

Anyway.... might explain why it's not been reproduced by others etc.

Revision history for this message
Rohanil Raje (rohanil) wrote :

I could still reproduce the same error in trusty. Getting error while creating vm on remote node. There is no error if I create vm on same node. When I check `nova service-list`, I get remote node as well in list. When I do `nova net-list` on both nodes, it gives same list of networks on both nodes. I don't know how to fix it. Any idea ?

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.