NeutronClientException: 404 Not Found

[Derek Higgins]

Tripleo isn't currently working with trunk nova, the undercloud is failing to build overcloud instances, nova compute is showing this exception

Nov 05 13:10:45 instack.localdomain nova-compute[21338]: 2015-11-05 13:10:45.163 21338 ERROR nova.virt.ironic.driver [req-7df4cae6-f00a-41a2-91e0-db1e6f130059 a800cb834fbd4a70915e2272dce924ac 102a2b78e079410f9afd8b8b46278c19 - - -] Error preparing deploy for instance 9ae5b605-58e3-40ee-b944-56cbf5806e51 on baremetal node f5c30846-4ada-444e-85d9-6e3be2a74782.

Nov 05 13:10:45 instack.localdomain nova-compute[21338]: 2015-11-05 13:10:45.434 21338 DEBUG nova.virt.ironic.driver [req-7df4cae6-f00a-41a2-91e0-db1e6f130059 a800cb834fbd4a70915e2272dce924ac 102a2b78e079410f9afd8b8b46278c19 - - -] unplug: instance_uuid=9ae5b605-58e3-40ee-b944-56cbf5806e51 vif=[] _unplug_vifs /usr/lib/python2.7/site-packages/nova/virt/ironic/driver.py:1093
 Instance failed to spawn
 Traceback (most recent call last):
   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2165, in _build_resources
     yield resources
   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2012, in _build_and_run_instance
     block_device_info=block_device_info)
   File "/usr/lib/python2.7/site-packages/nova/virt/ironic/driver.py", line 791, in spawn
     flavor=flavor)
   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 197, in __exit__
     six.reraise(self.type_, self.value, self.tb)
   File "/usr/lib/python2.7/site-packages/nova/virt/ironic/driver.py", line 782, in spawn
     self._plug_vifs(node, instance, network_info)
   File "/usr/lib/python2.7/site-packages/nova/virt/ironic/driver.py", line 1058, in _plug_vifs
     network_info_str = str(network_info)
   File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 515, in __str__
     return self._sync_wrapper(fn, *args, **kwargs)
   File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 498, in _sync_wrapper
     self.wait()
   File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 530, in wait
     self[:] = self._gt.wait()
   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 175, in wait
     return self._exit_event.wait()
   File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 125, in wait
     current.throw(*self._exc)
   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main
     result = function(*args, **kwargs)
   File "/usr/lib/python2.7/site-packages/nova/utils.py", line 1178, in context_wrapper
     return func(*args, **kwargs)
   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1574, in _allocate_network_async
     six.reraise(*exc_info)
   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1557, in _allocate_network_async
     dhcp_options=dhcp_options)
   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 733, in allocate_for_instance
     update_cells=True)
   File "/usr/lib/python2.7/site-packages/nova/network/base_api.py", line 244, in get_instance_nw_info
     result = self._get_instance_nw_info(context, instance, **kwargs)
   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 930, in _get_instance_nw_info
     preexisting_port_ids)
   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1708, in _build_network_info_model
     current_neutron_port)
   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1560, in _nw_info_get_ips
     client, fixed_ip['ip_address'], port['id'])
   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1491, in _get_floating_ips_by_fixed_and_port
     port_id=port)
   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1475, in _safe_get_floating_ips
     for k, v in six.iteritems(kwargs)]))
   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 197, in __exit__
     six.reraise(self.type_, self.value, self.tb)
   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1466, in _safe_get_floating_ips
     return client.list_floatingips(**kwargs)['floatingips']
   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 102, in with_params
     ret = self.function(instance, *args, **kwargs)
   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 734, in list_floatingips
     **_params)
   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 307, in list
     for r in self._pagination(collection, path, **params):
   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 320, in _pagination
     res = self.get(path, params=params)
   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 293, in get
     headers=headers, params=params)
   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 270, in retry_request
     headers=headers, params=params)
   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 211, in do_request
     self._handle_fault_response(status_code, replybody)
   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 185, in _handle_fault_response
     exception_handler_v20(status_code, des_error_body)
   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 83, in exception_handler_v20
     message=message)
 NeutronClientException: 404 Not Found

 The resource could not be found.

[Derek Higgins (derekh)]

proposed fix here https://review.openstack.org/#/c/242158/

[Sean Dague (sdague)]

This appears to be triggering because python-neutronclient is getting a non Neutron exception, from somewhere, and somewhere in the Neutron / Neutronclient path this isn't being translated.

Before just reverting the clean up in Nova, we should figure out exactly what that is. Because it's quite surprising that this kind of leak through is happening.

[Sean Dague (sdague)]

It would be great to get the relevant request logs for these request flows to figure out where the root of this unexpected exception by neutronclient really is. The right fix is probably in the neutron side to be consistent here.

[Sean Dague (sdague)]

The existing understanding is that this is how python-neutronclient functions if there is no L3 agent. However, that seems really wrong. Because it means you have to know the topology of the services on the neutron side in order to use python-neutronclient correctly.

That someone defeats the purpose of having a library to access your service. This really should be fixed in python-neutronclient.

[Derek Higgins (derekh)]

Essentially whats happening on neutronclient is self.deserialize is throwing an exception as its trying to parse non json

    response_body == '404 Not Found\n\nThe resource could not be found.\n\n '

exception_handler_v20 then throws a NeutronClientException as its "probably not a Neutron error", the nova code is then assuming a NotFound exception is being thrown (it used to catch the broader case NeutronClientException)

    def _handle_fault_response(self, status_code, response_body):
        # Create exception with HTTP status code and message
        _logger.debug("Error message: %s", response_body)
        # Add deserialized error message to exception arguments
        try:
            des_error_body = self.deserialize(response_body, status_code)
        except Exception:
            # If unable to deserialized body it is probably not a
            # Neutron error
            des_error_body = {'message': response_body}
        # Raise the appropriate exception
        exception_handler_v20(status_code, des_error_body)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/244219

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/242158
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=cbbbaa93aeeb7c7902867842ac8890928894cf96
Submitter: Jenkins
Branch: master

commit cbbbaa93aeeb7c7902867842ac8890928894cf96
Author: Derek Higgins <email address hidden>
Date: Thu Nov 5 16:36:55 2015 +0000

    Handle a NeutronClientException 404 Error for floating ips

    I9229d882712df1ad57afbda7477fa4d72f4fd83c introduced _safe_get_floating_ips
    which returns an empty list of floating ips in cases where neutron returns
    a 404. It also switched a check for NeutronClientException with NotFound.
    The NotFound Exception isn't used by neutron client so allow for both cases.

    Closes-bug: #1513879
    Change-Id: I1d37a0dadd4c9055ba545906535bb3e40e70a0f0

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/244219
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=69c39b731541415b6cd4c7b2d8b859f8134f7ed8
Submitter: Jenkins
Branch: master

commit 69c39b731541415b6cd4c7b2d8b859f8134f7ed8
Author: Dan Smith <email address hidden>
Date: Wed Nov 11 09:13:47 2015 -0800

    Add test coverage for both types of not-found-ness in neutronclient for floating

    This adds coverage for the exceptions.NotFound case when we're getting
    floating IPs from neutronclient. It makes the NeutronClientException case
    the "legacy" one that we can remove when we fix that bug.

    Change-Id: I9c8b20e076527f98085265755a2d2503c703f235
    Partial-Bug: 1513879

[Akihiro Motoki (amotoki)]

neutronclient at least should raise an exception corresponding to a status code returned from neutron server.

[Akihiro Motoki (amotoki)]

In addition, this means neutron-server returns an exception in non-usual format. It is also better to be investigated.

[Doug Wiegley (dougwig)]

Akihiro is chasing this one down on both sides of neutron. Though the upcoming pecan switch will likely render this moot as well.

[Akihiro Motoki (amotoki)]

I am now preparing a patch in neutronclient side which maps a response code into an exception (notfound, conflict or something) regardless of what message is returned. it will reduce pain points nova usually feels.

Once this patch lands, we don't need to rush to fix it in the server side and we can fix the root cause in the upcoming pecan switch.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/253267

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/253267
Committed: https://git.openstack.org/cgit/openstack/python-neutronclient/commit/?id=af1a55bfd2e47b0e3cd8349f0a9b1277474fee18
Submitter: Jenkins
Branch: master

commit af1a55bfd2e47b0e3cd8349f0a9b1277474fee18
Author: Akihiro Motoki <email address hidden>
Date: Fri Dec 4 01:53:58 2015 +0900

    Ensure to use exception per status code for all cases

    Previously, only when an exception has a content with
    {'NeutronError': {'type': xxxx, 'message': xxxx}},
    exception per status code is raised from neutronclient library.
    There are cases where this kind of message is not contained
    in exception messages, for example, some extension is loaded.

    Library users expect an exception is raised based on response
    status code and it should not depend on an exception message.
    This commit applies a fallback logic to map generic per-status
    exception to all exception types from the neutron server.

    Closes-Bug: #1513879
    Change-Id: Ib3d0a8359aed444b12217b3404d40443d61fc2c0

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/python-neutronclient 4.1.0 release.