router rebuild appears to be busted

Bug #1459445 reported by Adam Gandelman
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Astara
Fix Released
Critical
Adam Gandelman
akanda
Fix Released
Critical
Adam Gandelman
Kilo
Fix Released
Critical
Adam Gandelman
akanda-rug
Invalid
Undecided
Adam Gandelman

Bug Description

Steps to reproduce:

deploy devstack
wait for initial router to become active
rebuild the router using: rug-ctl router rebuild --router_image_uuid $image_uuid $router_id

The instance terminates and the akanda-rug-service starts spinning on an exception:

2015-05-27 14:39:44:DEBUG:akanda.rug.api.nova:29673:p01:t01:deleting vm for router ak-13e63061-3590-4296-8b63-945a144cd0b7
2015-05-27 14:39:44:ERROR:akanda.rug.state.13e63061-3590-4296-8b63-945a144cd0b7:29673:p01:t01:Error deleting router instance
Traceback (most recent call last):
  File "/opt/stack/akanda-rug/akanda/rug/vm_manager.py", line 281, in stop
    nova_client.destroy_instance(self.instance_info)
  File "/opt/stack/akanda-rug/akanda/rug/api/nova.py", line 146, in destroy_instance
    self.client.servers.delete(instance_info.id_)
  File "/usr/local/lib/python2.7/dist-packages/novaclient/v2/servers.py", line 946, in delete
    self._delete("/servers/%s" % base.getid(server))
  File "/usr/local/lib/python2.7/dist-packages/novaclient/base.py", line 170, in _delete
    _resp, _body = self.api.client.delete(url)
  File "/usr/local/lib/python2.7/dist-packages/novaclient/client.py", line 461, in delete
    return self._cs_request(url, 'DELETE', **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/novaclient/client.py", line 430, in _cs_request
    resp, body = self._time_request(url, method, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/novaclient/client.py", line 403, in _time_request
    resp, body = self.request(url, method, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/novaclient/client.py", line 397, in request
    raise exceptions.from_response(resp, body, url, method)
NotFound: Instance cd80be34-ffbf-41a5-b511-d20103631e60 could not be found. (HTTP 404) (Request-ID: req-b8942dd3-2c02-4840-9d64-36a4815d55dd)
2015-05-27 14:39:44:ERROR:akanda.rug.state.13e63061-3590-4296-8b63-945a144cd0b7:29673:p01:t01:RebuildVM.execute() failed for action: rebuild

Tags: akanda-rug
Revision history for this message
Adam Gandelman (gandelman-a) wrote :

a full log of the event

Changed in akanda:
status: New → Confirmed
importance: Undecided → Critical
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to akanda-rug (master)

Fix proposed to branch: master
Review: https://review.openstack.org/186216

Changed in akanda:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to akanda-rug (master)

Reviewed: https://review.openstack.org/186216
Committed: https://git.openstack.org/cgit/stackforge/akanda-rug/commit/?id=ceaf3ce53c29c20dea02fba1b9dcead437991d0c
Submitter: Jenkins
Branch: master

commit ceaf3ce53c29c20dea02fba1b9dcead437991d0c
Author: Adam Gandelman <email address hidden>
Date: Wed May 27 16:08:55 2015 -0700

    Fix get_instance_by_id, catch NotFound appopriately

    novaclient will raise a NotFound exception here instead of returning None,
    which the rest of the RUG assumes.

    Change-Id: I6812bd01bb4a56433bc1c408040470d3711a0005
    Closes-bug: #1459445

Changed in akanda:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to akanda-rug (stable/kilo)

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/187241

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to akanda-rug (stable/kilo)

Reviewed: https://review.openstack.org/187241
Committed: https://git.openstack.org/cgit/stackforge/akanda-rug/commit/?id=1023978760db822a61aa25aaaf6c5f3aa7ab6700
Submitter: Jenkins
Branch: stable/kilo

commit 1023978760db822a61aa25aaaf6c5f3aa7ab6700
Author: Adam Gandelman <email address hidden>
Date: Wed May 27 16:08:55 2015 -0700

    Fix get_instance_by_id, catch NotFound appopriately

    novaclient will raise a NotFound exception here instead of returning None,
    which the rest of the RUG assumes.

    Change-Id: I6812bd01bb4a56433bc1c408040470d3711a0005
    Closes-bug: #1459445
    (cherry picked from commit ceaf3ce53c29c20dea02fba1b9dcead437991d0c)

Sean Roberts (sarob)
Changed in akanda:
milestone: none → liberty-1
Changed in akanda-rug:
assignee: nobody → Adam Gandelman (gandelman-a)
status: New → Invalid
Changed in akanda:
assignee: nobody → Adam Gandelman (gandelman-a)
tags: added: akanda-rug
Sean Roberts (sarob)
Changed in akanda:
status: Fix Committed → Fix Released
Changed in astara:
importance: Undecided → Critical
status: New → Fix Released
milestone: none → 7.0.0
Changed in akanda:
milestone: liberty-1 → 7.0.0
Changed in astara:
assignee: nobody → Adam Gandelman (gandelman-a)
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Bug attachments

Remote bug watches

Bug watches keep track of this bug in other bug trackers.