"IndexError: list index out of range" in ExceptionPayload.from_exception during resize failure

[Matt Riedemann]

I noticed this from the nova-compute logs attached to bug 1777157:

2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [req-69050681-be73-48c5-be25-993f05b0b47e e511323a68a740f597880a1b056e3918 28f4e0797d544d758700e5be42aa82c6 - default default] [instance: daa5b10b-0655-4178-9413-66697004d7f8] Setting instance vm_state to ERROR: IndexError: list index out of range
2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8] Traceback (most recent call last):
2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 7445, in _error_out_instance_on_exception
2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8] yield
2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 4162, in prep_resize
2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8] filter_properties, host_list)
2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 4221, in _reschedule_resize_or_reraise
2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8] exception=exc_info[1])
2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/rpc.py", line 225, in wrapped
2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8] return f(*args, **kwargs)
2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/utils.py", line 367, in notify_about_instance_action
2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8] fault, priority = _get_fault_and_priority_from_exc(exception)
2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/utils.py", line 348, in _get_fault_and_priority_from_exc
2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8] exception)
2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/notifications/objects/exception.py", line 42, in from_exception
2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8] trace = inspect.trace()[-1]
2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8] IndexError: list index out of range
2018-06-15 16:23:46.954 7 ERROR nova.compute.manager [instance: daa5b10b-0655-4178-9413-66697004d7f8]

This is based on queens, and this is the problem:

https://github.com/openstack/nova/blob/530c24596a74a6115f2e944176c46e4f3d199811/nova/compute/manager.py#L4216

When we call that, we're not in an except block, so this fails later:

https://github.com/openstack/nova/blob/530c24596a74a6115f2e944176c46e4f3d199811/nova/notifications/objects/exception.py#L42

Although, looking higher up the stack, we are in an except block here:

https://github.com/openstack/nova/blob/530c24596a74a6115f2e944176c46e4f3d199811/nova/compute/manager.py#L4155

So this is a bit weird...

[Balazs Gibizer (balazs-gibizer)]

I agree, it feels strange. Let me try to reproduce it with a functional test.

[Balazs Gibizer (balazs-gibizer)]

I failed to reproduce so far. The failed trial is up in https://review.openstack.org/#/c/576593

[Balazs Gibizer (balazs-gibizer)]

The bug reported provided more information in the original bug report and from it seems [1] that the nova code is heavily diverged from upstream. Based on [1], the fact that I failed to reproduce the problem and that the reporter set the original bug to Invalid state I set this to invalid as well.

[1] https://bugs.launchpad.net/nova/+bug/1777157/comments/16