Nova returns 'The requested availability zone is not available (HTTP 400)' on any cell errors

[Andrey Volkov]

Description
===========

In case of for example DB errors `openstack server create --availability-zone ...` command returns
"The requested availability zone is not available (HTTP 400)" instead of actual 50x error.
This considered as client error and e.g. stops Heat to retry requests.

For example:

root@dev01:~# openstack server create --image cirros-0.5.1-x86_64-disk --flavor c1 vm5 --net public --availability-zone nova
The requested availability zone is not available (HTTP 400) (Request-ID: req-50190228-51ac-4303-ad3e-d8b920bb7ad8)

Logs:
Jan 19 12:37:32 dev01 <email address hidden>[4869]: ERROR nova.context [None req-50190228-51ac-4303-ad3e-d8b920bb7ad8 admin admin] Error gathering result from cell 00000000-0000-0000-0000-000000000000: ValueError: Cannot get service list (artificial error to help reproduce a bug)
Jan 19 12:37:32 dev01 <email address hidden>[4869]: ERROR nova.context Traceback (most recent call last):
Jan 19 12:37:32 dev01 <email address hidden>[4869]: ERROR nova.context File "/opt/stack/nova/nova/context.py", line 426, in gather_result
Jan 19 12:37:32 dev01 <email address hidden>[4869]: ERROR nova.context result = fn(*args, **kwargs)
Jan 19 12:37:32 dev01 <email address hidden>[4869]: ERROR nova.context File "/usr/local/lib/python3.6/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper
Jan 19 12:37:32 dev01 <email address hidden>[4869]: ERROR nova.context result = fn(cls, context, *args, **kwargs)
Jan 19 12:37:32 dev01 <email address hidden>[4869]: ERROR nova.context File "/opt/stack/nova/nova/objects/service.py", line 635, in get_all
Jan 19 12:37:32 dev01 <email address hidden>[4869]: ERROR nova.context raise ValueError('Cannot get service list (artificial error to help reproduce a bug)')
Jan 19 12:37:32 dev01 <email address hidden>[4869]: ERROR nova.context ValueError: Cannot get service list (artificial error to help reproduce a bug)
Jan 19 12:37:32 dev01 <email address hidden>[4869]: ERROR nova.context
Jan 19 12:37:32 dev01 <email address hidden>[4869]: DEBUG nova.objects.service [None req-50190228-51ac-4303-ad3e-d8b920bb7ad8 admin admin] ! service get_all {{(pid=4872) get_all /opt/stack/nova/nova/objects/service.py:632}}
Jan 19 12:37:32 dev01 <email address hidden>[4869]: ERROR nova.context [None req-50190228-51ac-4303-ad3e-d8b920bb7ad8 admin admin] Error gathering result from cell 558d4e82-cdc0-4020-a56b-6835326f58ec: ValueError: Cannot get service list (artificial error to help reproduce a bug)
Jan 19 12:37:32 dev01 <email address hidden>[4869]: ERROR nova.context Traceback (most recent call last):
Jan 19 12:37:32 dev01 <email address hidden>[4869]: ERROR nova.context File "/opt/stack/nova/nova/context.py", line 426, in gather_result
Jan 19 12:37:32 dev01 <email address hidden>[4869]: ERROR nova.context result = fn(*args, **kwargs)
Jan 19 12:37:32 dev01 <email address hidden>[4869]: ERROR nova.context File "/usr/local/lib/python3.6/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper
Jan 19 12:37:32 dev01 <email address hidden>[4869]: ERROR nova.context result = fn(cls, context, *args, **kwargs)
Jan 19 12:37:32 dev01 <email address hidden>[4869]: ERROR nova.context File "/opt/stack/nova/nova/objects/service.py", line 635, in get_all
Jan 19 12:37:32 dev01 <email address hidden>[4869]: ERROR nova.context raise ValueError('Cannot get service list (artificial error to help reproduce a bug)')
Jan 19 12:37:32 dev01 <email address hidden>[4869]: ERROR nova.context ValueError: Cannot get service list (artificial error to help reproduce a bug)
Jan 19 12:37:32 dev01 <email address hidden>[4869]: ERROR nova.context
Jan 19 12:37:32 dev01 <email address hidden>[4869]: WARNING nova.compute.api [None req-50190228-51ac-4303-ad3e-d8b920bb7ad8 admin admin] Cell 00000000-0000-0000-0000-000000000000 is not responding and hence skipped from the results.
Jan 19 12:37:32 dev01 <email address hidden>[4869]: WARNING nova.compute.api [None req-50190228-51ac-4303-ad3e-d8b920bb7ad8 admin admin] Cell 558d4e82-cdc0-4020-a56b-6835326f58ec is not responding and hence skipped from the results.
Jan 19 12:37:32 dev01 <email address hidden>[4869]: INFO nova.api.openstack.wsgi [None req-50190228-51ac-4303-ad3e-d8b920bb7ad8 admin admin] HTTP exception thrown: The requested availability zone is not available

Under the same conditions, `openstack compute service list` returns an empty success result.

Steps to reproduce
==================

To emulate runtime errors something like that can be used:

@@ -629,6 +629,10 @@ class ServiceList(base.ObjectListBase, base.NovaObject):

     @base.remotable_classmethod
     def get_all(cls, context, disabled=None, set_zones=False):
+ import datetime
+ if datetime.datetime.now() > datetime.datetime(2021, 1, 19, 12, 7, 0, 976979):
+ raise ValueError('Cannot get service list (artificial error to help reproduce a bug)')
         db_services = db.service_get_all(context, disabled=disabled)
         if set_zones:
             db_services = availability_zones.set_availability_zones(

[Balazs Gibizer (balazs-gibizer)]

Patch proposed https://review.opendev.org/c/openstack/nova/+/771440