os-hypervisors detail api throws 500

[Amit Uniyal]

Summary:
os-hypervisors detail api throws 500

Description
===========

After stopping compute service in one of the compute node, if we look for hypervisor list for compute status, its returning 500, howerever it should tell which hypervisor is up and down.

The API wait and try to retrieve (polling), but eventually its fails with 500, as it could not get info on particular node.

Steps to reproduce
==================
- stop a compute service in one of the compute node
- run openstack hypervisor list

Expected result
===============

Service status should be shown down

$ openstack hypervisor list
+--------------------------------------+---------------------+-----------------+--------------+-------+
| ID | Hypervisor Hostname | Hypervisor Type | Host IP | State |
+--------------------------------------+---------------------+-----------------+--------------+-------+
| 9c28d019-f9c8-4a53-a40c-4607c433a8bf | amit-mns-devstack-1 | QEMU | 10.0.109.22 | up |
| e289500b-a59c-4e57-87bd-2a9b4a506976 | amit-mns-devstack-2 | QEMU | 10.0.109.250 | down |
+--------------------------------------+---------------------+-----------------+--------------+-------+

Actual result
=============
API is throwing 500 Internal Server Error

Error:

HttpException: 500: Server Error for url: http://10.0.109.22/compute/v2.1/os-hypervisors/detail, : 500 Internal Server Error: Internal Server Error: The server encountered an internal error or: misconfiguration and was unable to complete: your request.: Please contact the server administrator at: root@localhost to inform them of the time this error occurred,: and the actions you performed just before this error.: More information about this error may be available: in the server error log.
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cliff/app.py", line 410, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/local/lib/python3.9/site-packages/osc_lib/command/command.py", line 38, in run
    return super(Command, self).run(parsed_args)
  File "/usr/local/lib/python3.9/site-packages/cliff/display.py", line 120, in run
    self.produce_output(parsed_args, column_names, data)
  File "/usr/local/lib/python3.9/site-packages/cliff/lister.py", line 122, in produce_output
    self.formatter.emit_list(
  File "/usr/local/lib/python3.9/site-packages/cliff/formatters/table.py", line 105, in emit_list
    self.add_rows(x, column_names, data)
  File "/usr/local/lib/python3.9/site-packages/cliff/formatters/table.py", line 84, in add_rows
    first_row = next(data_iter)
  File "/usr/local/lib/python3.9/site-packages/openstackclient/compute/v2/hypervisor.py", line 150, in <genexpr>
    (utils.get_item_properties(s, columns) for s in data),
  File "/usr/local/lib/python3.9/site-packages/openstack/resource.py", line 2093, in list
    exceptions.raise_from_response(response)
  File "/usr/local/lib/python3.9/site-packages/openstack/exceptions.py", line 247, in raise_from_response
    raise cls(
openstack.exceptions.HttpException: HttpException: 500: Server Error for url: http://10.0.109.22/compute/v2.1/os-hypervisors/detail, : 500 Internal Server Error: Internal Server Error: The server encountered an internal error or: misconfiguration and was unable to complete: your request.: Please contact the server administrator at: root@localhost to inform them of the time this error occurred,: and the actions you performed just before this error.: More information about this error may be available: in the server error log.
clean_up ListHypervisor: HttpException: 500: Server Error for url: http://10.0.109.22/compute/v2.1/os-hypervisors/detail, : 500 Internal Server Error: Internal Server Error: The server encountered an internal error or: misconfiguration and was unable to complete: your request.: Please contact the server administrator at: root@localhost to inform them of the time this error occurred,: and the actions you performed just before this error.: More information about this error may be available: in the server error log.

[Amit Uniyal (auniyal)]

Tested in different environment too with Antelope

Before service stop
--------------------
sh-5.1$ openstack hypervisor list
+--------------------------------------+--------------------------------+-----------------+-----------------+-------+
| ID | Hypervisor Hostname | Hypervisor Type | Host IP | State |
+--------------------------------------+--------------------------------+-----------------+-----------------+-------+
| 1b6ee03f-0a68-429c-982d-3b037888af38 | compute-2.ctlplane.example.com | QEMU | 192.168.122.102 | up |
| 680d4962-3f1e-4beb-9b52-7d9be573cf13 | compute-0.ctlplane.example.com | QEMU | 192.168.122.100 | up |
| c110dfc2-936e-4d4d-bd8d-629329e757cb | compute-1.ctlplane.example.com | QEMU | 192.168.122.101 | up |
+--------------------------------------+--------------------------------+-----------------+-----------------+-------+

After service stop
--------------------
sh-5.1$ openstack hypervisor list
HttpException: 504: Server Error for url: https://nova-public-openstack.apps.ocp.openstack.lab/v2.1/os-hypervisors/detail, The server didn't respond in time.: 504 Gateway Time-out
sh-5.1$

[Amit Uniyal (auniyal)]

this eventually (like >10 mins) stops failing, after some time, and we get expected output.

But again, when we start; and then stop compute service in compute node, the issue is reproducible