VMs listing with sort keys throws exception when trying to compare None values

[Anton Kurbatov]

The nova-api raises exception on attempt to get VMs sorted by i.e. task_state key.

Here are steps-to-reproduce:

- create two VMs: vm1 in ACTIVE state (cell1) and vm2 in ERROR state (cell0)
- try to list servers sorted by sort_key=task_state

[root@node0 ~]# openstack server create vm1 --network net1 --flavor m1.tiny --image cirros-0.5.2-x86_64-disk
[root@node0 ~]# openstack server create vm2 --network net1 --flavor m1.xlarge --image cirros-0.5.2-x86_64-disk
[root@node0 ~]# openstack server list -f json --long -c ID -c 'Task State' -c 'Status'
[
  {
    "ID": "3a3927c4-9f67-4356-8a3e-a3e58cf0744e",
    "Status": "ERROR",
    "Task State": null
  },
  {
    "ID": "9af631ec-3e59-45da-bafa-85141e3707da",
    "Status": "ACTIVE",
    "Task State": null
  }
]
[root@node0 ~]#
[root@node0 ~]# curl -k -H "x-auth-token: $s" 'http://10.136.16.186/compute/v2.1/servers/detail?sort_key=task_state'
{"computeFault": {"code": 500, "message": "Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.\n<class 'TypeError'>"}}[root@node0 ~]#

Traceback:

Nov 18 09:59:09 node0 <email address hidden>[1156072]: ERROR nova.api.openstack.wsgi [None req-59ce5d12-1c84-4c45-8b10-da863b721d6f demo admin] Unexpected exception in API method: TypeError: '<' not supported between instances of 'NoneType' and 'NoneType'
Nov 18 09:59:09 node0 <email address hidden>[1156072]: ERROR nova.api.openstack.wsgi Traceback (most recent call last):
Nov 18 09:59:09 node0 <email address hidden>[1156072]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 664, in wrapped
Nov 18 09:59:09 node0 <email address hidden>[1156072]: ERROR nova.api.openstack.wsgi return f(*args, **kwargs)
Nov 18 09:59:09 node0 <email address hidden>[1156072]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/api/validation/__init__.py", line 192, in wrapper
Nov 18 09:59:09 node0 <email address hidden>[1156072]: ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
Nov 18 09:59:09 node0 <email address hidden>[1156072]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/api/validation/__init__.py", line 192, in wrapper
Nov 18 09:59:09 node0 <email address hidden>[1156072]: ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
Nov 18 09:59:09 node0 <email address hidden>[1156072]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/api/validation/__init__.py", line 192, in wrapper
Nov 18 09:59:09 node0 <email address hidden>[1156072]: ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
Nov 18 09:59:09 node0 <email address hidden>[1156072]: ERROR nova.api.openstack.wsgi [Previous line repeated 2 more times]
Nov 18 09:59:09 node0 <email address hidden>[1156072]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 143, in detail
Nov 18 09:59:09 node0 <email address hidden>[1156072]: ERROR nova.api.openstack.wsgi servers = self._get_servers(req, is_detail=True)
Nov 18 09:59:09 node0 <email address hidden>[1156072]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 327, in _get_servers
Nov 18 09:59:09 node0 <email address hidden>[1156072]: ERROR nova.api.openstack.wsgi instance_list = self.compute_api.get_all(elevated or context,
Nov 18 09:59:09 node0 <email address hidden>[1156072]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/compute/api.py", line 3140, in get_all
Nov 18 09:59:09 node0 <email address hidden>[1156072]: ERROR nova.api.openstack.wsgi insts, down_cell_uuids = instance_list.get_instance_objects_sorted(
Nov 18 09:59:09 node0 <email address hidden>[1156072]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/compute/instance_list.py", line 176, in get_instance_objects_sorted
Nov 18 09:59:09 node0 <email address hidden>[1156072]: ERROR nova.api.openstack.wsgi instance_list = instance_obj._make_instance_list(ctx,
Nov 18 09:59:09 node0 <email address hidden>[1156072]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/objects/instance.py", line 1287, in _make_instance_list
Nov 18 09:59:09 node0 <email address hidden>[1156072]: ERROR nova.api.openstack.wsgi for db_inst in db_inst_list:
Nov 18 09:59:09 node0 <email address hidden>[1156072]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/compute/multi_cell_list.py", line 411, in get_records_sorted
Nov 18 09:59:09 node0 <email address hidden>[1156072]: ERROR nova.api.openstack.wsgi item = next(feeder)
Nov 18 09:59:09 node0 <email address hidden>[1156072]: ERROR nova.api.openstack.wsgi File "/usr/lib64/python3.9/heapq.py", line 353, in merge
Nov 18 09:59:09 node0 <email address hidden>[1156072]: ERROR nova.api.openstack.wsgi _heapify(h)
Nov 18 09:59:09 node0 <email address hidden>[1156072]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/compute/multi_cell_list.py", line 84, in __lt__
Nov 18 09:59:09 node0 <email address hidden>[1156072]: ERROR nova.api.openstack.wsgi r = self._sort_ctx.compare_records(self._db_record,
Nov 18 09:59:09 node0 <email address hidden>[1156072]: ERROR nova.api.openstack.wsgi File "/opt/stack/nova/nova/compute/multi_cell_list.py", line 45, in compare_records
Nov 18 09:59:09 node0 <email address hidden>[1156072]: ERROR nova.api.openstack.wsgi if rec1[skey] < rec2[skey]:
Nov 18 09:59:09 node0 <email address hidden>[1156072]: ERROR nova.api.openstack.wsgi TypeError: '<' not supported between instances of 'NoneType' and 'NoneType'

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/865037

[sean mooney (sean-k-mooney)]

This is a very old and not well tested part of the nova API.
it was added by https://specs.openstack.org/openstack/nova-specs/specs/kilo/implemented/nova-pagination.html#

I'm not surprised that there are bugs in that feature.

[sean mooney (sean-k-mooney)]

actullly this comes form moving to python3 which was not supported when that was intoduced

https://docs.python.org/release/3.0.1/whatsnew/3.0.html#ordering-comparisons

[Amit Uniyal (auniyal)]

Could not reproduce with latest master, curl did not throw 500.

stack@devstack-amit:~$ openstack server list -f json --long -c Name -c 'Task State' -c Status
[
  {
    "Name": "vm3",
    "Status": "ERROR",
    "Task State": null
  },
  {
    "Name": "vm2",
    "Status": "ACTIVE",
    "Task State": null
  },
  {
    "Name": "vm1",
    "Status": "ERROR",
    "Task State": null
  }
]

================================================

stack@devstack-amit:~$ curl -k -H "x-auth-token: $TT" 'http://10.0.108.24/compute/v2.1/servers/detail?sort_key=task_state'
{"servers": [

{"id": "0ed7bcf3-0419-49b8-8710-dd9306c23977", "name": "vm2", "status": "ACTIVE",

"tenant_id": "44b9b5d2e2994bc8ae77c4aaebf4fc6e", "user_id": "eb09a73c253449fb819fbca70e7686c9", "metadata": {}, "hostId": "029f5e1d2b3b3fec2c2cbb8b7a72302dc04e64e450ed1df802372115", "image": {"id": "dd7e27dc-43ce-48b8-beb0-5438bff59644", "links": [{"rel": "bookmark", "href": "http://10.0.108.24/compute/images/dd7e27dc-43ce-48b8-beb0-5438bff59644"}]}, "flavor": {"id": "3", "links": [{"rel": "bookmark", "href": "http://10.0.108.24/compute/flavors/3"}]}, "created": "2022-12-02T06:58:24Z", "updated": "2022-12-02T06:58:29Z", "addresses": {"public": [{"version": 4, "addr": "172.24.4.48", "OS-EXT-IPS:type": "fixed", "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:cc:f7:9d"}, {"version": 6, "addr": "2001:db8::38d", "OS-EXT-IPS:type": "fixed", "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:cc:f7:9d"}]}, "accessIPv4": "", "accessIPv6": "", "links": [{"rel": "self", "href": "http://10.0.108.24/compute/v2.1/servers/0ed7bcf3-0419-49b8-8710-dd9306c23977"}, {"rel": "bookmark", "href": "http://10.0.108.24/compute/servers/0ed7bcf3-0419-49b8-8710-dd9306c23977"}], "OS-DCF:diskConfig": "MANUAL", "progress": 0, "OS-EXT-AZ:availability_zone": "nova", "config_drive": "", "key_name": null, "OS-SRV-USG:launched_at": "2022-12-02T06:58:28.000000", "OS-SRV-USG:terminated_at": null, "OS-EXT-SRV-ATTR:host": "devstack-amit", "OS-EXT-SRV-ATTR:instance_name": "instance-0000000a", "OS-EXT-SRV-ATTR:hypervisor_hostname": "devstack-amit", "OS-EXT-STS:task_state": null, "OS-EXT-STS:vm_state": "active", "OS-EXT-STS:power_state": 1, "os-extended-volumes:volumes_attached": [], "security_groups": [{"name": "default"}]},

{"id": "166cda20-b609-4cdf-96cd-1c2e029aa080", "name": "vm1", "status": "ERROR",

"tenant_id": "44b9b5d2e2994bc8ae77c4aaebf4fc6e", "user_id": "eb09a73c253449fb819fbca70e7686c9", "metadata": {}, "hostId": "029f5e1d2b3b3fec2c2cbb8b7a72302dc04e64e450ed1df802372115", "image": {"id": "dd7e27dc-43ce-48b8-beb0-5438bff59644", "links": [{"rel": "bookmark", "href": "http://10.0.108.24/compute/images/dd7e27dc-43ce-48b8-beb0-5438bff59644"}]}, "flavor": {"id": "1", "links": [{"rel": "bookmark", "href": "http://10.0.108.24/compute/flavors/1"}]}, "created": "2022-12-01T07:51:25Z", "updated": "2022-12-02T08:46:07Z", "addresses": {"public": [{"version": 4, "addr": "172.24.4.89", "OS-EXT-IPS:type": "fixed", "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:9e:f8:8d"}, {"version": 6, "addr": "2001:db8::2a", "OS-EXT-IPS:type": "fixed", "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:9e:f8:8d"}]}, "accessIPv4": "", "accessIPv6": "", "links": [{"rel": "self", "href": "http://10.0.108....