could not list node sorted by "created_at:desc"

[Lawrance]

HOW TO REPRODUCE:
senlin node-list -o "created_at:desc"

[stack@localhost ~]$ senlin node-list -o "created_at:desc"
WARNING (shell) "senlin node-list" is deprecated, please use "openstack cluster node list" instead.
ERROR(500): Remote error: ArgumentError Only '=', '!=', 'is_()', 'isnot()' operators can be used with None/True/False
[u'

############Senlin api log##################
2016-06-13 21:24:01.357 ERROR senlin.api.common.wsgi [req-cad884cd-d7e3-4054-bbdf-da2f31706ee9 demo demo] Unexpected error occurred serving API: Remote error: Remote error: ArgumentError Only '=', '!=', 'is_()', 'isnot()' operators can be used with None/True/False
[u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply\n incoming.message))\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch\n result = func(ctxt, **new_args)\n', u' File "/opt/stack/senlin/senlin/engine/service.py", line 68, in wrapped\n return func(self, ctx, *args, **kwargs)\n', u' File "/opt/stack/senlin/senlin/engine/service.py", line 1370, in node_list\n project_safe=project_safe)\n', u' File "/opt/stack/senlin/senlin/engine/node.py", line 180, in load_all\n project_safe=project_safe)\n', u' File "/opt/stack/senlin/senlin/objects/node.py", line 67, in get_all\n objs = db_api.node_get_all(context, **kwargs)\n', u' File "/opt/stack/senlin/senlin/db/api.py", line 102, in node_get_all\n project_safe=project_safe)\n', u' File "/opt/stack/senlin/senlin/db/sqlalchemy/api.py", line 270, in node_get_all\n marker=marker, sort_dirs=dirs).all()\n', u' File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/utils.py", line 168, in paginate_query\n crit_attrs.append((model_attr < marker_values[i]))\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/operators.py", line 282, in __lt__\n return self.operate(lt, other)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/attributes.py", line 175, in operate\n return op(self.comparator, *other, **kwargs)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/operators.py", line 282, in __lt__\n return self.operate(lt, other)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/properties.py", line 269, in operate\n return op(self.__clause_element__(), *other, **kwargs)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/operators.py", line 282, in __lt__\n return self.operate(lt, other)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 739, in operate\n return op(self.comparator, *other, **kwargs)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/operators.py", line 282, in __lt__\n return self.operate(lt, other)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/type_api.py", line 735, in operate\n op, *other, **kwargs)\n', u' File "<string>", line 1, in <lambda>\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/type_api.py", line 60, in operate\n return o[0](self.expr, op, *(other + o[1:]), **kwargs)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/default_comparator.py", line 53, in _boolean_compare\n "Only \'=\', \'!=\', \'is_()\', \'isnot()\' operators can "\n', u"ArgumentError: Only '=', '!=', 'is_()', 'isnot()' operators can be used with None/True/False\n"]. None
None.
2016-06-13 21:24:01.357 TRACE senlin.api.common.wsgi Traceback (most recent call last):
2016-06-13 21:24:01.357 TRACE senlin.api.common.wsgi File "/opt/stack/senlin/senlin/api/common/wsgi.py", line 645, in __call__
2016-06-13 21:24:01.357 TRACE senlin.api.common.wsgi request, **action_args)
2016-06-13 21:24:01.357 TRACE senlin.api.common.wsgi File "/opt/stack/senlin/senlin/api/common/wsgi.py", line 715, in dispatch
2016-06-13 21:24:01.357 TRACE senlin.api.common.wsgi return method(*args, **kwargs)
2016-06-13 21:24:01.357 TRACE senlin.api.common.wsgi File "/opt/stack/senlin/senlin/api/common/util.py", line 39, in policy_checker
2016-06-13 21:24:01.357 TRACE senlin.api.common.wsgi return handler(controller, req, **kwargs)
2016-06-13 21:24:01.357 TRACE senlin.api.common.wsgi File "/opt/stack/senlin/senlin/api/openstack/v1/nodes.py", line 101, in index
2016-06-13 21:24:01.357 TRACE senlin.api.common.wsgi **params)
2016-06-13 21:24:01.357 TRACE senlin.api.common.wsgi File "/opt/stack/senlin/senlin/rpc/client.py", line 218, in node_list
2016-06-13 21:24:01.357 TRACE senlin.api.common.wsgi project_safe=project_safe))
2016-06-13 21:24:01.357 TRACE senlin.api.common.wsgi File "/opt/stack/senlin/senlin/rpc/client.py", line 44, in call
2016-06-13 21:24:01.357 TRACE senlin.api.common.wsgi return client.call(ctxt, method, **kwargs)
2016-06-13 21:24:01.357 TRACE senlin.api.common.wsgi File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 413, in call
2016-06-13 21:24:01.357 TRACE senlin.api.common.wsgi return self.prepare().call(ctxt, method, **kwargs)
2016-06-13 21:24:01.357 TRACE senlin.api.common.wsgi File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 158, in call
2016-06-13 21:24:01.357 TRACE senlin.api.common.wsgi retry=self.retry)
2016-06-13 21:24:01.357 TRACE senlin.api.common.wsgi File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
2016-06-13 21:24:01.357 TRACE senlin.api.common.wsgi timeout=timeout, retry=retry)
2016-06-13 21:24:01.357 TRACE senlin.api.common.wsgi File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 470, in send
2016-06-13 21:24:01.357 TRACE senlin.api.common.wsgi retry=retry)
2016-06-13 21:24:01.357 TRACE senlin.api.common.wsgi File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 461, in _send
2016-06-13 21:24:01.357 TRACE senlin.api.common.wsgi raise result
2016-06-13 21:24:01.357 TRACE senlin.api.common.wsgi RemoteError: Remote error: ArgumentError Only '=', '!=', 'is_()', 'isnot()' operators can be used with None/True/False
2016-06-13 21:24:01.357 TRACE senlin.api.common.wsgi [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply\n incoming.message))\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch\n result = func(ctxt, **new_args)\n', u' File "/opt/stack/senlin/senlin/engine/service.py", line 68, in wrapped\n return func(self, ctx, *args, **kwargs)\n', u' File "/opt/stack/senlin/senlin/engine/service.py", line 1370, in node_list\n project_safe=project_safe)\n', u' File "/opt/stack/senlin/senlin/engine/node.py", line 180, in load_all\n project_safe=project_safe)\n', u' File "/opt/stack/senlin/senlin/objects/node.py", line 67, in get_all\n objs = db_api.node_get_all(context, **kwargs)\n', u' File "/opt/stack/senlin/senlin/db/api.py", line 102, in node_get_all\n project_safe=project_safe)\n', u' File "/opt/stack/senlin/senlin/db/sqlalchemy/api.py", line 270, in node_get_all\n marker=marker, sort_dirs=dirs).all()\n', u' File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/utils.py", line 168, in paginate_query\n crit_attrs.append((model_attr < marker_values[i]))\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/operators.py", line 282, in __lt__\n return self.operate(lt, other)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/attributes.py", line 175, in operate\n return op(self.comparator, *other, **kwargs)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/operators.py", line 282, in __lt__\n return self.operate(lt, other)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/properties.py", line 269, in operate\n return op(self.__clause_element__(), *other, **kwargs)\n', u' File "/usr/lib64/pyth
on2.7/site-packages/sqlalchemy/sql/operators.py", line 282, in __lt__\n return self.operate(lt, other)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 739, in operate\n return op(self.comparator, *other, **kwargs)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/operators.py", line 282, in __lt__\n return self.operate(lt, other)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/type_api.py", line 735, in operate\n op, *other, **kwargs)\n', u' File "<string>", line 1, in <lambda>\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/type_api.py", line 60, in operate\n return o[0](self.expr, op, *(other + o[1:]), **kwargs)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/default_comparator.py", line 53, in _boolean_compare\n "Only \'=\', \'!=\', \'is_()\', \'isnot()\' operators can "\n', u"ArgumentError: Only '=', '!=', 'is_()', 'isnot()' operators can be used with None/True/False\n"].
2016-06-13 21:24:01.357 TRACE senlin.api.common.wsgi

##############Senlin engine log####################
2016-06-13 21:24:57.178 ERROR oslo_messaging.rpc.dispatcher [req-775d248b-adc1-4b25-8567-9632eedb076f demo demo] Exception during message handling: Only '=', '!=', 'is_()', 'isnot()' operators can be used with None/True/False
2016-06-13 21:24:57.178 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-06-13 21:24:57.178 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply
2016-06-13 21:24:57.178 TRACE oslo_messaging.rpc.dispatcher incoming.message))
2016-06-13 21:24:57.178 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch
2016-06-13 21:24:57.178 TRACE oslo_messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2016-06-13 21:24:57.178 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch
2016-06-13 21:24:57.178 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2016-06-13 21:24:57.178 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/senlin/senlin/engine/service.py", line 68, in wrapped
2016-06-13 21:24:57.178 TRACE oslo_messaging.rpc.dispatcher return func(self, ctx, *args, **kwargs)
2016-06-13 21:24:57.178 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/senlin/senlin/engine/service.py", line 1370, in node_list
2016-06-13 21:24:57.178 TRACE oslo_messaging.rpc.dispatcher project_safe=project_safe)
2016-06-13 21:24:57.178 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/senlin/senlin/engine/node.py", line 180, in load_all
2016-06-13 21:24:57.178 TRACE oslo_messaging.rpc.dispatcher project_safe=project_safe)
2016-06-13 21:24:57.178 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/senlin/senlin/objects/node.py", line 67, in get_all
2016-06-13 21:24:57.178 TRACE oslo_messaging.rpc.dispatcher objs = db_api.node_get_all(context, **kwargs)
2016-06-13 21:24:57.178 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/senlin/senlin/db/api.py", line 102, in node_get_all
2016-06-13 21:24:57.178 TRACE oslo_messaging.rpc.dispatcher project_safe=project_safe)
2016-06-13 21:24:57.178 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/senlin/senlin/db/sqlalchemy/api.py", line 270, in node_get_all
2016-06-13 21:24:57.178 TRACE oslo_messaging.rpc.dispatcher marker=marker, sort_dirs=dirs).all()
2016-06-13 21:24:57.178 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/utils.py", line 168, in paginate_query
2016-06-13 21:24:57.178 TRACE oslo_messaging.rpc.dispatcher crit_attrs.append((model_attr < marker_values[i]))
2016-06-13 21:24:57.178 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/operators.py", line 282, in __lt__
2016-06-13 21:24:57.178 TRACE oslo_messaging.rpc.dispatcher return self.operate(lt, other)
2016-06-13 21:24:57.178 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/attributes.py", line 175, in operate
2016-06-13 21:24:57.178 TRACE oslo_messaging.rpc.dispatcher return op(self.comparator, *other, **kwargs)
2016-06-13 21:24:57.178 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/operators.py", line 282, in __lt__
2016-06-13 21:24:57.178 TRACE oslo_messaging.rpc.dispatcher return self.operate(lt, other)
2016-06-13 21:24:57.178 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/properties.py", line 269, in operate
2016-06-13 21:24:57.178 TRACE oslo_messaging.rpc.dispatcher return op(self.__clause_element__(), *other, **kwargs)
2016-06-13 21:24:57.178 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/operators.py", line 282, in __lt__
2016-06-13 21:24:57.178 TRACE oslo_messaging.rpc.dispatcher return self.operate(lt, other)
2016-06-13 21:24:57.178 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 739, in operate
2016-06-13 21:24:57.178 TRACE oslo_messaging.rpc.dispatcher return op(self.comparator, *other, **kwargs)
2016-06-13 21:24:57.178 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/operators.py", line 282, in __lt__
2016-06-13 21:24:57.178 TRACE oslo_messaging.rpc.dispatcher return self.operate(lt, other)
2016-06-13 21:24:57.178 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/type_api.py", line 735, in operate
2016-06-13 21:24:57.178 TRACE oslo_messaging.rpc.dispatcher op, *other, **kwargs)
2016-06-13 21:24:57.178 TRACE oslo_messaging.rpc.dispatcher File "<string>", line 1, in <lambda>
2016-06-13 21:24:57.178 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/type_api.py", line 60, in operate
2016-06-13 21:24:57.178 TRACE oslo_messaging.rpc.dispatcher return o[0](self.expr, op, *(other + o[1:]), **kwargs)
2016-06-13 21:24:57.178 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/default_comparator.py", line 53, in _boolean_compare
2016-06-13 21:24:57.178 TRACE oslo_messaging.rpc.dispatcher "Only '=', '!=', 'is_()', 'isnot()' operators can "
2016-06-13 21:24:57.178 TRACE oslo_messaging.rpc.dispatcher ArgumentError: Only '=', '!=', 'is_()', 'isnot()' operators can be used with None/True/False
2016-06-13 21:24:57.178 TRACE oslo_messaging.rpc.dispatcher
2016-06-13 21:24:57.180 ERROR oslo_messaging._drivers.common [req-775d248b-adc1-4b25-8567-9632eedb076f demo demo] Returning exception Only '=', '!=', 'is_()', 'isnot()' operators can be used with None/True/False to caller
2016-06-13 21:24:57.180 ERROR oslo_messaging._drivers.common [req-775d248b-adc1-4b25-8567-9632eedb076f demo demo] ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply\n incoming.message))\n', ' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/opt/stack/senlin/senlin/engine/service.py", line 68, in wrapped\n return func(self, ctx, *args, **kwargs)\n', ' File "/opt/stack/senlin/senlin/engine/service.py", line 1370, in node_list\n project_safe=project_safe)\n', ' File "/opt/stack/senlin/senlin/engine/node.py", line 180, in load_all\n project_safe=project_safe)\n', ' File "/opt/stack/senlin/senlin/objects/node.py", line 67, in get_all\n objs = db_api.node_get_all(context, **kwargs)\n', ' File "/opt/stack/senlin/senlin/db/api.py", line 102, in node_get_all\n project_safe=project_safe)\n', ' File "/opt/stack/senlin/senlin/db/sqlalchemy/api.py", line 270, in node_get_all\n marker=marker, sort_dirs=dirs).all()\n', ' File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/utils.py", line 168, in paginate_query\n crit_attrs.append((model_attr < marker_values[i]))\n', ' File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/operators.py", line 282, in __lt__\n return self.operate(lt, other)\n', ' File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/attributes.py", line 175, in operate\n return op(self.comparator, *other, **kwargs)\n', ' File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/operators.py", line 282, in __lt__\n return self.operate(lt, other)\n', ' File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/properties.py", line 269, in operate\n return op(self.__clause_element__(), *other, **kwargs)\n', ' File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/operators.py", line 282, in __lt__\n return self.operate(lt, other)\n', ' File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 739, in operate\n return op(self.comparator, *other, **kwargs)\n', ' File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/operators.py", line 282, in __lt__\n return self.operate(lt, other)\n', ' File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/type_api.py", line 735, in operate\n op, *other, **kwargs)\n', ' File "<string>", line 1, in <lambda>\n', ' File "
/usr/lib64/python2.7/site-packages/sqlalchemy/sql/type_api.py", line 60, in operate\n return o[0](self.expr, op, *(other + o[1:]), **kwargs)\n', ' File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/default_comparator.py", line 53, in _boolean_compare\n "Only \'=\', \'!=\', \'is_()\', \'isnot()\' operators can "\n', "ArgumentError: Only '=', '!=', 'is_()', 'isnot()' operators can be used with None/True/False\n"]
2016-06-13 21:24:57.180 DEBUG oslo_messaging._drivers.amqpdriver [req-775d248b-adc1-4b25-8567-9632eedb076f demo demo] sending reply msg_id: 5a18a6695eab4da0b16e57daf4c5d16d reply queue: reply_81fc1dafc46642c998e258580108664a time elapsed: 0.0104696810013s from (pid=12285) _send_reply /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:74

[Yanyan Hu (yanyanhu)]

hi, Lawrance, just made a test and didn't meet this error. Did you test using the latest code? Thanks.

[Lawrance (jing)]

HI Yanyan, will try the latest code asap:-)
thanks

[Lawrance (jing)]

Please use latest code

[Yanyan Hu (yanyanhu)]

This happened when created_at value of node is None. May need a fix here.

[Yanyan Hu (yanyanhu)]

Made some investigation and found it is difficult to fix this issue unless sqlalchemy supports comparison operation for 'None' value. An alternative is giving created_at, updated_at a default value which is not None. But that doesn't make much sense since created_at/updated_at should not be given a valid datatime value before cluster creation/updating is really performed. Need more thinking here.

[Yanyan Hu (yanyanhu)]

looks I was wrong. Keeping digging this issue.

[Yanyan Hu (yanyanhu)]

looks like "-nullslast" suffix for sort_dir doesn't work correctly.

[Yanyan Hu (yanyanhu)]

The reason is the second 'list' request which started by sdk is performed with incorrect 'marker' given when '-nullslast' suffix is applied for sorting. Using raw Senlin API request, this problem won't happen. May need a fix in SDK side.

[Yanyan Hu (yanyanhu)]

Summarize the reason:

Currently, Senlinclient relies on openstacksdk to talk with Senlin API service. The implementation of node list in senlinclient is as following:
http://git.openstack.org/cgit/openstack/python-senlinclient/tree/senlinclient/v1/shell.py#n896

The invoked method in openstacksdk is:
http://git.openstack.org/cgit/openstack/python-openstacksdk/tree/openstack/cluster/v1/_proxy.py#n482

Then the following resource call in sdk will be invoked:
http://git.openstack.org/cgit/openstack/python-openstacksdk/tree/openstack/resource.py#n843

we can see in 'list' method, at least two list API request will be performed if "paginated" is set to True and query "limit" is given and is not broken in first query.

So, in second query, the "marker" will be set to the ID of last resource object returned in resp of the first API request. So the second list API request will be performed with both "sort" and "marker". That why we saw the following debug message when perform "senlin node-list -o 'updated_at:asc'" cmd.

http://paste.openstack.org/show/525430/

We can see the "marker" is 541ed8d2-f748-47c4-a234-45716aa472e8 in the second list API request. It is the ID of the last node object(node-004) returned in resp of the first API request.

For sorting with option "updated_at:desc", the progress is the same. But in that case, the last node object will be node-03. Therefore, the second list API request will be performed with marker '2bb8d9ac-abab-487b-a55f-e099d8a1d1cd'. However, when Senlin service accepted this request and invoked query method of sqlalchemy lib, it found the "updated_at" attr of given marker node object is "NULL". This value is illegal for comparison operators. Therefore, the following exception will be raised and failed the second list API request:

ArgumentError: Only '=', '!=', 'is_()', 'isnot()' operators can be used with None/True/False

So this is the exact reason for this bug. And the same problem exists for other Senlin resources, like cluster, profile.

One possible way to fix it is revising the "list" resource call of sdk to make it handle "marker" more smartly. But not sure this can be implemented in generic way since different API service could have different definition for query option(e.g. Senlin API support "sort" which is a mixture of "sort_dir" and "keys" options which are supported by Heat API). An alternative is overriding "list" resource call of Senlin resources. Anyway, this is not the fault of Senlin API in my mind. It is just a very trick problem of sdk.

[Yanyan Hu (yanyanhu)]

Hi, Qiming, any idea to fix this issue?

[Qiming Teng (tengqim)]

Related bug report: https://launchpad.net/bugs/1615938
Fix proposed to oslo.db: https://review.openstack.org/#/c/358410

[milan k (vetrisko)]

Hi guys,

unfortunately this bug affect me as well; TL;DR I'm trying to implement pagination in Ironic Inspector with sorting based on started_at and finished_at attributes, with newer items first (desc-nullsfirst).

[milan k (vetrisko)]

Hi,

in my case if marker.finished_at is None (e.g introspection didn't finish for that node yet) and since I have a fixed descending, nulls-first order, I was able to workaround the issue by sanitizing the marker:

    if marker.finished_at is None:
        marker.finished_at = float('inf')

Since I'm dealing with time values here, I would use 0.0 in case of ascending order.
HTH.

Cheers,
milan

[Qiming Teng (tengqim)]

with https://review.openstack.org/358410 merged, hopefully this issue is solved.