After restarting database, the first command will fail. for example:
after restarting Database, and wait for a few minutes.
Then run heat stack-list, result will be like below:
ERROR: Remote error: DBConnectionError (OperationalError) ibm_db_dbi::OperationalError: SQLNumResultCols failed: [IBM][CLI Driver] SQL30081N A communication error has been detected. Communication protocol being used: "TCP/IP". Communication API being used: "SOCKETS". Location where the error was detected: "10.11.1.14". Communication function detecting the error: "send". Protocol specific error code(s): "2", "*", "*". SQLSTATE=08001 SQLCODE=-30081 'SELECT stack.status_reason AS stack_status_reason, stack.created_at AS stack_created_at, stack.deleted_at AS stack_deleted_at, stack.action AS stack_action, stack.status AS stack_status, stack.id AS stack_id, stack.name AS stack_name, stack.raw_template_id AS stack_raw_template_id, stack.username AS stack_username, stack.tenant AS stack_tenant, stack.parameters AS stack_parameters, stack.user_creds_id AS stack_user_creds_id, stack.owner_id AS stack_owner_id, stack.timeout AS stack_timeout, stack.disable_rollback AS stack_disable_rollback, stack.stack_user_project_id AS stack_stack_user_project_id, stack.backup AS stack_backup, stack.updated_at AS stack_updated_at \nFROM stack \nWHERE stack.deleted_at IS NULL AND stack.owner_id IS NULL AND stack.tenant = ? ORDER BY stack.created_at DESC, stack.id DESC' ('a3a14c6f82bd4ce88273822407a0829b',)
[u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply\n incoming.message))\n', u' File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', u' File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch\n result = getattr(endpoint, method)(ctxt, **new_args)\n', u' File "/usr/lib/python2.6/site-packages/heat/engine/service.py", line 69, in wrapped\n return func(self, ctx, *args, **kwargs)\n', u' File "/usr/lib/python2.6/site-packages/heat/engine/service.py", line 490, in list_stacks\n return [api.format_stack(stack) for stack in stacks]\n', u' File "/usr/lib/python2.6/site-packages/heat/engine/stack.py", line 264, in load_all\n show_deleted, show_nested) or []\n', u' File "/usr/lib/python2.6/site-packages/heat/db/api.py", line 130, in stack_get_all\n show_deleted, show_nested)\n', u' File "/usr/lib/python2.6/site-packages/heat/db/sqlalchemy/api.py", line 368, in stack_get_all\n marker, sort_dir, filters).all()\n', u' File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2241, in all\n return list(self)\n', u' File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2353, in __iter__\n return self._execute_and_instances(context)\n', u' File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2368, in _execute_and_instances\n result = conn.execute(querycontext.statement, self._params)\n', u' File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 662, in execute\n params)\n', u' File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 761, in _execute_clauseelement\n compiled_sql, distilled_params\n', u' File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 874, in _execute_context\n context)\n', u' File "/usr/lib/python2.6/site-packages/oslo/db/sqlalchemy/compat/handle_error.py", line 125, in _handle_dbapi_exception\n six.reraise(type(newraise), newraise, sys.exc_info()[2])\n', u' File "/usr/lib/python2.6/site-packages/oslo/db/sqlalchemy/compat/handle_error.py", line 102, in _handle_dbapi_exception\n per_fn = fn(ctx)\n', u' File "/usr/lib/python2.6/site-packages/oslo/db/sqlalchemy/exc_filters.py", line 323, in handler\n context.is_disconnect)\n', u' File "/usr/lib/python2.6/site-packages/oslo/db/sqlalchemy/exc_filters.py", line 263, in _is_db_connection_error\n raise exception.DBConnectionError(operational_error)\n', u'DBConnectionError: (OperationalError) ibm_db_dbi::OperationalError: SQLNumResultCols failed: [IBM][CLI Driver] SQL30081N A communication error has been detected. Communication protocol being used: "TCP/IP". Communication API being used: "SOCKETS". Location where the error was detected: "10.11.1.14". Communication function detecting the error: "send". Protocol specific error code(s): "2", "*", "*". SQLSTATE=08001 SQLCODE=-30081 \'SELECT stack.status_reason AS stack_status_reason, stack.created_at AS stack_created_at, stack.deleted_at AS stack_deleted_at, stack.action AS stack_action, stack.status AS stack_status, stack.id AS stack_id, stack.name AS stack_name, stack.raw_template_id AS stack_raw_template_id, stack.username AS stack_username, stack.tenant AS stack_tenant, stack.parameters AS stack_parameters, stack.user_creds_id AS stack_user_creds_id, stack.owner_id AS stack_owner_id, stack.timeout AS stack_timeout, stack.disable_rollback AS stack_disable_rollback, stack.stack_user_project_id AS stack_stack_user_project_id, stack.backup AS stack_backup, stack.updated_at AS stack_updated_at \\nFROM stack \\nWHERE stack.deleted_at IS NULL AND stack.owner_id IS NULL AND stack.tenant = ? ORDER BY stack.created_at DESC, stack.id DESC\' (\'a3a14c6f82bd4ce88273822407a0829b\',)\n'].
then run heat stack-list or other command, the command will be ok.
Seems like this is an oslo.db issue, not specific to any of its consumers.