nova show can not get an instance information, and this instance can be queried from nova list.

[Charlotte Han]

Description
===========
nova delete raise "NO server with a name or ID of 8150bb45-7758-4112-8593-e602b33ab4a5 exists", but this instance is queried from "nova list" command.

Steps to reproduce
==================
A chronological list of steps which will bring off the
issue you noticed:
* I did create some instances, instance with uuid 8150bb45-7758-4112-8593-e602b33ab4a5 is scheduling all the time.
nova list --all | grep scheduling
| 8150bb45-7758-4112-8593-e602b33ab4a5 | test_aa | BUILD | scheduling | NOSTATE |

* then I did delete this instance: nova delete 8150bb45-7758-4112-8593-e602b33ab4a5
NO server with a name or ID of 8150bb45-7758-4112-8593-e602b33ab4a5 exists
Error(commandError): unable to delete this specified server(s).

Expected result
===============
Delete operation can success.

Actual result
=============
Delete operation failed.
Instance information in Database is inconsistent;
I look at database, instance's cell is None, can't see this instance in build_request table.
And this instance's detail information was found in cell0 database.

Environment
===========
1. Exact version of OpenStack you are running. See the following
  list for all releases: http://docs.openstack.org/releases/

 Pike

Logs & Configs
==============
[u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 160, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch\n result = func(ctxt, **new_args)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 232, in inner\n return func(*args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py", line 153, in select_destinations\n alloc_reqs_by_rp_uuid, provider_summaries)\n', u' File "/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 89, in select_destinations\n alloc_reqs_by_rp_uuid, provider_summaries)\n', u' File "/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 163, in _schedule\n provider_summaries)\n', u' File "/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 344, in _get_all_host_states\n spec_obj)\n', u' File "/usr/lib/python2.7/site-packages/nova/scheduler/host_manager.py", line 659, in get_host_states_by_uuids\n context, cells, compute_uuids=compute_uuids)\n', u' File "/usr/lib/python2.7/site-packages/nova/scheduler/host_manager.py", line 627, in _get_computes_for_cells\n cctxt, compute_uuids))\n', u' File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper\n result = fn(cls, context, *args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/nova/objects/compute_node.py", line 460, in get_all_by_uuids\n compute_uuids)\n', u' File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 235, in wrapper\n return f(*args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/nova/objects/compute_node.py", line 454, in _db_compute_node_get_all_by_uuids\n models.ComputeNode.uuid.in_(compute_uuids)).all()\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2703, in all\n return list(self)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2855, in __iter__\n return self._execute_and_instances(context)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2876, in _execute_and_instances\n close_with_result=True)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2885, in _get_bind_args\n **kw\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2867, in _connection_from_session\n conn = self.session.connection(**kw)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 998, in connection\n execution_options=execution_options)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1003, in _connection_for_bind\n engine, execution_options)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 403, in _connection_for_bind\n conn = bind.contextual_connect()\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2112, in contextual_connect\n self._wrap_pool_connect(self.pool.connect, None),\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2151, in _wrap_pool_connect\n e, dialect, self)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1461, in _handle_dbapi_exception_noconnection\n util.raise_from_cause(newraise, exc_info)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause\n reraise(type(exception), exception, tb=exc_tb, cause=cause)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2147, in _wrap_pool_connect\n return fn()\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 387, in connect\n return _ConnectionFairy._checkout(self)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 766, in _checkout\n fairy = _ConnectionRecord.checkout(pool)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 516, in checkout\n rec = pool._do_get()\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1138, in _do_get\n self._dec_overflow()\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__\n compat.reraise(exc_type, exc_value, exc_tb)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1135, in _do_get\n return self._create_connection()\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 333, in _create_connection\n return _ConnectionRecord(self)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 461, in __init__\n self.__connect(first_connect_check=True)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 651, in __connect\n connection = pool._invoke_creator(self)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect\n return dialect.connect(*cargs, **cparams)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 393, in connect\n return self.dbapi.connect(*cargs, **cparams)\n', u' File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect\n return Connection(*args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__\n self.connect()\n', u' File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 932, in connect\n self._request_authentication()\n', u' File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1152, in _request_authentication\n auth_packet = self._read_packet()\n', u' File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1014, in _read_packet\n packet.check_error()\n', u' File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 393, in check_error\n err.raise_mysql_exception(self._data)\n', u' File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\n', u"OperationalError: (pymysql.err.OperationalError) (1040, u'Too many connections')\n"].
2018-01-30 06:57:52.850 3670 WARNING nova.scheduler.utils [req-6f3db77c-6b4b-4525-b8d2-702dedb626b3 9ca3661a7c594f1ba7848ef69ce2ca04 9e1cb8200d164336be675deeaad131b3 - - -] [instance: 8150bb45-7758-4112-8593-e602b33ab4a5] Setting instance to ERROR state.: RemoteError: Remote error: OperationalError (pymysql.err.OperationalError) (1040, u'Too many connections')

[Sylvain Bauza (sylvain-bauza)]

Instances are only created in cell databases once they are scheduled to a host, so we can know which specific cell DB to call for a new instance record.
When the scheduler passes a NoValidHost, then given there wouldn't be a host for that instance, the instance record is rather created to a specific cell0 database.

See https://docs.openstack.org/nova/latest/user/cellsv2-layout.html#concepts for more explanations.

Now, it looks like you had a problem with your DB given the above log. When the scheduler tried to look at all the compute nodes, it got a SQL exception hence why it didn't provided a valid host.
Looks to me that specific problem created some issue hence why you weren't possible to delete that instance.

Can you please try to reproduce that problem ?

[Matt Riedemann (mriedem)]

Which version of Pike are you running? The GA 16.0.0 or the latest fix release 16.0.4?

[Sylvain Bauza (sylvain-bauza)]

Setting to Incomplete. Once you reply, please set the status back to 'New'. Thanks.

[Charlotte Han (hanrong)]

pike 16.0.2.

Sorry, I have not reproduce this problem. This log is sent back by the client.

I want information on DB between nova-api and cell0 is coincident.

[Sylvain Bauza (sylvain-bauza)]

You really need to provide more details about the specific issue you had because at first glance, it looks like an operational problem with your DB :

Setting instance to ERROR state.: RemoteError: Remote error: OperationalError (pymysql.err.OperationalError) (1040, u'Too many connections')

[Launchpad Janitor (janitor)]

[Expired for OpenStack Compute (nova) because there has been no activity for 60 days.]