Comment 1 for bug 1714433

Revision history for this message
tianhui (tianhui) wrote :

I log in the instance,there is also a error in the log:
2017-09-01 02:00:18.175 935 DEBUG trove.guestagent.datastore.experimental.mongodb.service [-] Disconnecting from MongoDB. __exit__ /usr/local/lib/python2.7/dist-packages/trove/guestagent/datastore/experimental/mongodb/service.py:823
2017-09-01 02:00:18.176 935 ERROR trove.guestagent.datastore.service [-] Timeout while waiting for database status to change.Expected state running, current state is shutdown
2017-09-01 02:00:18.177 935 INFO trove.guestagent.datastore.service [-] Service status did not change to running within the given timeout: 180s
2017-09-01 02:00:18.177 935 DEBUG trove.guestagent.datastore.service [-] Attempting to cleanup stalled services. _wait_for_database_service_status /usr/local/lib/python2.7/dist-packages/trove/guestagent/datastore/service.py:343
2017-09-01 02:00:18.178 935 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): ps xaco pid,cmd | awk '/mongo(d|db|s)/ {print $1}' execute /usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:344
2017-09-01 02:00:18.244 935 DEBUG oslo_concurrency.processutils [-] CMD "ps xaco pid,cmd | awk '/mongo(d|db|s)/ {print $1}'" returned: 0 in 0.066s execute /usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:374
2017-09-01 02:00:18.246 935 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): sudo kill execute /usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:344
2017-09-01 02:00:18.274 935 DEBUG oslo_concurrency.processutils [-] CMD "sudo kill " returned: 1 in 0.028s execute /usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:374
2017-09-01 02:00:18.275 935 DEBUG oslo_concurrency.processutils [-] u'sudo kill ' failed. Not Retrying. execute /usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:422
2017-09-01 02:00:18.276 935 DEBUG trove.guestagent.datastore.service [-] Cleanup failed. _wait_for_database_service_status /usr/local/lib/python2.7/dist-packages/trove/guestagent/datastore/service.py:347
2017-09-01 02:00:18.276 935 ERROR trove.guestagent.datastore.service Traceback (most recent call last):
2017-09-01 02:00:18.276 935 ERROR trove.guestagent.datastore.service File "/usr/local/lib/python2.7/dist-packages/trove/guestagent/datastore/service.py", line 345, in _wait_for_database_service_status
2017-09-01 02:00:18.276 935 ERROR trove.guestagent.datastore.service self.cleanup_stalled_db_services()
2017-09-01 02:00:18.276 935 ERROR trove.guestagent.datastore.service File "/usr/local/lib/python2.7/dist-packages/trove/guestagent/datastore/experimental/mongodb/service.py", line 462, in cleanup_stalled_db_services
2017-09-01 02:00:18.276 935 ERROR trove.guestagent.datastore.service utils.execute_with_timeout(system.MONGODB_KILL % pid, shell=True)
2017-09-01 02:00:18.276 935 ERROR trove.guestagent.datastore.service File "/usr/local/lib/python2.7/dist-packages/trove/common/utils.py", line 239, in execute_with_timeout
2017-09-01 02:00:18.276 935 ERROR trove.guestagent.datastore.service return execute(*args, **kwargs)
2017-09-01 02:00:18.276 935 ERROR trove.guestagent.datastore.service File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 389, in execute
2017-09-01 02:00:18.276 935 ERROR trove.guestagent.datastore.service cmd=sanitized_cmd)
2017-09-01 02:00:18.276 935 ERROR trove.guestagent.datastore.service ProcessExecutionError: Unexpected error while running command.
2017-09-01 02:00:18.276 935 ERROR trove.guestagent.datastore.service Command: sudo kill
2017-09-01 02:00:18.276 935 ERROR trove.guestagent.datastore.service Exit code: 1
2017-09-01 02:00:18.276 935 ERROR trove.guestagent.datastore.service Stdout: u''
2017-09-01 02:00:18.276 935 ERROR trove.guestagent.datastore.service Stderr: u'\nUsage:\n kill [options] <pid> [...]\n\nOptions:\n <pid> [...] send signal to every <pid> listed\n -<signal>, -s, --signal <signal>\n specify the <signal> to be sent\n -l, --list=[<signal>] list all signal names, or convert one to a name\n -L, --table list all signal names in a nice table\n\n -h, --help display this help and exit\n -V, --version output version information and exit\n\nFor more details see kill(1).\n'
2017-09-01 02:00:18.276 935 ERROR trove.guestagent.datastore.service
2017-09-01 02:00:18.321 935 DEBUG trove.guestagent.datastore.service [-] Prepare has not completed yet, skipping heartbeat. set_status /usr/local/lib/python2.7/dist-packages/trove/guestagent/datastore/service.py:193
2017-09-01 02:00:18.322 935 ERROR oslo_messaging.rpc.dispatcher [-] Exception during message handling: Database failed to start.
2017-09-01 02:00:18.322 935 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2017-09-01 02:00:18.322 935 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply
2017-09-01 02:00:18.322 935 ERROR oslo_messaging.rpc.dispatcher incoming.message))
2017-09-01 02:00:18.322 935 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch
2017-09-01 02:00:18.322 935 ERROR oslo_messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2017-09-01 02:00:18.322 935 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch
2017-09-01 02:00:18.322 935 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2017-09-01 02:00:18.322 935 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 117, in wrapper
2017-09-01 02:00:18.322 935 ERROR oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2017-09-01 02:00:18.322 935 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/trove/guestagent/datastore/experimental/mongodb/manager.py", line 225, in add_config_servers
2017-09-01 02:00:18.322 935 ERROR oslo_messaging.rpc.dispatcher self.app.add_config_servers(config_servers)
2017-09-01 02:00:18.322 935 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/trove/guestagent/datastore/experimental/mongodb/service.py", line 263, in add_config_servers
2017-09-01 02:00:18.322 935 ERROR oslo_messaging.rpc.dispatcher self.start_db(True)
2017-09-01 02:00:18.322 935 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/trove/guestagent/datastore/experimental/mongodb/service.py", line 94, in start_db
2017-09-01 02:00:18.322 935 ERROR oslo_messaging.rpc.dispatcher enable_on_boot=True, update_db=update_db)
2017-09-01 02:00:18.322 935 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/trove/guestagent/datastore/service.py", line 266, in start_db_service
2017-09-01 02:00:18.322 935 ERROR oslo_messaging.rpc.dispatcher self.wait_for_database_service_start(timeout, update_db=update_db)
2017-09-01 02:00:18.322 935 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/trove/guestagent/datastore/service.py", line 286, in wait_for_database_service_start
2017-09-01 02:00:18.322 935 ERROR oslo_messaging.rpc.dispatcher raise RuntimeError(_("Database failed to start."))
2017-09-01 02:00:18.322 935 ERROR oslo_messaging.rpc.dispatcher RuntimeError: Database failed to start.