Brief Description
-----------------
Playbook replay caused password regeneration - system returns: Authorization failed: Internal Server Error (HTTP 500)
[sysadmin@controller-0 ~(keystone_admin)]$ system --debug host-list
DEBUG (base:187) Making authentication request to http://[abde::2]:5000/v3/auth/tokens
DEBUG (connectionpool:207) Starting new HTTP connection (1): abde::2
DEBUG (connectionpool:395) http://abde::2:5000 "POST /v3/auth/tokens HTTP/1.1" 500 141
DEBUG (session:442) Request returned failure status: 500
Authorization failed: Internal Server Error (HTTP 500)
[sysadmin@controller-0 ~(keystone_admin)]$
2019-08-01 18:29:23.562 3671518 INFO barbican.cmd.keystone_listener [-] Booting up Barbican Keystone listener node...
2019-08-01 18:29:23.643 3671518 CRITICAL barbican [-] Unhandled error: BarbicanException: Error configuring registry database with supplied sql_connection. Got error: (psycopg2.OperationalError) FATAL: password authentication failed for user "admin-barbican"
2019-08-01 18:29:23.643 3671518 ERROR barbican Traceback (most recent call last):
2019-08-01 18:29:23.643 3671518 ERROR barbican File "/bin/barbican-keystone-listener", line 10, in <module>
2019-08-01 18:29:23.643 3671518 ERROR barbican sys.exit(main())
2019-08-01 18:29:23.643 3671518 ERROR barbican File "/usr/lib/python2.7/site-packages/barbican/cmd/keystone_listener.py", line 76, in main
2019-08-01 18:29:23.643 3671518 ERROR barbican keystone_listener.MessageServer(CONF),
2019-08-01 18:29:23.643 3671518 ERROR barbican File "/usr/lib/python2.7/site-packages/barbican/queue/keystone_listener.py", line 153, in __init__
2019-08-01 18:29:23.643 3671518 ERROR barbican NotificationTask.__init__(self, conf)
2019-08-01 18:29:23.643 3671518 ERROR barbican File "/usr/lib/python2.7/site-packages/barbican/queue/keystone_listener.py", line 45, in __init__
2019-08-01 18:29:23.643 3671518 ERROR barbican repositories.setup_database_engine_and_factory()
2019-08-01 18:29:23.643 3671518 ERROR barbican File "/usr/lib/python2.7/site-packages/barbican/model/repositories.py", line 108, in setup_database_engine_and_factory
2019-08-01 18:29:23.643 3671518 ERROR barbican _ENGINE = _get_engine(_ENGINE)
2019-08-01 18:29:23.643 3671518 ERROR barbican File "/usr/lib/python2.7/site-packages/barbican/model/repositories.py", line 192, in _get_engine
2019-08-01 18:29:23.643 3671518 ERROR barbican raise exception.BarbicanException(msg)
2019-08-01 18:29:23.643 3671518 ERROR barbican BarbicanException: Error configuring registry database with supplied sql_connection. Got error: (psycopg2.OperationalError) FATAL: password authentication failed for user "admin-barbican"
/etc/sysinv/sysinv.conf
2019-08-02 17:58:10.497 1089187 ERROR sysinv.openstack.common.threadgroup [-] (psycopg2.OperationalError) FATAL: password authentication failed for user "admin-sysinv"
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup Traceback (most recent call last):
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/threadgroup.py", line 117, in wait
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup x.wait()
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/threadgroup.py", line 49, in wait
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup return self.thread.wait()
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 175, in wait
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup return self._exit_event.wait()
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 121, in wait
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup return hubs.get_hub().switch()
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup return self.greenlet.switch()
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup result = function(*args, **kwargs)
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/service.py", line 450, in run_service
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup service.start()
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 180, in start
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup self._start()
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 197, in _start
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup system = self._create_default_system()
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 240, in _create_default_system
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup system = self.dbapi.isystem_get_one()
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sysinv/objects/__init__.py", line 102, in wrapper
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup result = fn(*args, **kwargs)
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sysinv/db/sqlalchemy/api.py", line 1185, in isystem_get_one
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup query = model_query(models.isystem)
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sysinv/db/sqlalchemy/api.py", line 132, in model_query
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup with _session_for_read() as session:
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup return self.gen.next()
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1042, in _transaction_scope
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup context=context) as resource:
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup return self.gen.next()
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 645, in _session
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup bind=self.connection, mode=self.mode)
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 409, in _create_session
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup self._start()
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 496, in _start
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup engine_args, maker_args)
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 520, in _setup_for_connection
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup sql_connection=sql_connection, **engine_kwargs)
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/debtcollector/renames.py", line 43, in decorator
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup return wrapped(*args, **kwargs)
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 202, in create_engine
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup test_conn = _test_connection(engine, max_retries, retry_interval)
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 380, in _test_connection
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup return engine.connect()
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2091, in connect
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup return self._connection_cls(self, **kwargs)
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 90, in __init__
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup if connection is not None else engine.raw_connection()
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2177, in raw_connection
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup self.pool.unique_connection, _connection)
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2151, in _wrap_pool_connect
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup e, dialect, self)
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1461, in _handle_dbapi_exception_noconnection
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup util.raise_from_cause(newraise, exc_info)
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup reraise(type(exception), exception, tb=exc_tb, cause=cause)
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2147, in _wrap_pool_connect
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup return fn()
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 328, in unique_connection
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup return _ConnectionFairy._checkout(self)
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 766, in _checkout
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup fairy = _ConnectionRecord.checkout(pool)
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 516, in checkout
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup rec = pool._do_get()
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1138, in _do_get
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup self._dec_overflow()
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup compat.reraise(exc_type, exc_value, exc_tb)
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1135, in _do_get
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup return self._create_connection()
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 333, in _create_connection
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup return _ConnectionRecord(self)
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 461, in __init__
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup self.__connect(first_connect_check=True)
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 651, in __connect
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup connection = pool._invoke_creator(self)
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup return dialect.connect(*cargs, **cparams)
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 393, in connect
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup return self.dbapi.connect(*cargs, **cparams)
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/psycopg2/__init__.py", line 164, in connect
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup conn = _connect(dsn, connection_factory=connection_factory, async=async)
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/support/psycopg2_patcher.py", line 46, in eventlet_wait_callback
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup state = conn.poll()
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup OperationalError: (psycopg2.OperationalError) FATAL: password authentication failed for user "admin-sysinv"
2019-08-02 17:58:10.497 1089187 TRACE sysinv.openstack.common.threadgroup
Severity
--------
Major
Steps to Reproduce
------------------
Install the following system:
(IPv6)
2 controlers
2 storage
4 workers
Expected Behavior
------------------
Replaying ansible-playbook /usr/share/ansible/stx-ansible/playbooks/bootstrap/bootstrap.yml
should not cause password regeneration
Actual Behavior
----------------
Replaying ansible-playbook /usr/share/ansible/stx-ansible/playbooks/bootstrap/bootstrap.yml
results in:
[sysadmin@controller-0 ~(keystone_admin)]$ system --debug host-list
DEBUG (base:187) Making authentication request to http://[abde::2]:5000/v3/auth/tokens
DEBUG (connectionpool:207) Starting new HTTP connection (1): abde::2
DEBUG (connectionpool:395) http://abde::2:5000 "POST /v3/auth/tokens HTTP/1.1" 500 141
DEBUG (session:442) Request returned failure status: 500
Authorization failed: Internal Server Error (HTTP 500)
[sysadmin@controller-0 ~(keystone_admin)]$
Reproducibility
---------------
100%
System Configuration
--------------------
(IPv6)
2 controlers
2 storage
4 workers
Wind River internal lab name: cgcs-wildcat-15-22.cumulus.wrs.com
Branch/Pull Time/Commit
-----------------------
###
### StarlingX
### Built from master
###
OS="centos"
SW_VERSION="19.01"
BUILD_TARGET="Host Installer"
BUILD_TYPE="Formal"
BUILD_ID="20190728T233000Z"
JOB="STX_build_master_master"
<email address hidden>"
BUILD_NUMBER="196"
BUILD_HOST="starlingx_mirror"
BUILD_DATE="2019-07-28 23:30:00 +0000"
Last Pass
---------
not known
Timestamp/Logs
--------------
Attached
Test Activity
-------------
Regression: Installation and Configuration
Marking as high priority since user cannot replay a playbook which is a requirement for ansible.