upgrade_storage aborts on denied access to cinder@'localhost' from R2.21-102 and R2.23-119 to ML 3.0-2713 juno.
2016-02-15 01:48:51:876384: [root@10.87.129.173] out: [10.87.129.173] out: [localhost] local: sudo cinder-manage db sync^M
2016-02-15 01:48:51:908190: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 CRITICAL cinder [-] OperationalError: (OperationalError) (1045, "Access denied for user 'cinder'@'localhost' (using password: YES)") None None^M
2016-02-15 01:48:52:609851: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder Traceback (most recent call last):^M
2016-02-15 01:48:52:610044: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/bin/cinder-manage", line 552, in <module>^M
2016-02-15 01:48:52:610145: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder main()^M
2016-02-15 01:48:52:610243: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/bin/cinder-manage", line 549, in main^M
2016-02-15 01:48:52:610371: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder fn(*fn_args)^M
2016-02-15 01:48:52:610592: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/bin/cinder-manage", line 229, in sync^M
2016-02-15 01:48:52:610699: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder return migration.db_sync(version)^M
2016-02-15 01:48:52:610795: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/db/migration.py", line 28, in db_sync^M
2016-02-15 01:48:52:610890: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder return IMPL.db_sync(version=version)^M
2016-02-15 01:48:52:610985: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/db/sqlalchemy/migration.py", line 40, in db_sync^M
2016-02-15 01:48:52:611082: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder current_version = db_version()^M
2016-02-15 01:48:52:611177: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/db/sqlalchemy/migration.py", line 52, in db_version^M
2016-02-15 01:48:52:611272: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder return versioning_api.db_version(get_engine(), repository)^M
2016-02-15 01:48:52:611367: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/db/sqlalchemy/api.py", line 80, in get_engine^M
2016-02-15 01:48:52:611462: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder facade = _create_facade_lazily()^M
2016-02-15 01:48:52:611742: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder **dict(CONF.database.iteritems())^M
2016-02-15 01:48:52:611864: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/lib/python2.7/dist-packages/oslo/db/sqlalchemy/session.py", line 711, in __init__^M
2016-02-15 01:48:52:611970: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder **engine_kwargs)^M
2016-02-15 01:48:52:612066: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/lib/python2.7/dist-packages/oslo/db/sqlalchemy/session.py", line 386, in create_engine^M
2016-02-15 01:48:52:612161: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder connection_trace=connection_trace^M
2016-02-15 01:48:52:612256: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/lib/python2.7/dist-packages/oslo/db/sqlalchemy/utils.py", line 890, in __call__^M
2016-02-15 01:48:52:612350: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder self._url_from_target(target), target, arg, kw)^M
2016-02-15 01:48:52:612593: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/lib/python2.7/dist-packages/oslo/db/sqlalchemy/utils.py", line 927, in _dispatch_on^M
2016-02-15 01:48:52:612715: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder return self._dispatch_on_db_driver(dbname, driver, arg, kw)^M
2016-02-15 01:48:52:612813: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/lib/python2.7/dist-packages/oslo/db/sqlalchemy/utils.py", line 981, in _dispatch_on_db_driver^M
2016-02-15 01:48:52:612909: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder if self._invoke_fn(fn, arg, kw) is not None:^M
2016-02-15 01:48:52:613022: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/lib/python2.7/dist-packages/oslo/db/sqlalchemy/utils.py", line 930, in _invoke_fn^M
2016-02-15 01:48:52:613118: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder return fn(*arg, **kw)^M
2016-02-15 01:48:52:613213: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/lib/python2.7/dist-packages/oslo/db/sqlalchemy/session.py", line 456, in _init_events^M
2016-02-15 01:48:52:613306: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder realmode = engine.execute("SHOW VARIABLES LIKE 'sql_mode'").fetchone()^M
2016-02-15 01:48:52:615783: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1751, in execute^M
2016-02-15 01:48:52:615967: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder connection = self.contextual_connect(close_with_result=True)^M
2016-02-15 01:48:52:616072: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1799, in contextual_connect^M
2016-02-15 01:48:52:616172: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder self.pool.connect(),^M
2016-02-15 01:48:52:616270: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 338, in connect^M
2016-02-15 01:48:52:616366: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder return _ConnectionFairy._checkout(self)^M
2016-02-15 01:48:52:616461: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 641, in _checkout^M
2016-02-15 01:48:52:616556: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder fairy = _ConnectionRecord.checkout(pool)^M
2016-02-15 01:48:52:616651: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 440, in checkout^M
2016-02-15 01:48:52:616746: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder rec = pool._do_get()^M
2016-02-15 01:48:52:616875: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 961, in _do_get^M
2016-02-15 01:48:52:617039: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder return self._create_connection()^M
2016-02-15 01:48:52:617209: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 285, in _create_connection^M
2016-02-15 01:48:52:617370: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder return _ConnectionRecord(self)^M
2016-02-15 01:48:52:617529: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 411, in __init__^M
2016-02-15 01:48:52:617699: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder self.connection = self.__connect()^M
2016-02-15 01:48:52:617865: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 537, in __connect^M
2016-02-15 01:48:52:618029: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder connection = self.__pool._creator()^M
2016-02-15 01:48:52:618198: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line 96, in connect^M
2016-02-15 01:48:52:618344: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder connection_invalidated=invalidated^M
2016-02-15 01:48:52:618494: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause^M
2016-02-15 01:48:52:618641: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder reraise(type(exception), exception, tb=exc_tb)^M
2016-02-15 01:48:52:618783: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line 90, in connect^M
2016-02-15 01:48:52:618882: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder return dialect.connect(*cargs, **cparams)^M
2016-02-15 01:48:52:618979: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 377, in connect^M
2016-02-15 01:48:52:619073: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder return self.dbapi.connect(*cargs, **cparams)^M
2016-02-15 01:48:52:619168: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/lib/python2.7/dist-packages/MySQLdb/__init__.py", line 81, in Connect^M
2016-02-15 01:48:52:619263: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder return Connection(*args, **kwargs)^M
2016-02-15 01:48:52:619358: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 187, in __init__^M
2016-02-15 01:48:52:619263: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder return Connection(*args, **kwargs)^M
2016-02-15 01:48:52:619358: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 187, in __init__^M
2016-02-15 01:48:52:619453: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder super(Connection, self).__init__(*args, **kwargs2)^M
2016-02-15 01:48:52:619547: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder OperationalError: (OperationalError) (1045, "Access denied for user 'cinder'@'localhost' (using password: YES)") None None^M
2016-02-15 01:48:52:622061: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder ^M
2016-02-15 01:48:52:622187: [root@10.87.129.173] out: [10.87.129.173] out: ^M
2016-02-15 01:48:52:622281: [root@10.87.129.173] out: [10.87.129.173] out: Fatal error: local() encountered an error (return code 1) while executing 'sudo cinder-manage db sync'^M
2016-02-15 01:48:52:622374: [root@10.87.129.173] out: [10.87.129.173] out: ^M
2016-02-15 01:48:52:622466: [root@10.87.129.173] out: [10.87.129.173] out: Aborting.^M
2016-02-15 01:48:52:622617: [root@10.87.129.173] out: [10.87.129.173] out: ^M
2016-02-15 01:48:52:723390: [root@10.87.129.173] out: ^M
2016-02-15 01:48:52:723521: [root@10.87.129.173] out: ^M
2016-02-15 01:48:52:723739: [root@10.87.129.173] out: Fatal error: run() received nonzero return code 1 while executing!^M
2016-02-15 01:48:52:723865: [root@10.87.129.173] out: ^M
2016-02-15 01:48:52:723960: [root@10.87.129.173] out: Requested: sudo storage-fs-setup --storage-master 10.87.129.173 --storage-setup-mode upgrade --storage-hostnames cmbu-vse2100-1 cmbu-vse2100-10 cmbu-vse2100-11 --storage-compute-hostnames cmbu-vse2100-10 cmbu-vse2100-11 --storage-hosts 10.87.129.173 10.87.129.174 10.87.129.175 --storage-host-tokens n1keenA n1keenA n1keenA --storage-disk-config cmbu-vse2100-10:/dev/sdc cmbu-vse2100-11:/dev/sdc --storage-ssd-disk-config none --storage-journal-config cmbu-vse2100-10:/dev/sdb cmbu-vse2100-11:/dev/sdb --storage-local-disk-config none --storage-local-ssd-disk-config none --storage-nfs-disk-config none --storage-directory-config none --storage-chassis-config none --storage-mon-hosts none --collector-hosts 10.87.129.173 --collector-host-tokens n1keenA --cfg-host 10.87.129.173 --cinder-vip none --config-hosts none --storage-os-hosts none --storage-os-host-tokens none --cfg-vip none --storage-replica-size None --openstack-ip 10.87.129.173 --orig-hostnames cmbu-vse2100-1 cmbu-vse2100-10 cmbu-vse2100-11 --service-dbpass c0ntrail123^M
2016-02-15 01:48:52:724215: [root@10.87.129.173] out: ^M
2016-02-15 01:48:52:724346: [root@10.87.129.173] out: Aborting.^M
Review in progress for https:/ /review. opencontrail. org/17616
Submitter: Jeya ganesh babu (<email address hidden>)