Juniper Openstack

upgrade_storage abort on denied access to cinder@'localhost'

Bug #1545774 reported by wenqing liang on 2016-02-15

This bug affects 1 person

	Status	Importance	Assigned to	Milestone
Juniper Openstack	Status tracked in Trunk
R3.0	Fix Committed	High	Jeya ganesh babu J	Juniper Openstack r3.0-fcs
Trunk	Fix Committed	High	Jeya ganesh babu J

Bug Description

upgrade_storage aborts on denied access to cinder@'localhost' from R2.21-102 and R2.23-119 to ML 3.0-2713 juno.

2016-02-15 01:48:51:876384: [root@10.87.129.173] out: [10.87.129.173] out: [localhost] local: sudo cinder-manage db sync^M
2016-02-15 01:48:51:908190: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 CRITICAL cinder [-] OperationalError: (OperationalError) (1045, "Access denied for user 'cinder'@'localhost' (using password: YES)") None None^M
2016-02-15 01:48:52:609851: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder Traceback (most recent call last):^M
2016-02-15 01:48:52:610044: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/bin/cinder-manage", line 552, in <module>^M
2016-02-15 01:48:52:610145: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder main()^M
2016-02-15 01:48:52:610243: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/bin/cinder-manage", line 549, in main^M
2016-02-15 01:48:52:610371: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder fn(*fn_args)^M
2016-02-15 01:48:52:610592: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/bin/cinder-manage", line 229, in sync^M
2016-02-15 01:48:52:610699: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder return migration.db_sync(version)^M
2016-02-15 01:48:52:610795: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/db/migration.py", line 28, in db_sync^M
2016-02-15 01:48:52:610890: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder return IMPL.db_sync(version=version)^M
2016-02-15 01:48:52:610985: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/db/sqlalchemy/migration.py", line 40, in db_sync^M
2016-02-15 01:48:52:611082: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder current_version = db_version()^M
2016-02-15 01:48:52:611177: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/db/sqlalchemy/migration.py", line 52, in db_version^M
2016-02-15 01:48:52:611272: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder return versioning_api.db_version(get_engine(), repository)^M
2016-02-15 01:48:52:611367: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/db/sqlalchemy/api.py", line 80, in get_engine^M
2016-02-15 01:48:52:611462: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder facade = _create_facade_lazily()^M
2016-02-15 01:48:52:611742: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder **dict(CONF.database.iteritems())^M
2016-02-15 01:48:52:611864: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/lib/python2.7/dist-packages/oslo/db/sqlalchemy/session.py", line 711, in __init__^M
2016-02-15 01:48:52:611970: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder **engine_kwargs)^M
2016-02-15 01:48:52:612066: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/lib/python2.7/dist-packages/oslo/db/sqlalchemy/session.py", line 386, in create_engine^M
2016-02-15 01:48:52:612161: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder connection_trace=connection_trace^M
2016-02-15 01:48:52:612256: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/lib/python2.7/dist-packages/oslo/db/sqlalchemy/utils.py", line 890, in __call__^M
2016-02-15 01:48:52:612350: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder self._url_from_target(target), target, arg, kw)^M
2016-02-15 01:48:52:612593: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/lib/python2.7/dist-packages/oslo/db/sqlalchemy/utils.py", line 927, in _dispatch_on^M
2016-02-15 01:48:52:612715: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder return self._dispatch_on_db_driver(dbname, driver, arg, kw)^M
2016-02-15 01:48:52:612813: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/lib/python2.7/dist-packages/oslo/db/sqlalchemy/utils.py", line 981, in _dispatch_on_db_driver^M
2016-02-15 01:48:52:612909: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder if self._invoke_fn(fn, arg, kw) is not None:^M
2016-02-15 01:48:52:613022: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/lib/python2.7/dist-packages/oslo/db/sqlalchemy/utils.py", line 930, in _invoke_fn^M
2016-02-15 01:48:52:613118: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder return fn(*arg, **kw)^M
2016-02-15 01:48:52:613213: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/lib/python2.7/dist-packages/oslo/db/sqlalchemy/session.py", line 456, in _init_events^M
2016-02-15 01:48:52:613306: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder realmode = engine.execute("SHOW VARIABLES LIKE 'sql_mode'").fetchone()^M
2016-02-15 01:48:52:615783: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1751, in execute^M
2016-02-15 01:48:52:615967: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder connection = self.contextual_connect(close_with_result=True)^M
2016-02-15 01:48:52:616072: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1799, in contextual_connect^M
2016-02-15 01:48:52:616172: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder self.pool.connect(),^M
2016-02-15 01:48:52:616270: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 338, in connect^M
2016-02-15 01:48:52:616366: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder return _ConnectionFairy._checkout(self)^M
2016-02-15 01:48:52:616461: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 641, in _checkout^M
2016-02-15 01:48:52:616556: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder fairy = _ConnectionRecord.checkout(pool)^M
2016-02-15 01:48:52:616651: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 440, in checkout^M
2016-02-15 01:48:52:616746: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder rec = pool._do_get()^M
2016-02-15 01:48:52:616875: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 961, in _do_get^M
2016-02-15 01:48:52:617039: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder return self._create_connection()^M
2016-02-15 01:48:52:617209: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 285, in _create_connection^M
2016-02-15 01:48:52:617370: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder return _ConnectionRecord(self)^M
2016-02-15 01:48:52:617529: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 411, in __init__^M
2016-02-15 01:48:52:617699: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder self.connection = self.__connect()^M
2016-02-15 01:48:52:617865: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 537, in __connect^M
2016-02-15 01:48:52:618029: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder connection = self.__pool._creator()^M
2016-02-15 01:48:52:618198: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line 96, in connect^M
2016-02-15 01:48:52:618344: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder connection_invalidated=invalidated^M
2016-02-15 01:48:52:618494: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause^M
2016-02-15 01:48:52:618641: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder reraise(type(exception), exception, tb=exc_tb)^M
2016-02-15 01:48:52:618783: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line 90, in connect^M
2016-02-15 01:48:52:618882: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder return dialect.connect(*cargs, **cparams)^M
2016-02-15 01:48:52:618979: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 377, in connect^M
2016-02-15 01:48:52:619073: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder return self.dbapi.connect(*cargs, **cparams)^M
2016-02-15 01:48:52:619168: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/lib/python2.7/dist-packages/MySQLdb/__init__.py", line 81, in Connect^M
2016-02-15 01:48:52:619263: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder return Connection(*args, **kwargs)^M
2016-02-15 01:48:52:619358: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 187, in __init__^M
2016-02-15 01:48:52:619263: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder return Connection(*args, **kwargs)^M
2016-02-15 01:48:52:619358: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 187, in __init__^M
2016-02-15 01:48:52:619453: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder super(Connection, self).__init__(*args, **kwargs2)^M
2016-02-15 01:48:52:619547: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder OperationalError: (OperationalError) (1045, "Access denied for user 'cinder'@'localhost' (using password: YES)") None None^M
2016-02-15 01:48:52:622061: [root@10.87.129.173] out: [10.87.129.173] out: 2016-02-15 01:48:52.542 6363 TRACE cinder ^M
2016-02-15 01:48:52:622187: [root@10.87.129.173] out: [10.87.129.173] out: ^M
2016-02-15 01:48:52:622281: [root@10.87.129.173] out: [10.87.129.173] out: Fatal error: local() encountered an error (return code 1) while executing 'sudo cinder-manage db sync'^M
2016-02-15 01:48:52:622374: [root@10.87.129.173] out: [10.87.129.173] out: ^M
2016-02-15 01:48:52:622466: [root@10.87.129.173] out: [10.87.129.173] out: Aborting.^M
2016-02-15 01:48:52:622617: [root@10.87.129.173] out: [10.87.129.173] out: ^M
2016-02-15 01:48:52:723390: [root@10.87.129.173] out: ^M
2016-02-15 01:48:52:723521: [root@10.87.129.173] out: ^M
2016-02-15 01:48:52:723739: [root@10.87.129.173] out: Fatal error: run() received nonzero return code 1 while executing!^M
2016-02-15 01:48:52:723865: [root@10.87.129.173] out: ^M
2016-02-15 01:48:52:723960: [root@10.87.129.173] out: Requested: sudo storage-fs-setup --storage-master 10.87.129.173 --storage-setup-mode upgrade --storage-hostnames cmbu-vse2100-1 cmbu-vse2100-10 cmbu-vse2100-11 --storage-compute-hostnames cmbu-vse2100-10 cmbu-vse2100-11 --storage-hosts 10.87.129.173 10.87.129.174 10.87.129.175 --storage-host-tokens n1keenA n1keenA n1keenA --storage-disk-config cmbu-vse2100-10:/dev/sdc cmbu-vse2100-11:/dev/sdc --storage-ssd-disk-config none --storage-journal-config cmbu-vse2100-10:/dev/sdb cmbu-vse2100-11:/dev/sdb --storage-local-disk-config none --storage-local-ssd-disk-config none --storage-nfs-disk-config none --storage-directory-config none --storage-chassis-config none --storage-mon-hosts none --collector-hosts 10.87.129.173 --collector-host-tokens n1keenA --cfg-host 10.87.129.173 --cinder-vip none --config-hosts none --storage-os-hosts none --storage-os-host-tokens none --cfg-vip none --storage-replica-size None --openstack-ip 10.87.129.173 --orig-hostnames cmbu-vse2100-1 cmbu-vse2100-10 cmbu-vse2100-11 --service-dbpass c0ntrail123^M

2016-02-15 01:48:52:724215: [root@10.87.129.173] out: ^M
2016-02-15 01:48:52:724346: [root@10.87.129.173] out: Aborting.^M

See original description

Tags:

wenqing liang (wliang) on 2016-02-15

information type:	Proprietary → Public
description:	updated

wenqing liang (wliang) on 2016-02-19

tags:

added: blocker

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-02-23: [Review update] R3.0

Review in progress for https://review.opencontrail.org/17616
Submitter: Jeya ganesh babu (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-02-23:

Review in progress for https://review.opencontrail.org/17673
Submitter: Jeya ganesh babu (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-02-24: A change has been merged

Reviewed: https://review.opencontrail.org/17673
Committed: http://github.org/Juniper/contrail-provisioning/commit/7c1571bbc4186183a45f76de6b51103f17859033
Submitter: Zuul
Branch: R3.0

commit 7c1571bbc4186183a45f76de6b51103f17859033
Author: Jeya ganesh babu J <email address hidden>
Date: Tue Feb 23 12:31:48 2016 -0800

Cinder/Storage provision issue fix in HA/kilo

Closes-Bug: #1547750
Closes-Bug: #1545774

Change-Id: I819062b75360cf20faa886b1ad523e73c7f235c2

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-02-29: [Review update] master

Review in progress for https://review.opencontrail.org/18002
Submitter: Jeya ganesh babu (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-02-29:

Review in progress for https://review.opencontrail.org/18041
Submitter: Jeya ganesh babu (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-03-01: A change has been merged

#10

Reviewed: https://review.opencontrail.org/18041
Committed: http://github.org/Juniper/contrail-provisioning/commit/d5694dcf5c5a73adaad154964f8f5069e3d5a6d2
Submitter: Zuul
Branch: master

commit d5694dcf5c5a73adaad154964f8f5069e3d5a6d2
Author: Jeya ganesh babu J <email address hidden>
Date: Mon Feb 29 13:51:39 2016 -0800

Cinder/Storage provision issue fix in HA/kilo

Closes-Bug: #1547750
Closes-Bug: #1545774
Closes-Bug: #1550589

1. With Kilo, instead of v1 cinder, v2 cinder is created.
2. During upgrade storage, need to use the existing db password
instead of the new password.
3. Fixed issue related to lvm provisioning in storage for kilo
4. Fixed issue related to version check for cinder packages during
keystone service creation.

Change-Id: Iecb922d4f10a5feb300e481fb230f1f57fa97dd0

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.