cinder/0 not allowed to connect to percona-cluster

Bug #1442766 reported by Andreas Hasenack
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
cinder (Juju Charms Collection)
Fix Released
High
James Page

Bug Description

During an HA run of autopilot, cinder/0 failed because it was not allowed to connect to mysql:

2015-04-10 18:08:08 INFO unit.cinder/0.shared-db-relation-changed logger.go:40 2015-04-10 18:08:08.450 20651 TRACE cinder OperationalError: (OperationalError) (1130, "Host '10-96-3-196.scapestack' is not allowed to connect to this MySQL server") None None

Full logs attached. Looks very similar to bug #1442526 where the same happened to glance.

cinder/0 is 10.96.3.196
cinder/1 is 10.96.3.116
cinder/2 is 10.96.3.111

Related branches

Revision history for this message
Andreas Hasenack (ahasenack) wrote :
information type: Proprietary → Public
affects: landscape → cinder (Juju Charms Collection)
description: updated
tags: added: landscape
Changed in cinder (Juju Charms Collection):
assignee: nobody → Liam Young (gnuoy)
summary: - cinder/0 not allowed to connect to mysql
+ cinder/0 not allowed to connect to percona-cluster
James Page (james-page)
Changed in cinder (Juju Charms Collection):
importance: Undecided → High
Revision history for this message
James Page (james-page) wrote :

unit-mysql-0[905]: 2015-04-10 18:08:08 DEBUG unit.mysql/0.juju-log cmd.go:247 shared-db:44: Grant does NOT exist for host '10.96.3.196' on db 'cinder'
unit-mysql-0[905]: 2015-04-10 18:08:08 DEBUG unit.mysql/0.juju-log cmd.go:247 shared-db:44: Grant exists for host '10.96.3.116' on db 'cinder'
unit-mysql-0[905]: 2015-04-10 18:08:09 DEBUG unit.mysql/0.juju-log cmd.go:247 shared-db:44: Grant exists for host '10.96.3.111' on db 'cinder'

Revision history for this message
James Page (james-page) wrote :
Download full text (15.4 KiB)

unit-cinder-0[898]: 2015-04-10 18:08:06 INFO unit.cinder/0.juju-log cmd.go:247 shared-db:44: [cinder] Cluster leader, performing db sync
unit-cinder-0[898]: 2015-04-10 18:08:08 INFO unit.cinder/0.shared-db-relation-changed logger.go:40 2015-04-10 18:08:08.450 20651 CRITICAL cinder [-] OperationalError: (OperationalError) (1130, "Host '10-96-3-196.scapestack' is not allowed to connect to this MySQL server") None None
unit-cinder-0[898]: 2015-04-10 18:08:08 INFO unit.cinder/0.shared-db-relation-changed logger.go:40 2015-04-10 18:08:08.450 20651 TRACE cinder Traceback (most recent call last):
unit-cinder-0[898]: 2015-04-10 18:08:08 INFO unit.cinder/0.shared-db-relation-changed logger.go:40 2015-04-10 18:08:08.450 20651 TRACE cinder File "/usr/bin/cinder-manage", line 552, in <module>
unit-cinder-0[898]: 2015-04-10 18:08:08 INFO unit.cinder/0.shared-db-relation-changed logger.go:40 2015-04-10 18:08:08.450 20651 TRACE cinder main()
unit-cinder-0[898]: 2015-04-10 18:08:08 INFO unit.cinder/0.shared-db-relation-changed logger.go:40 2015-04-10 18:08:08.450 20651 TRACE cinder File "/usr/bin/cinder-manage", line 549, in main
unit-cinder-0[898]: 2015-04-10 18:08:08 INFO unit.cinder/0.shared-db-relation-changed logger.go:40 2015-04-10 18:08:08.450 20651 TRACE cinder fn(*fn_args)
unit-cinder-0[898]: 2015-04-10 18:08:08 INFO unit.cinder/0.shared-db-relation-changed logger.go:40 2015-04-10 18:08:08.450 20651 TRACE cinder File "/usr/bin/cinder-manage", line 229, in sync
unit-cinder-0[898]: 2015-04-10 18:08:08 INFO unit.cinder/0.shared-db-relation-changed logger.go:40 2015-04-10 18:08:08.450 20651 TRACE cinder return migration.db_sync(version)
unit-cinder-0[898]: 2015-04-10 18:08:08 INFO unit.cinder/0.shared-db-relation-changed logger.go:40 2015-04-10 18:08:08.450 20651 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/db/migration.py", line 28, in db_sync
unit-cinder-0[898]: 2015-04-10 18:08:08 INFO unit.cinder/0.shared-db-relation-changed logger.go:40 2015-04-10 18:08:08.450 20651 TRACE cinder return IMPL.db_sync(version=version)
unit-cinder-0[898]: 2015-04-10 18:08:08 INFO unit.cinder/0.shared-db-relation-changed logger.go:40 2015-04-10 18:08:08.450 20651 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/db/sqlalchemy/migration.py", line 40, in db_sync
unit-cinder-0[898]: 2015-04-10 18:08:08 INFO unit.cinder/0.shared-db-relation-changed logger.go:40 2015-04-10 18:08:08.450 20651 TRACE cinder current_version = db_version()
unit-cinder-0[898]: 2015-04-10 18:08:08 INFO unit.cinder/0.shared-db-relation-changed logger.go:40 2015-04-10 18:08:08.450 20651 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/db/sqlalchemy/migration.py", line 52, in db_version
unit-cinder-0[898]: 2015-04-10 18:08:08 INFO unit.cinder/0.shared-db-relation-changed logger.go:40 2015-04-10 18:08:08.450 20651 TRACE cinder return versioning_api.db_version(get_engine(), repository)
unit-cinder-0[898]: 2015-04-10 18:08:08 INFO unit.cinder/0.shared-db-relation-changed logger.go:40 2015-04-10 18:08:08.450 20651 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/db/sqlalchemy/api.py", line 80, in get_engine
unit-cinder-0[898]...

Revision history for this message
James Page (james-page) wrote :

This is the same as bug 1442526

This issue happens due to the fact that allowed_units is only presented from the lead mysql unit, whereas the username and password required to complete the remote context is presented by all mysql units.

This means that:

    if eligible_leader(CLUSTER_RES):
        # Bugs 1353135 & 1187508. Dbs can appear to be ready before the units
        # acl entry has been added. So, if the db supports passing a list of
        # permitted units then check if we're in the list.
        allowed_units = relation_get('allowed_units')
        if allowed_units and local_unit() not in allowed_units.split():
            juju_log('Allowed_units list provided and this unit not present')
            return
        juju_log('Cluster leader, performing db sync')
        migrate_database()

fires when shared-db-relation-changed fires due to username/password change on a non-lead unit, which does not present allowed_units, but the local leader is not yet in the access list.

This resolves as the hooks complete execution and the access is granted - resolved --retry will just syncup and complete at that point in time.

Changed in cinder (Juju Charms Collection):
status: New → Triaged
status: Triaged → Confirmed
James Page (james-page)
Changed in cinder (Juju Charms Collection):
assignee: Liam Young (gnuoy) → James Page (james-page)
status: Confirmed → In Progress
milestone: none → 15.04
James Page (james-page)
Changed in cinder (Juju Charms Collection):
status: In Progress → Fix Committed
James Page (james-page)
Changed in cinder (Juju Charms Collection):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.