Comment 0 for bug 1728132

On Artful x86_64, percona-cluster shared-db-relation-changed hooks frequently error out, but not always. Various db migrate operations fail (sometimes keystone, sometimes glance).

Juju unit logs indicate db migrate operations were underway on the api unit, and system logs indicate mysql (percona-cluster) crashed on the db unit.

Artifacts:
https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_amulet_full/openstack/charm-ceph/508041/4/743/consoleText.test_charm_amulet_full_1111.txt

percona-cluster/0* error idle 8 172.17.106.18 3306/tcp hook failed: "shared-db-relation-changed" for glance:shared-db

# A simple percona-cluster exercise on Artful, can readily repro with this on serverstack
series: artful
relations:
- - keystone:shared-db
  - percona-cluster:shared-db
- - glance:identity-service
  - keystone:identity-service
- - glance:amqp
  - rabbitmq-server:amqp
services:
  percona-cluster:
    charm: cs:~openstack-charmers-next/percona-cluster
    num_units: 1
    constraints: "cpu-cores=4 mem=4G"
    options:
      max-connections: 1000
      innodb-buffer-pool-size: 256M
  keystone:
    charm: cs:~openstack-charmers-next/keystone
    num_units: 1
    constraints: "cpu-cores=2 mem=2G"
    options:
      admin-password: openstack
      worker-multiplier: 0.25
  glance:
    charm: cs:~openstack-charmers-next/glance
    num_units: 1
    constraints: "cpu-cores=2 mem=2G"
    options:
      worker-multiplier: 0.25
  rabbitmq-server:
    charm: cs:~openstack-charmers-next/rabbitmq-server
    num_units: 1
    constraints: "cpu-cores=4 mem=4G"

DEBUG:bundletester.utils:Updating JUJU_MODEL: "auto-osci-sv06:admin/auto-osci-sv06" -> ""
DEBUG:bundletester.fetchers:git rev-parse HEAD: d0d41964227bc1dc46887fa77ba4be3bc5065fae

ERROR: InvocationError: '/var/lib/jenkins/checkout/0/ceph/.tox/func27/bin/bundletester -vl DEBUG -r json -o func-results.json --test-pattern gate-* --no-destroy'
___________________________________ summary ____________________________________
ERROR: func27: commands failed
 ! Amulet test failed.
Model Controller Cloud/Region Version SLA
auto-osci-sv06 auto-osci-sv06 serverstack/serverstack 2.2.4 unsupported

App Version Status Scale Charm Store Rev OS Notes
ceph 12.2.0 active 3 ceph local 105 ubuntu
ceph-osd 12.2.0 active 1 ceph-osd jujucharms 297 ubuntu
cinder 11.0.0 waiting 1 cinder jujucharms 298 ubuntu
cinder-ceph waiting 0/1 cinder-ceph jujucharms 247 ubuntu
glance 15.0.0 waiting 1 glance jujucharms 292 ubuntu
keystone 12.0.0 waiting 1 keystone jujucharms 322 ubuntu
nova-compute 16.0.1 active 1 nova-compute jujucharms 336 ubuntu
percona-cluster 5.6.34-26.19 error 1 percona-cluster jujucharms 274 ubuntu
rabbitmq-server 3.6.10 active 1 rabbitmq-server jujucharms 280 ubuntu

Unit Workload Agent Machine Public address Ports Message
ceph-osd/0* active idle 3 172.17.106.17 Unit is ready (2 OSD)
ceph/0* active idle 0 172.17.106.10 Unit is ready and clustered
ceph/1 active idle 1 172.17.106.5 Unit is ready and clustered
ceph/2 active idle 2 172.17.106.3 Unit is ready and clustered
cinder/0* waiting executing 4 172.17.106.20 8776/tcp Incomplete relations: messaging, identity, database
  cinder-ceph/0* waiting allocating 172.17.106.20 agent initializing
glance/0* waiting executing 5 172.17.106.22 9292/tcp Incomplete relations: identity, database
keystone/0* waiting executing 6 172.17.106.21 5000/tcp Incomplete relations: database
nova-compute/0* active executing 7 172.17.106.4 Unit is ready
percona-cluster/0* error idle 8 172.17.106.18 3306/tcp hook failed: "shared-db-relation-changed" for glance:shared-db
rabbitmq-server/0* active idle 9 172.17.106.26 5672/tcp Unit is ready

Machine State DNS Inst id Series AZ Message
0 started 172.17.106.10 cc716b8d-f4ff-450d-b088-a3742a5ffffd artful nova ACTIVE
1 started 172.17.106.5 e0498107-15fe-4a5e-93a2-a1f6cfd9523b artful nova ACTIVE
2 started 172.17.106.3 6a02e6cc-2fdb-4b81-b4ab-0467c5e5705d artful nova ACTIVE
3 started 172.17.106.17 cc5a2f76-73a6-4f50-a97e-ae391ed96e63 artful nova ACTIVE
4 started 172.17.106.20 4d42c847-9dd3-4f1a-a668-0b5eb7819b9a artful nova ACTIVE
5 started 172.17.106.22 8dd50778-396b-4c27-b063-34ff63ddb3f2 artful nova ACTIVE
6 started 172.17.106.21 58eac09b-88d5-4f1b-a97d-b22176cc231f artful nova ACTIVE
7 started 172.17.106.4 534d23a0-5c30-451a-8963-68057bac27f8 artful nova ACTIVE
8 started 172.17.106.18 091cb17d-5898-44b4-9f0a-3bffbe808d05 artful nova ACTIVE
9 started 172.17.106.26 eb03b662-332e-48d5-8b5d-c39e3deadb4a artful nova ACTIVE

Relation provider Requirer Interface Type
ceph:client cinder-ceph:ceph ceph-client regular
ceph:client glance:ceph ceph-client regular
ceph:client nova-compute:ceph ceph-client regular
ceph:mon ceph:mon ceph peer
ceph:osd ceph-osd:mon ceph-osd regular
cinder-ceph:storage-backend cinder:storage-backend cinder-backend subordinate
cinder:cluster cinder:cluster cinder-ha peer
glance:cluster glance:cluster glance-ha peer
glance:image-service cinder:image-service glance regular
glance:image-service nova-compute:image-service glance regular
keystone:cluster keystone:cluster keystone-ha peer
keystone:identity-service cinder:identity-service keystone regular
keystone:identity-service glance:identity-service keystone regular
nova-compute:compute-peer nova-compute:compute-peer nova peer
percona-cluster:cluster percona-cluster:cluster percona-cluster peer
percona-cluster:shared-db cinder:shared-db mysql-shared regular
percona-cluster:shared-db glance:shared-db mysql-shared regular
percona-cluster:shared-db keystone:shared-db mysql-shared regular
percona-cluster:shared-db nova-compute:shared-db mysql-shared regular
rabbitmq-server:amqp cinder:amqp rabbitmq regular
rabbitmq-server:amqp glance:amqp rabbitmq regular
rabbitmq-server:amqp nova-compute:amqp rabbitmq regular
rabbitmq-server:cluster rabbitmq-server:cluster rabbitmq-ha peer