HA: R3.1-build-18-mysql service not up after rebooting one of the cfgm

[shajuvk]

my sql service down after rebooting a cfgm node on HA setup. Before restart all the nodes are in sync and sql was running.

before restarting 10.84.13.38 :
=======================
root@a6s32:/var/log/nova# mysql -uroot -p4b781ef3ba3d8da503b7 -e "show status like '%wsrep%'";
+----------------------------+----------------------------------------------------+
| Variable_name | Value |
+----------------------------+----------------------------------------------------+
| wsrep_local_state_uuid | b270ca19-5dc6-11e6-ba62-5a5b07bc3f52 |
| wsrep_protocol_version | 4 |
| wsrep_last_committed | 96322 |
| wsrep_replicated | 76221 |
| wsrep_replicated_bytes | 186951862 |
| wsrep_received | 19834 |
| wsrep_received_bytes | 101604441 |
| wsrep_local_commits | 75692 |
| wsrep_local_cert_failures | 0 |
| wsrep_local_replays | 0 |
| wsrep_local_send_queue | 0 |
| wsrep_local_send_queue_avg | 0.000000 |
| wsrep_local_recv_queue | 0 |
| wsrep_local_recv_queue_avg | 0.000000 |
| wsrep_flow_control_paused | 0.000000 |
| wsrep_flow_control_sent | 0 |
| wsrep_flow_control_recv | 0 |
| wsrep_cert_deps_distance | 63.105634 |
| wsrep_apply_oooe | 0.000000 |
| wsrep_apply_oool | 0.000000 |
| wsrep_apply_window | 0.000000 |
| wsrep_commit_oooe | 0.000000 |
| wsrep_commit_oool | 0.000000 |
| wsrep_commit_window | 0.000000 |
| wsrep_local_state | 4 |
| wsrep_local_state_comment | Synced |
| wsrep_cert_index_size | 114 |
| wsrep_causal_reads | 2244236 |
| wsrep_incoming_addresses | 10.84.13.38:3306,10.84.13.32:3306,10.84.13.33:3306 |
| wsrep_cluster_conf_id | 7 |
| wsrep_cluster_size | 3 |
| wsrep_cluster_state_uuid | b270ca19-5dc6-11e6-ba62-5a5b07bc3f52 |
| wsrep_cluster_status | Primary |
| wsrep_connected | ON |
| wsrep_local_bf_aborts | 0 |
| wsrep_local_index | 1 |
| wsrep_provider_name | Galera |
| wsrep_provider_vendor | Codership Oy <email address hidden> |
| wsrep_provider_version | 25.2.9(r172) |
| wsrep_ready | ON |

=====

sql/error.log
=====

160809 16:52:34 [Note] Event Scheduler: Purging the queue. 1 events
160809 16:52:34 [Note] WSREP: dtor state: CLOSED
160809 16:52:34 [Note] WSREP: apply mon: entered 0
160809 16:52:34 [Note] WSREP: apply mon: entered 0
160809 16:52:34 [Note] WSREP: mon: entered 36 oooe fraction 0 oool fraction 0.0277778
160809 16:52:34 [Note] WSREP: cert index usage at exit 41
160809 16:52:34 [Note] WSREP: cert trx map usage at exit 31
160809 16:52:34 [Note] WSREP: deps set usage at exit 0
160809 16:52:34 [Note] WSREP: avg deps dist 12.0323
160809 16:52:34 [Note] WSREP: wsdb trx map usage 0 conn query map usage 0
160809 16:52:34 [Note] WSREP: Shifting CLOSED -> DESTROYED (TO: 96466)
160809 16:52:34 [Note] WSREP: Flushing memory map to disk...
160809 16:52:34 InnoDB: Starting shutdown...
160809 16:52:35 InnoDB: Shutdown completed; log sequence number 354370014
160809 16:52:35 [Note] /usr/sbin/mysqld: Shutdown complete

160809 16:52:35 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended

work around: service mysql restart

[Jeba Paulaiyan (jebap)]

Reasonable workaround available.

[Sudheendra Rao (sudheendra-k)]

on R3.1 build38 mitaka HA setup, the mysql comes up after restarting it, but VM creation fails with following stack trace in nova-api.log

Setup details:
Config Nodes : [u'nodec7', u'nodec8', u'nodeg36']
Control Nodes : [u'nodec7', u'nodec8', u'nodeg36']
Compute Nodes : [u'nodei1', u'nodei2', u'nodei3']
Openstack Node : [u'nodec7', u'nodec8', u'nodeg36']
WebUI Node : nodec8
Analytics Nodes : [u'nodec7', u'nodec8', u'nodeg36']
Database Nodes : [u'nodec7', u'nodec8', u'nodeg36']
Physical Devices : [u'blr-mx1', u'hooper', u"'hooper'"]

nova-api.log:

2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions [req-71adf19d-cc99-41dc-a4e2-6500a4cc70fc 0fb41841bee2429a8389f30a34d9f29f e337e7852def47dc8bc604edb90f93f1 - - -] Unexpected exception in API method
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/extensions.py", line 478, in wrapped
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions return f(*args, **kwargs)
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/images.py", line 145, in detail
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions **page_params)
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/image/api.py", line 68, in get_all
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions return session.detail(context, **kwargs)
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 300, in detail
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions for image in images:
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/glanceclient/v1/images.py", line 254, in list
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions for image in paginate(params, return_request_id):
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/glanceclient/v1/images.py", line 238, in paginate
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions images, resp = self._list(url, "images")
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/glanceclient/v1/images.py", line 63, in _list
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions resp, body = self.client.get(url)
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 275, in get
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions return self._request('GET', url, **kwargs)
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 256, in _request
2016-11-07 17:48:04.098 12887 ERROR nova.api.openstack.extensions raise exc.Communic...

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/25828
Submitter: Ranjeet R (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/25829
Submitter: Ranjeet R (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/25828
Submitter: Ranjeet R (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/25829
Submitter: Ranjeet R (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/25828
Committed: http://github.org/Juniper/contrail-provisioning/commit/7d3e1a6838b50eae74904912d7e29095b9461c75
Submitter: Zuul
Branch: master

commit 7d3e1a6838b50eae74904912d7e29095b9461c75
Author: Ranjeet R <email address hidden>
Date: Tue Nov 8 11:37:22 2016 -0800

Fixes: VM Spawn fails in Mitaka in a HA setup

In Mitaka, the neutron section should auth_url specified as against
admin_auth_url

Change-Id: I9e11ec61280dc933ff4758f84cc623a244ce48a7
Closes-Bug: 1611566

[Atul Moghe (moghea)]

We need similar changes in compute-server-setup.sh to configure auth_url based on different VIP conditions.

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/25829
Submitter: Ranjeet R (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/25982
Submitter: Ranjeet R (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/25829
Submitter: Ranjeet R (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/25986
Submitter: Ranjeet R (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/25982
Committed: http://github.org/Juniper/contrail-provisioning/commit/c0442a5846b8907d410949d1701b9615b6ee40ac
Submitter: Zuul
Branch: master

commit c0442a5846b8907d410949d1701b9615b6ee40ac
Author: Ranjeet R <email address hidden>
Date: Thu Nov 10 21:48:44 2016 -0800

Fixes: VM Spawn fails in Mitaka in a HA setup

In Mitaka, the neutron section should auth_url specified as against
admin_auth_url. Changing both nova.conf in controller and compute.

Change-Id: I91fe332002f917ebecd003f91c50266e892e58d6
Closes-Bug: 1611566

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/25986
Committed: http://github.org/Juniper/contrail-provisioning/commit/22fe1258d07f1f5c65cc48719da84a7c94dbd8f3
Submitter: Zuul
Branch: R3.2

commit 22fe1258d07f1f5c65cc48719da84a7c94dbd8f3
Author: Ranjeet R <email address hidden>
Date: Thu Nov 10 22:42:37 2016 -0800

Fixes: VM Spawn fails in Mitaka in a HA setup

In Mitaka, the neutron section should auth_url specified as against
admin_auth_url. Changing both nova.conf in controller and compute.

Change-Id: I3abdda8f454be2c3896cd959a77308f257d021ba
Closes-Bug: 1611566

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/25829
Committed: http://github.org/Juniper/contrail-provisioning/commit/e13bc3eb5a78abc63c93bb44f0b19f3a6b1148b7
Submitter: Zuul
Branch: R3.1

commit e13bc3eb5a78abc63c93bb44f0b19f3a6b1148b7
Author: Ranjeet R <email address hidden>
Date: Tue Nov 8 11:40:27 2016 -0800

Fixes: VM Spawn fails in Mitaka in a HA setup

In Mitaka, the neutron section should auth_url specified as against
admin_auth_url. Changing both nova.conf in controller and compute.

Change-Id: I727e5e1967c3ca9e23c801d4d145950ce393194a
Closes-Bug: 1611566