After updating the percona cluster vip, nova api calls became very slow.
Looking at the nova-api-os-compute.log, we could see that nova was attempting to connect to the mysql db on the old VIP.
Troubleshooting the issue we found that in nova_api DB in the cell_mappings table the database connection of cell0 was still showing the old VIP:
=================================
Actions taken
=================================
$ juju config percona-cluster vip
10.35.102.41
$ select * from nova_api.cell_mappings;
+---------------------+---------------------+----+--------------------------------------+-------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------+
| created_at | updated_at | id | uuid | name | transport_url | database_connection |
+---------------------+---------------------+----+--------------------------------------+-------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------+
| 2018-05-09 10:07:47 | NULL | 3 | 00000000-0000-0000-0000-000000000000 | cell0 | none:/// | mysql://nova:mysql_pass@10.35.102.41/nova_cell0 |
| 2018-05-09 10:07:50 | 2019-12-05 23:47:43 | 6 | 15a8a4a7-252b-47ba-8881-17a04b814e92 | cell1 | rabbit://nova:rabbit_pass@10.35.102.104:5672,nova:rabbit_pass@10.35.102.69:5672,nova:rabbit_pass@10.35.102.79:5672/openstack | mysql+pymysql://nova:mysql_pass@10.35.102.41/nova |
+---------------------+---------------------+----+--------------------------------------+-------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------+
$ juju config percona-cluster vip=10.35.102.45
=================================
Expected Result
=================================
+---------------------+---------------------+----+--------------------------------------+-------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------+
| created_at | updated_at | id | uuid | name | transport_url | database_connection |
+---------------------+---------------------+----+--------------------------------------+-------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------+
| 2018-05-09 10:07:47 | NULL | 3 | 00000000-0000-0000-0000-000000000000 | cell0 | none:/// | mysql://nova:mysql_pass@10.35.102.45/nova_cell0 |
| 2018-05-09 10:07:50 | 2019-12-05 23:47:43 | 6 | 15a8a4a7-252b-47ba-8881-17a04b814e92 | cell1 | rabbit://nova:rabbit_pass@10.35.102.104:5672,nova:rabbit_pass@10.35.102.69:5672,nova:rabbit_pass@10.35.102.79:5672/openstack | mysql+pymysql://nova:mysql_pass@10.35.102.45/nova |
+---------------------+---------------------+----+--------------------------------------+-------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------+
=================================
Actual Result
=================================
+---------------------+---------------------+----+--------------------------------------+-------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------+
| created_at | updated_at | id | uuid | name | transport_url | database_connection |
+---------------------+---------------------+----+--------------------------------------+-------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------+
| 2018-05-09 10:07:47 | NULL | 3 | 00000000-0000-0000-0000-000000000000 | cell0 | none:/// | mysql://nova:mysql_pass@10.35.102.41/nova_cell0 |
| 2018-05-09 10:07:50 | 2019-12-05 23:47:43 | 6 | 15a8a4a7-252b-47ba-8881-17a04b814e92 | cell1 | rabbit://nova:rabbit_pass@10.35.102.104:5672,nova:rabbit_pass@10.35.102.69:5672,nova:rabbit_pass@10.35.102.79:5672/openstack | mysql+pymysql://nova:mysql_pass@10.35.102.45/nova |
+---------------------+---------------------+----+--------------------------------------+-------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------+
cell0 database connection was not updated with the new VIP
=================================
Workaround
=================================
$ update nova_api.cell_mappings set database_connection="mysql://nova:mysql_pass@10.35.102.45/nova_cell0" where id=3;
API calls started working correctly once the cached database connection value was refreshed
=================================
Charm version
=================================
Revision 339
For the record, this also happens when moving from a single mysql unit to a HA cluster.