MAAS scenario deploys MySQL per service, 21 MySQL instances out of the box

Bug #2066585 reported by Nobuto Murata
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Snap
New
Medium
Unassigned

Bug Description

By following the tutorial:
https://microstack.run/docs/multi-node-maas

3 MySQL instances (for HA) per service will be deployed as follows. It comes with 21 MySQL instances out of the box, and more to be added when adding more services like octavia for example.

Splitting database into per-service might be a good idea for scalability and to be more microservice-y. However, those 21 instances are running on the same set of physical machines so it doesn't give many benefits in terms of performance.

Moreover, this micro DB everywhere approach comes with operational burden. An operator has to take a backup of all databases instead of one, and I don't think OpenStack allows inconsistency between those data bases. Meaning OpenStack can work with separate database per service in normal circumstances, but snapshot of the database must be consistent across all of the databases otherwise Nova, Neutron, and Cinder can disagree about the status of a VM (Nova might think the VM is deleted, but Neutron and Cinder might think the resources are still used, etc.).

cinder-mysql/0
cinder-mysql/1*
cinder-mysql/2
glance-mysql/0
glance-mysql/1
glance-mysql/2
horizon-mysql/0
horizon-mysql/1
horizon-mysql/2
keystone-mysql/0
keystone-mysql/1
keystone-mysql/2
neutron-mysql/0
neutron-mysql/1
neutron-mysql/2
nova-mysql/0
nova-mysql/1
nova-mysql/2
placement-mysql/0
placement-mysql/1
placement-mysql/2*

Tags: backup
Revision history for this message
Andre Ruiz (andre-ruiz) wrote :

The first instance of each database should indeed be running on the first node as per the bootstrap process. But once you add more machines and (only at the end) run a resize, it should spread the other ones on the extra nodes. Not sure if the process for that is already in place, but I understand that it is an intended feature.

There are many cases where it will be unbalanced (right after a bootstrap, or when you lose or reboot a node it will come back empty, etc.). This will need to be an active process that monitors and re-balance on demand.

Nobuto Murata (nobuto)
description: updated
Revision history for this message
Nobuto Murata (nobuto) wrote :

It looks like this many mysql containers are killing the underlying microk8s due to resource congestion.
https://bugs.launchpad.net/snap-openstack/+bug/2067451

Revision history for this message
Nobuto Murata (nobuto) wrote :
Revision history for this message
James Page (james-page) wrote :

MAAS deployments should get better distribution out of the box for deployments with > 3 control plane machines.

Appreciate the consistency for backups - we need to think that one through - however many MySQL's is part of the scaling approach for a larger cloud as well so I'm loath to move away from this.

Revision history for this message
James Page (james-page) wrote :

I can see how we might get some orphaned objects if the backup schedule is not consistent between services but I'm not sure of the impact this would have - needs investigation.

tags: added: backup
Changed in snap-openstack:
importance: Undecided → Medium
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.