In an environment without a common L2 network across all of the mysql units (e.g. no shared L2 between racks with MAAS, or AWS VPC), mysql units will reject connections from other units, and bootstrap never completes.
$ juju bootstrap aws/ap-northeast-1
$ juju spaces
Name Space ID Subnets
alpha 0 172.31.0.0/20
172.31.16.0/20
172.31.32.0/20
252.0.0.0/12
252.16.0.0/12
252.32.0.0/12
$ juju deploy --series focal -n3 cs:~openstack-charmers-next/mysql-innodb-cluster
Located charm "cs:~openstack-charmers-next/mysql-innodb-cluster-74".
Deploying charm "cs:~openstack-charmers-next/mysql-innodb-cluster-74".
"group_replication_ip_allowlist" is set as AUTOMATIC (default). However, it assumes common L2 networks so I think we need to manage the allowlist explicitly for pure L3 environment.
mysql> SHOW GLOBAL VARIABLES LIKE 'group_replication%list';
+--------------------------------+-----------+
| Variable_name | Value |
+--------------------------------+-----------+
| group_replication_ip_allowlist | AUTOMATIC |
| group_replication_ip_whitelist | AUTOMATIC |
+--------------------------------+-----------+
2 rows in set (0.01 sec)
https://dev.mysql.com/doc/refman/8.0/en/group-replication-options.html#sysvar_group_replication_ip_allowlist
[mysql.err on leader]
2021-04-28T14:48:56.073823Z 0 [System] [MY-011507] [Repl] Plugin group_replication reported: 'A new primary with address 172.31.36.189:3306 was elected. The new primary will execute all previous group transa
ctions before allowing writes.'
2021-04-28T14:48:56.074753Z 29 [System] [MY-011566] [Repl] Plugin group_replication reported: 'Setting super_read_only=OFF.'
2021-04-28T14:48:56.074931Z 29 [System] [MY-011510] [Repl] Plugin group_replication reported: 'This server is working as primary member.'
2021-04-28T14:48:56.728594Z 12 [System] [MY-010597] [Repl] 'CHANGE MASTER TO FOR CHANNEL 'group_replication_recovery' executed'. Previous state master_host='', master_port= 3306, master_log_file='', master_l
og_pos= 4, master_bind=''. New state master_host='', master_port= 3306, master_log_file='', master_log_pos= 4, master_bind=''.
2021-04-28T14:50:33.105783Z 0 [Warning] [MY-011735] [Repl] Plugin group_replication reported: '[GCS] Connection attempt from IP address ::ffff:172.31.3.41 refused. Address is not in the IP allowlist.'
2021-04-28T14:50:33.205767Z 0 [Warning] [MY-011735] [Repl] Plugin group_replication reported: '[GCS] Connection attempt from IP address ::ffff:172.31.3.41 refused. Address is not in the IP allowlist.'
[mysql.err on another unit]
2021-04-28T14:50:32.856401Z 13 [System] [MY-013587] [Repl] Plugin group_replication reported: 'Plugin 'group_replication' is starting.'
2021-04-28T14:50:32.885916Z 16 [System] [MY-010597] [Repl] 'CHANGE MASTER TO FOR CHANNEL 'group_replication_applier' executed'. Previous state master_host='', master_port= 3306, master_log_file='', master_log_pos= 4, master_bind=''. New state master_host='<NULL>', master_port= 0, master_log_file='', master_log_pos= 4, master_bind=''.
2021-04-28T14:50:33.103868Z 0 [ERROR] [MY-011735] [Repl] Plugin group_replication reported: '[GCS] Error on opening a connection to 172.31.36.189:33061 on local port: 33061.'
2021-04-28T14:50:33.203606Z 0 [ERROR] [MY-011735] [Repl] Plugin group_replication reported: '[GCS] Error on opening a connection to 172.31.36.189:33061 on local port: 33061.'
::ffff: was a red herring. I just need to check ACL again.