I just tried upgrading a stable/mitaka OpenStack deployment in an onMetal server to master, and it failed with these errors below, most of them related to Galera:
stderr: /usr/bin/mysqladmin: connect to server at 'localhost' failed
error: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111 "Connection refused")'
Check that mysqld is running and that the socket: '/var/run/mysqld/mysqld.sock' exists!
stderr: /usr/bin/mysqladmin: connect to server at 'localhost' failed
error: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111 "Connection refused")'
Check that mysqld is running and that the socket: '/var/run/mysqld/mysqld.sock' exists!
TASK [galera_server : Check that WSREP is ready] *******************************
skipping: [infra01_galera_container-f664885c]
...
...
TASK [galera_server : Gather mysql facts] **************************************
fatal: [infra01_galera_container-f664885c]: FAILED! => {"changed": false, "failed": true, "msg": "Mysql fact collection failed: \"ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111 \"Connection refused\")\"."}
...ignoring
msg: Mysql fact collection failed: "ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111 "Connection refused")".
msg: Mysql fact collection failed: "ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111 "Connection refused")".
TASK [galera_server : Check for cluster state failure] *************************
TASK [galera_server : Check for cluster state failure] *************************
skipping: [infra01_galera_container-f664885c] => (item=infra01_galera_container-f664885c)
...
...
TASK [galera_server : Install galera packages] *********************************
FAILED - RETRYING: TASK: galera_server : Install galera packages (5 retries left).
FAILED - RETRYING: TASK: galera_server : Install galera packages (4 retries left).
FAILED - RETRYING: TASK: galera_server : Install galera packages (3 retries left).
FAILED - RETRYING: TASK: galera_server : Install galera packages (2 retries left).
FAILED - RETRYING: TASK: galera_server : Install galera packages (1 retries left).
failed: [infra01_galera_container-f664885c] (item=[u'mariadb-client', u'mariadb-galera-server-10.0', u'galera-3', u'rsync', u'socat']) => {"cache_update_time": 0, "cache_updated": false, "failed": true, "item": ["mariadb-client", "mariadb-galera-server-10.0", "galera-3", "rsync", "socat"], "msg": "'/usr/bin/apt-get -y -o \"Dpkg::Options::=--force-confdef\" -o \"Dpkg::Options::=--force-confold\" install 'mariadb-galera-server-10.0'' failed: E: Package 'mariadb-galera-server-10.0' has no installation candidate
", "stderr": "E: Package 'mariadb-galera-server-10.0' has no installation candidate
", "stdout": "Reading package lists...
Building dependency tree...
Reading state information...
Package mariadb-galera-server-10.0 is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "Package mariadb-galera-server-10.0 is not available, but is referred to by another package.", "This may mean that the package is missing, has been obsoleted, or", "is only available from another source", ""]}
msg: One or more items failed
msg: One or more items failed
NO MORE HOSTS LEFT *************************************************************
I just tried upgrading a stable/mitaka OpenStack deployment in an onMetal server to master, and it failed with these errors below, most of them related to Galera:
ASK [galera_server : Confirm service connectivity] ******* ******* ******* ******* galera_ container- f664885c]
FAILED - RETRYING: TASK: galera_server : Confirm service connectivity (1 retries left).
changed: [infra01_
cmd: /usr/bin/mysqladmin --defaults- file=/etc/ mysql/debian. cnf ping
start: 2016-09-22 00:59:21.388814
end: 2016-09-22 00:59:21.392056
delta: 0:00:00.003242
stderr: /usr/bin/ mysqladmin: connect to server at 'localhost' failed mysqld/ mysqld. sock' (111 "Connection refused")' mysqld/ mysqld. sock' exists!
error: 'Can't connect to local MySQL server through socket '/var/run/
Check that mysqld is running and that the socket: '/var/run/
cmd: /usr/bin/mysqladmin --defaults- file=/etc/ mysql/debian. cnf ping
start: 2016-09-22 00:59:21.388814
end: 2016-09-22 00:59:21.392056
delta: 0:00:00.003242
stderr: /usr/bin/ mysqladmin: connect to server at 'localhost' failed mysqld/ mysqld. sock' (111 "Connection refused")' mysqld/ mysqld. sock' exists!
error: 'Can't connect to local MySQL server through socket '/var/run/
Check that mysqld is running and that the socket: '/var/run/
TASK [galera_server : Check that WSREP is ready] ******* ******* ******* ******* *** galera_ container- f664885c]
skipping: [infra01_
...
...
TASK [galera_server : Gather mysql facts] ******* ******* ******* ******* ******* *** galera_ container- f664885c] : FAILED! => {"changed": false, "failed": true, "msg": "Mysql fact collection failed: \"ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111 \"Connection refused\")\"."}
fatal: [infra01_
...ignoring
msg: Mysql fact collection failed: "ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111 "Connection refused")".
msg: Mysql fact collection failed: "ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111 "Connection refused")".
TASK [galera_server : Check for cluster state failure] ******* ******* ******* ****
TASK [galera_server : Check for cluster state failure] ******* ******* ******* **** galera_ container- f664885c] => (item=infra01_ galera_ container- f664885c)
skipping: [infra01_
...
...
TASK [galera_server : Install galera packages] ******* ******* ******* ******* ***** galera_ container- f664885c] (item=[ u'mariadb- client' , u'mariadb- galera- server- 10.0', u'galera-3', u'rsync', u'socat']) => {"cache_ update_ time": 0, "cache_updated": false, "failed": true, "item": ["mariadb-client", "mariadb- galera- server- 10.0", "galera-3", "rsync", "socat"], "msg": "'/usr/bin/apt-get -y -o \"Dpkg: :Options: :=--force- confdef\ " -o \"Dpkg: :Options: :=--force- confold\ " install 'mariadb- galera- server- 10.0'' failed: E: Package 'mariadb- galera- server- 10.0' has no installation candidate galera- server- 10.0' has no installation candidate galera- server- 10.0 is not available, but is referred to by another package.
FAILED - RETRYING: TASK: galera_server : Install galera packages (5 retries left).
FAILED - RETRYING: TASK: galera_server : Install galera packages (4 retries left).
FAILED - RETRYING: TASK: galera_server : Install galera packages (3 retries left).
FAILED - RETRYING: TASK: galera_server : Install galera packages (2 retries left).
FAILED - RETRYING: TASK: galera_server : Install galera packages (1 retries left).
failed: [infra01_
", "stderr": "E: Package 'mariadb-
", "stdout": "Reading package lists...
Building dependency tree...
Reading state information...
Package mariadb-
This may mean that the package is missing, has been obsoleted, or
is only available from another source
", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "Package mariadb- galera- server- 10.0 is not available, but is referred to by another package.", "This may mean that the package is missing, has been obsoleted, or", "is only available from another source", ""]}
msg: One or more items failed
msg: One or more items failed
NO MORE HOSTS LEFT ******* ******* ******* ******* ******* ******* ******* ******* *****
PLAY RECAP ******* ******* ******* ******* ******* ******* ******* ******* ******* ****** galera_ container- f664885c : ok=71 changed=23 unreachable=0 failed=1 galera_ container- 170b474e : ok=8 changed=5 unreachable=0 failed=0 galera_ container- 5487f68c : ok=8 changed=5 unreachable=0 failed=0
infra01_
infra02_
infra03_
TASK: galera_server : Install galera packages ------- ------- ------- ----- 16.56s ------- ------- ------- ------- ------- --- 8.42s ------- ------- ------- ------- ------- -- 5.66s ------- ------- ------- ---- 4.08s ------- ------- - 3.60s ------- ------- ------- ---- 3.20s ------- ------- ------- ------- 3.02s ------- ------- ------- ------- ------ 2.94s ------- ------- --- 2.18s upgrade= true' ******* ****** failure ******* ******* ****** upgrade= true'" upgrade= true' ansible/ scripts/ upgrade- utilities/ playbooks/ galera- cluster- rolling- restart. yml install. yml install. yml -e 'rabbitmq_ upgrade= true' ansible/ scripts/ upgrade- utilities/ playbooks/ memcached- flush.yml ansible/ scripts/ upgrade- utilities/ playbooks/ aodh-api- init-delete. yml ******* ****** failure ******* ******* ******
TASK: Lxc container restart -------
TASK: Wait for container ssh -------
TASK: pip_install : Update apt if needed -------
TASK: galera_server : Confirm service connectivity -------
TASK: galera_client : Add galera repo(s) -------
TASK: galera_server : Add galera repo -------
TASK: pip_install : Install PIP -------
TASK: galera_server : UN-Install galera-server package ------------------ 2.92s
TASK: galera_server : Preseed galera password(s) -------
ran galera-install.yml -e 'galera_
*******
The upgrade script has encountered a failure.
Failed on task "galera-install.yml -e 'galera_
Re-run the run-upgrade.sh script, or
execute the remaining tasks manually:
openstack-ansible galera-install.yml -e 'galera_
openstack-ansible /opt/openstack-
openstack-ansible memcached-
openstack-ansible rabbitmq-
openstack-ansible utility-install.yml
openstack-ansible rsyslog-install.yml
openstack-ansible /opt/openstack-
openstack-ansible /opt/openstack-
openstack-ansible setup-openstack.yml
*******
I´m not sure if these Galera issues are related to the problem described above.