Comment 2 for bug 1625722

Revision history for this message
Castulo J. Martinez (castulo-martinez) wrote :

I just tried upgrading a stable/mitaka OpenStack deployment in an onMetal server to master, and it failed with these errors below, most of them related to Galera:

ASK [galera_server : Confirm service connectivity] ****************************
FAILED - RETRYING: TASK: galera_server : Confirm service connectivity (1 retries left).
changed: [infra01_galera_container-f664885c]

cmd: /usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf ping

start: 2016-09-22 00:59:21.388814

end: 2016-09-22 00:59:21.392056

delta: 0:00:00.003242

stderr: /usr/bin/mysqladmin: connect to server at 'localhost' failed
error: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111 "Connection refused")'
Check that mysqld is running and that the socket: '/var/run/mysqld/mysqld.sock' exists!

cmd: /usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf ping

start: 2016-09-22 00:59:21.388814

end: 2016-09-22 00:59:21.392056

delta: 0:00:00.003242

stderr: /usr/bin/mysqladmin: connect to server at 'localhost' failed
error: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111 "Connection refused")'
Check that mysqld is running and that the socket: '/var/run/mysqld/mysqld.sock' exists!

TASK [galera_server : Check that WSREP is ready] *******************************
skipping: [infra01_galera_container-f664885c]

...
...

TASK [galera_server : Gather mysql facts] **************************************
fatal: [infra01_galera_container-f664885c]: FAILED! => {"changed": false, "failed": true, "msg": "Mysql fact collection failed: \"ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111 \"Connection refused\")\"."}
...ignoring

msg: Mysql fact collection failed: "ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111 "Connection refused")".

msg: Mysql fact collection failed: "ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111 "Connection refused")".

TASK [galera_server : Check for cluster state failure] *************************

TASK [galera_server : Check for cluster state failure] *************************
skipping: [infra01_galera_container-f664885c] => (item=infra01_galera_container-f664885c)

...
...

TASK [galera_server : Install galera packages] *********************************
FAILED - RETRYING: TASK: galera_server : Install galera packages (5 retries left).
FAILED - RETRYING: TASK: galera_server : Install galera packages (4 retries left).
FAILED - RETRYING: TASK: galera_server : Install galera packages (3 retries left).
FAILED - RETRYING: TASK: galera_server : Install galera packages (2 retries left).
FAILED - RETRYING: TASK: galera_server : Install galera packages (1 retries left).
failed: [infra01_galera_container-f664885c] (item=[u'mariadb-client', u'mariadb-galera-server-10.0', u'galera-3', u'rsync', u'socat']) => {"cache_update_time": 0, "cache_updated": false, "failed": true, "item": ["mariadb-client", "mariadb-galera-server-10.0", "galera-3", "rsync", "socat"], "msg": "'/usr/bin/apt-get -y -o \"Dpkg::Options::=--force-confdef\" -o \"Dpkg::Options::=--force-confold\" install 'mariadb-galera-server-10.0'' failed: E: Package 'mariadb-galera-server-10.0' has no installation candidate
", "stderr": "E: Package 'mariadb-galera-server-10.0' has no installation candidate
", "stdout": "Reading package lists...
Building dependency tree...
Reading state information...
Package mariadb-galera-server-10.0 is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "Package mariadb-galera-server-10.0 is not available, but is referred to by another package.", "This may mean that the package is missing, has been obsoleted, or", "is only available from another source", ""]}

msg: One or more items failed

msg: One or more items failed

NO MORE HOSTS LEFT *************************************************************

PLAY RECAP *********************************************************************
infra01_galera_container-f664885c : ok=71 changed=23 unreachable=0 failed=1
infra02_galera_container-170b474e : ok=8 changed=5 unreachable=0 failed=0
infra03_galera_container-5487f68c : ok=8 changed=5 unreachable=0 failed=0

TASK: galera_server : Install galera packages -------------------------- 16.56s
TASK: Lxc container restart --------------------------------------------- 8.42s
TASK: Wait for container ssh -------------------------------------------- 5.66s
TASK: pip_install : Update apt if needed -------------------------------- 4.08s
TASK: galera_server : Confirm service connectivity ---------------------- 3.60s
TASK: galera_client : Add galera repo(s) -------------------------------- 3.20s
TASK: galera_server : Add galera repo ----------------------------------- 3.02s
TASK: pip_install : Install PIP ----------------------------------------- 2.94s
TASK: galera_server : UN-Install galera-server package ------------------ 2.92s
TASK: galera_server : Preseed galera password(s) ------------------------ 2.18s
ran galera-install.yml -e 'galera_upgrade=true'
******************** failure ********************
The upgrade script has encountered a failure.
Failed on task "galera-install.yml -e 'galera_upgrade=true'"
Re-run the run-upgrade.sh script, or
execute the remaining tasks manually:
openstack-ansible galera-install.yml -e 'galera_upgrade=true'
openstack-ansible /opt/openstack-ansible/scripts/upgrade-utilities/playbooks/galera-cluster-rolling-restart.yml
openstack-ansible memcached-install.yml
openstack-ansible rabbitmq-install.yml -e 'rabbitmq_upgrade=true'
openstack-ansible utility-install.yml
openstack-ansible rsyslog-install.yml
openstack-ansible /opt/openstack-ansible/scripts/upgrade-utilities/playbooks/memcached-flush.yml
openstack-ansible /opt/openstack-ansible/scripts/upgrade-utilities/playbooks/aodh-api-init-delete.yml
openstack-ansible setup-openstack.yml
******************** failure ********************

I´m not sure if these Galera issues are related to the problem described above.