| 2019-07-15 07:54:35 |
Sagi (Sergey) Shnaidman |
description |
After docker-rm role moved to tripleo-ansible repo and was converged with container-rm role, standalone upgrades job started to fail.
http://logs.openstack.org/76/670276/2/check/tripleo-ci-centos-7-standalone-upgrade/c659cd3/logs/undercloud/home/zuul/standalone_upgrade.log.txt.gz#_2019-07-12_06_12_16
2019-07-12 06:12:15 | 2019-07-12 06:12:15.824 108749 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] TASK [Check and upgrade Mysql database after major version upgrade] ************[00m
2019-07-12 06:12:16 | 2019-07-12 06:12:16.083 108749 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] fatal: [standalone]: FAILED! => {"changed": true, "cmd": ["podman", "exec", "-u", "root", "mysql", "mysql_upgrade"], "delta": "0:00:00.081067", "end": "2019-07-12 06:12:16.059565", "msg": "non-zero return code", "rc": 125, "start": "2019-07-12 06:12:15.978498", "stderr": "Error: unable to exec into mysql: no container with name or ID mysql found: no such container", "stderr_lines": ["Error: unable to exec into mysql: no container with name or ID mysql found: no such container"], "stdout": "", "stdout_lines": []}[00m
2019-07-12 06:12:16 | 2019-07-12 06:12:16.085 108749 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] [00m
2019-07-12 06:12:16 | 2019-07-12 06:12:16.085 108749 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] NO MORE HOSTS LEFT *************************************************************[00m
2019-07-12 06:12:16 | 2019-07-12 06:12:16.087 108749 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] [00m
2019-07-12 06:12:16 | 2019-07-12 06:12:16.088 108749 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] PLAY RECAP *********************************************************************[00m
2019-07-12 06:12:16 | 2019-07-12 06:12:16.088 108749 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] standalone : ok=414 changed=162 unreachable=0 failed=1 skipped=155 rescued=0 ignored=0[00m
2019-07-12 06:12:16 | 2019-07-12 06:12:16.088 108749 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] [00m
2019-07-12 06:12:16 | 2019-07-12 06:12:16.247 108749 ERROR tripleoclient.v1.tripleo_upgrade.Upgrade [-] Exception: Post Upgrade failed: DeploymentError: Post Upgrade failed[00m
2019-07-12 06:12:16 | Traceback (most recent call last):
2019-07-12 06:12:16 | File "/usr/lib/python2.7/site-packages/tripleoclient/v1/tripleo_deploy.py", line 1292, in _standalone_deploy
2019-07-12 06:12:16 | raise exceptions.DeploymentError('Post Upgrade failed')
2019-07-12 06:12:16 | DeploymentError: Post Upgrade failed
2019-07-12 06:12:16 | 2019-07-12 06:12:16.279 108749 ERROR tripleoclient.v1.tripleo_upgrade.Upgrade [-] None: DeploymentError: Post Upgrade failed[00m
2019-07-12 06:12:18 | 2019-07-12 06:12:18.054 108749 ERROR tripleoclient.v1.tripleo_upgrade.Upgrade [-] ** Found ansible errors for standalone deployment! **: DeploymentError: Post Upgrade failed[00m
2019-07-12 06:12:18 | 2019-07-12 06:12:18.055 108749 ERROR tripleoclient.v1.tripleo_upgrade.Upgrade [-] [
In task:
- when: step|int == 1
import_role:
name: tripleo-docker-rm
vars:
containers_to_rm:
- mysql
- name: Check and upgrade Mysql database after major version upgrade
command: "{{ container_cli }} exec -u root mysql mysql_upgrade"
when: step|int == 2
https://github.com/openstack/tripleo-heat-templates/blob/master/deployment/database/mysql-container-puppet.yaml#L327
Before that containers removing role had condition "container_cli == 'podman'":
https://github.com/openstack/tripleo-common/blob/59f823175c29726b3cca08ea430f4b356528e434/roles/tripleo-docker-rm/tasks/main.yaml#L19
Tripleo container-rm role that supposed to work for both docker and podman didn't have such condition:
https://github.com/openstack/tripleo-common/blob/59f823175c29726b3cca08ea430f4b356528e434/roles/tripleo-container-rm/tasks/docker.yaml#L18
And seems like mysql container is removed before upgrade and it's not found later. |
After docker-rm role moved to tripleo-ansible repo and was converged with container-rm role, standalone upgrades job started to fail.
http://logs.openstack.org/76/670276/2/check/tripleo-ci-centos-7-standalone-upgrade/c659cd3/logs/undercloud/home/zuul/standalone_upgrade.log.txt.gz#_2019-07-12_06_12_16
2019-07-12 06:12:15 | 2019-07-12 06:12:15.824 108749 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] TASK [Check and upgrade Mysql database after major version upgrade] ************[00m
2019-07-12 06:12:16 | 2019-07-12 06:12:16.083 108749 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] fatal: [standalone]: FAILED! => {"changed": true, "cmd": ["podman", "exec", "-u", "root", "mysql", "mysql_upgrade"], "delta": "0:00:00.081067", "end": "2019-07-12 06:12:16.059565", "msg": "non-zero return code", "rc": 125, "start": "2019-07-12 06:12:15.978498", "stderr": "Error: unable to exec into mysql: no container with name or ID mysql found: no such container", "stderr_lines": ["Error: unable to exec into mysql: no container with name or ID mysql found: no such container"], "stdout": "", "stdout_lines": []}[00m
2019-07-12 06:12:16 | 2019-07-12 06:12:16.085 108749 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] [00m
2019-07-12 06:12:16 | 2019-07-12 06:12:16.085 108749 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] NO MORE HOSTS LEFT *************************************************************[00m
2019-07-12 06:12:16 | 2019-07-12 06:12:16.087 108749 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] [00m
2019-07-12 06:12:16 | 2019-07-12 06:12:16.088 108749 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] PLAY RECAP *********************************************************************[00m
2019-07-12 06:12:16 | 2019-07-12 06:12:16.088 108749 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] standalone : ok=414 changed=162 unreachable=0 failed=1 skipped=155 rescued=0 ignored=0[00m
2019-07-12 06:12:16 | 2019-07-12 06:12:16.088 108749 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] [00m
2019-07-12 06:12:16 | 2019-07-12 06:12:16.247 108749 ERROR tripleoclient.v1.tripleo_upgrade.Upgrade [-] Exception: Post Upgrade failed: DeploymentError: Post Upgrade failed[00m
2019-07-12 06:12:16 | Traceback (most recent call last):
2019-07-12 06:12:16 | File "/usr/lib/python2.7/site-packages/tripleoclient/v1/tripleo_deploy.py", line 1292, in _standalone_deploy
2019-07-12 06:12:16 | raise exceptions.DeploymentError('Post Upgrade failed')
2019-07-12 06:12:16 | DeploymentError: Post Upgrade failed
2019-07-12 06:12:16 | 2019-07-12 06:12:16.279 108749 ERROR tripleoclient.v1.tripleo_upgrade.Upgrade [-] None: DeploymentError: Post Upgrade failed[00m
2019-07-12 06:12:18 | 2019-07-12 06:12:18.054 108749 ERROR tripleoclient.v1.tripleo_upgrade.Upgrade [-] ** Found ansible errors for standalone deployment! **: DeploymentError: Post Upgrade failed[00m
2019-07-12 06:12:18 | 2019-07-12 06:12:18.055 108749 ERROR tripleoclient.v1.tripleo_upgrade.Upgrade [-] [
In task:
- when: step|int == 1
import_role:
name: tripleo-docker-rm
vars:
containers_to_rm:
- mysql
- name: Check and upgrade Mysql database after major version upgrade
command: "{{ container_cli }} exec -u root mysql mysql_upgrade"
when: step|int == 2
https://github.com/openstack/tripleo-heat-templates/blob/master/deployment/database/mysql-container-puppet.yaml#L327
Before that containers removing role had condition "container_cli == 'podman'":
https://github.com/openstack/tripleo-common/blob/59f823175c29726b3cca08ea430f4b356528e434/roles/tripleo-docker-rm/tasks/main.yaml#L19
Tripleo container-rm role that supposed to work for both docker and podman didn't have such condition:
https://github.com/openstack/tripleo-common/blob/59f823175c29726b3cca08ea430f4b356528e434/roles/tripleo-container-rm/tasks/docker.yaml#L18
And seems like mysql container is removed before upgrade and it's not found later.
P.S. Now we have only container-rm role in tripleo-ansible repo, where "docker-rm" it's just a link to it. |
|
