Upgrade CI broken: "image is being used by running container"

Bug #1831022 reported by Jiří Stránský
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Jiří Stránský

Bug Description

Likely we broke CI by landing https://review.opendev.org/#/c/651594/. The patch passed upgrade CI, but now we're hitting errors which look related to the mangling of the pacemaker stopping tasks in that patch:

2019-05-29 17:24:24 | TASK [Remove previous Mariadb images] ******************************************
2019-05-29 17:24:24 | task path: /var/lib/mistral/ef65d809-d45f-42ee-b2a2-ad3902d183d7/Controller/upgrade_tasks_step3.yaml:38
2019-05-29 17:24:24 | Wednesday 29 May 2019 17:24:23 +0000 (0:00:02.447) 0:02:46.347 *********
2019-05-29 17:24:24 | Using module file /usr/lib/python2.7/site-packages/ansible/modules/commands/command.py
2019-05-29 17:24:24 | <192.168.24.3> ESTABLISH SSH CONNECTION FOR USER: tripleo-admin
2019-05-29 17:24:24 | <192.168.24.3> SSH: EXEC ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o ControlMaster=auto -o ControlPersist=30m -o ServerAliveInterval=5 -o ServerAliveCountMax=5 -o StrictHostKeyChecking=no -o 'IdentityFile="/var/lib/mistral/ef65d809-d45f-42ee-b2a2-ad3902d183d7/ssh_private_key"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=tripleo-admin -o ConnectTimeout=30 -o ControlPath=/var/lib/mistral/ef65d809-d45f-42ee-b2a2-ad3902d183d7/ansible-ssh/82430b422b 192.168.24.3 '/bin/sh -c '"'"'sudo -H -S -n -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-tebcoczfbhdcryvifoyozigsndeftqeq; /usr/bin/python'"'"'"'"'"'"'"'"' && sleep 0'"'"''
2019-05-29 17:24:24 | Escalation succeeded
2019-05-29 17:24:24 | <192.168.24.3> (1, '\n{"changed": true, "end": "2019-05-29 17:24:23.574155", "stdout": "", "cmd": "docker rmi -f 16b688c3c351", "failed": true, "delta": "0:00:00.041786", "stderr": "Error response from daemon: conflict: unable to delete 16b688c3c351 (cannot be forced) - image is being used by running container fe7b35744674", "rc": 1, "invocation": {"module_args": {"warn": true, "executable": null, "_uses_shell": true, "_raw_params": "docker rmi -f 16b688c3c351", "removes": null, "argv": null, "creates": null, "chdir": null, "stdin": null}}, "start": "2019-05-29 17:24:23.532369", "msg": "non-zero return code"}\n', '')
2019-05-29 17:24:24 | <192.168.24.3> Failed to connect to the host via ssh:
2019-05-29 17:24:24 | fatal: [centos-7-rax-dfw-0006736479]: FAILED! => {
2019-05-29 17:24:24 | "changed": true,
2019-05-29 17:24:24 | "cmd": "docker rmi -f 16b688c3c351",
2019-05-29 17:24:24 | "delta": "0:00:00.041786",
2019-05-29 17:24:24 | "end": "2019-05-29 17:24:23.574155",
2019-05-29 17:24:24 | "invocation": {
2019-05-29 17:24:24 | "module_args": {
2019-05-29 17:24:24 | "_raw_params": "docker rmi -f 16b688c3c351",
2019-05-29 17:24:24 | "_uses_shell": true,
2019-05-29 17:24:24 | "argv": null,
2019-05-29 17:24:24 | "chdir": null,
2019-05-29 17:24:24 | "creates": null,
2019-05-29 17:24:24 | "executable": null,
2019-05-29 17:24:24 | "removes": null,
2019-05-29 17:24:24 | "stdin": null,
2019-05-29 17:24:24 | "warn": true
2019-05-29 17:24:24 | }
2019-05-29 17:24:24 | },
2019-05-29 17:24:24 | "msg": "non-zero return code",
2019-05-29 17:24:24 | "rc": 1,
2019-05-29 17:24:24 | "start": "2019-05-29 17:24:23.532369",
2019-05-29 17:24:24 | "stderr": "Error response from daemon: conflict: unable to delete 16b688c3c351 (cannot be forced) - image is being used by running container fe7b35744674",
2019-05-29 17:24:24 | "stderr_lines": [
2019-05-29 17:24:24 | "Error response from daemon: conflict: unable to delete 16b688c3c351 (cannot be forced) - image is being used by running container fe7b35744674"
2019-05-29 17:24:24 | ],
2019-05-29 17:24:24 | "stdout": "",
2019-05-29 17:24:24 | "stdout_lines": []
2019-05-29 17:24:24 | }

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (master)

Fix proposed to branch: master
Review: https://review.opendev.org/662150

Changed in tripleo:
status: Triaged → In Progress
tags: added: ci
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (master)

Reviewed: https://review.opendev.org/662150
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=f8ca0d41add4254401654c89f146d8c38a3d9ac3
Submitter: Zuul
Branch: master

commit f8ca0d41add4254401654c89f146d8c38a3d9ac3
Author: Jiri Stransky <email address hidden>
Date: Thu May 30 07:32:12 2019 +0000

    Reintroduce upgrade tasks for stopping pacemaker cluster

    It is likely that change I50a1289a864f804d02a2e2bc0ca8738a186beff0
    broke upgrade CI, even though the job somehow passed on it. MariaDB
    upgrade tasks now cannot remove the container image, because it's
    being used by a running container. Let's keep all tasks for stopping
    Pacemaker cluster for now, both untagged ones and the ones tagged for
    system_upgrade_prepare step.

    Change-Id: Ic45b74c83b99dc58cd6e0f0f45d421b88c7e97a1
    Closes-Bug: #1831022

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 11.0.0

This issue was fixed in the openstack/tripleo-heat-templates 11.0.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/667137

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/stein)

Reviewed: https://review.opendev.org/667137
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=6c60a77c1b16d848ed9e616ec5351925193c2aa3
Submitter: Zuul
Branch: stable/stein

commit 6c60a77c1b16d848ed9e616ec5351925193c2aa3
Author: Jiri Stransky <email address hidden>
Date: Thu May 30 07:32:12 2019 +0000

    Reintroduce upgrade tasks for stopping pacemaker cluster

    It is likely that change I50a1289a864f804d02a2e2bc0ca8738a186beff0
    broke upgrade CI, even though the job somehow passed on it. MariaDB
    upgrade tasks now cannot remove the container image, because it's
    being used by a running container. Let's keep all tasks for stopping
    Pacemaker cluster for now, both untagged ones and the ones tagged for
    system_upgrade_prepare step.

    Change-Id: Ic45b74c83b99dc58cd6e0f0f45d421b88c7e97a1
    Closes-Bug: #1831022
    (cherry picked from commit f8ca0d41add4254401654c89f146d8c38a3d9ac3)

tags: added: in-stable-stein
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 10.6.0

This issue was fixed in the openstack/tripleo-heat-templates 10.6.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.