kayobe overcloud service destroy "Check whether inspection store volumes are present" fails if volumes are not present

Bug #2050092 reported by Martin Ananda Boeker
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kayobe
Fix Released
Medium
Unassigned
Antelope
Fix Released
Medium
Unassigned
Bobcat
Fix Released
Medium
Unassigned
Caracal
Fix Released
Medium
Unassigned
Yoga
Fix Committed
Medium
Unassigned
Zed
Fix Committed
Medium
Unassigned

Bug Description

Kayobe 14.1, OpenStack 2023.1

Had a test environment spun up, and wanted to wipe it again. Ran service destroy, and it failed at this point:

TASK [inspection-store : Check whether inspection store volumes are present] ***********************************************************************************************************************************************************
task path: /home/ubuntu/venvs/kayobe/share/kayobe/ansible/roles/inspection-store/tasks/destroy.yml:8
Monday 22 January 2024 13:06:20 +0100 (0:00:00.629) 0:00:02.088 ********
skipping: [control0] => (item=[{'container_name': 'inspection_store', 'enabled': False, 'image': 'nginx:stable', 'network_mode': 'host'}, '/etc/localtime:/etc/localtime:ro']) => changed=false
  ansible_loop_var: item
  item:
  - container_name: inspection_store
    enabled: false
    image: nginx:stable
    network_mode: host
  - /etc/localtime:/etc/localtime:ro
  skip_reason: Conditional result was False
skipping: [control0] => (item=[{'container_name': 'inspection_store', 'enabled': False, 'image': 'nginx:stable', 'network_mode': 'host'}, '/home/ubuntu/etc/inspection-store/nginx.conf:/etc/nginx/nginx.conf:ro']) => changed=false
  ansible_loop_var: item
  item:
  - container_name: inspection_store
    enabled: false
    image: nginx:stable
    network_mode: host
  - /home/ubuntu/etc/inspection-store/nginx.conf:/etc/nginx/nginx.conf:ro
  skip_reason: Conditional result was False
failed: [control0] (item=[{'container_name': 'inspection_store', 'enabled': False, 'image': 'nginx:stable', 'network_mode': 'host'}, 'inspection_store:/data']) => changed=false
  ansible_loop_var: item
  cmd:
  - docker
  - volume
  - inspect
  - inspection_store
  delta: '0:00:00.017810'
  end: '2024-01-22 13:06:37.246449'
  failed_when_result: true
  item:
  - container_name: inspection_store
    enabled: false
    image: nginx:stable
    network_mode: host
  - inspection_store:/data
  msg: non-zero return code
  rc: 1
  start: '2024-01-22 13:06:37.228639'
  stderr: 'Error response from daemon: get inspection_store: no such volume'
  stderr_lines: <omitted>
  stdout: '[]'
  stdout_lines: <omitted>

When I ssh to control0 and run `docker ps` or `docker volume ls` I see that no containers are running and no volumes are shown. Since this task is supposed to see what volumes are present, it really should not be failing if the answer is 'none'.

Revision history for this message
Will Szumski (willjs) wrote :

Yikes, good catch again. I guess we've only tested this when overcloud service deploy has run to completion. That said, this is one of the final playbooks and I imagine everything will have been destroyed by the point of this failure (we should definitely fix the bug though)

Will Szumski (willjs)
Changed in kayobe:
importance: Undecided → Medium
Revision history for this message
Martin Ananda Boeker (mboeker) wrote :

So the provision playbook did finish, even post configure and all that. I had even created networks and uploaded an image via horizon, and launched a VM. But I wanted to reconfigure some things from scratch, and instead of doing deprovision and then provision, which is very time consuming especially if you have hosts that take 4.5min to POST, I tried just service destroy. It took down the controllers, failed with nova_libvirt (other bug report here), and then after I manually killed qemu in the computes it cleared those up to this point.

Will Szumski (willjs)
Changed in kayobe:
status: New → Triaged
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kayobe (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/kayobe/+/909594

Changed in kayobe:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kayobe (master)

Reviewed: https://review.opendev.org/c/openstack/kayobe/+/909594
Committed: https://opendev.org/openstack/kayobe/commit/27ece2622b934eb6f4d9712d1879ac9bab8d236e
Submitter: "Zuul (22348)"
Branch: master

commit 27ece2622b934eb6f4d9712d1879ac9bab8d236e
Author: Will Szumski <email address hidden>
Date: Tue Feb 20 15:30:56 2024 +0000

    Fix issue removing docker volumes

    The error response from the docker daemon has changed to:

        Error response from daemon: get opensm_logs: no such volume

    This change should support both old and new responses.

    Change-Id: Ibef1713d7de2e0a2c6fe046f607b2a8e60dae3da
    Closes-Bug: #2050092

Changed in kayobe:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kayobe (stable/2023.2)

Fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/kayobe/+/920126

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kayobe (stable/2023.1)

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/kayobe/+/920127

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kayobe (unmaintained/zed)

Fix proposed to branch: unmaintained/zed
Review: https://review.opendev.org/c/openstack/kayobe/+/920128

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kayobe (unmaintained/yoga)

Fix proposed to branch: unmaintained/yoga
Review: https://review.opendev.org/c/openstack/kayobe/+/920129

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kayobe (stable/2023.2)

Reviewed: https://review.opendev.org/c/openstack/kayobe/+/920126
Committed: https://opendev.org/openstack/kayobe/commit/16147f173ffe253acf6392c8f674f716434cb07d
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit 16147f173ffe253acf6392c8f674f716434cb07d
Author: Will Szumski <email address hidden>
Date: Tue Feb 20 15:30:56 2024 +0000

    Fix issue removing docker volumes

    The error response from the docker daemon has changed to:

        Error response from daemon: get opensm_logs: no such volume

    This change should support both old and new responses.

    Change-Id: Ibef1713d7de2e0a2c6fe046f607b2a8e60dae3da
    Closes-Bug: #2050092
    (cherry picked from commit 27ece2622b934eb6f4d9712d1879ac9bab8d236e)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kayobe (stable/2023.1)

Reviewed: https://review.opendev.org/c/openstack/kayobe/+/920127
Committed: https://opendev.org/openstack/kayobe/commit/d0062df243a1e3489954025e99addd181f6a7c09
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit d0062df243a1e3489954025e99addd181f6a7c09
Author: Will Szumski <email address hidden>
Date: Tue Feb 20 15:30:56 2024 +0000

    Fix issue removing docker volumes

    The error response from the docker daemon has changed to:

        Error response from daemon: get opensm_logs: no such volume

    This change should support both old and new responses.

    Change-Id: Ibef1713d7de2e0a2c6fe046f607b2a8e60dae3da
    Closes-Bug: #2050092
    (cherry picked from commit 27ece2622b934eb6f4d9712d1879ac9bab8d236e)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kayobe (unmaintained/zed)

Reviewed: https://review.opendev.org/c/openstack/kayobe/+/920128
Committed: https://opendev.org/openstack/kayobe/commit/cbb9a22f82baaf9a04593dfa435d5288cb4f856a
Submitter: "Zuul (22348)"
Branch: unmaintained/zed

commit cbb9a22f82baaf9a04593dfa435d5288cb4f856a
Author: Will Szumski <email address hidden>
Date: Tue Feb 20 15:30:56 2024 +0000

    Fix issue removing docker volumes

    The error response from the docker daemon has changed to:

        Error response from daemon: get opensm_logs: no such volume

    This change should support both old and new responses.

    Change-Id: Ibef1713d7de2e0a2c6fe046f607b2a8e60dae3da
    Closes-Bug: #2050092
    (cherry picked from commit 27ece2622b934eb6f4d9712d1879ac9bab8d236e)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kayobe (unmaintained/yoga)

Reviewed: https://review.opendev.org/c/openstack/kayobe/+/920129
Committed: https://opendev.org/openstack/kayobe/commit/0c722dc49ca74835b274d674217d1c455c1d23c2
Submitter: "Zuul (22348)"
Branch: unmaintained/yoga

commit 0c722dc49ca74835b274d674217d1c455c1d23c2
Author: Will Szumski <email address hidden>
Date: Tue Feb 20 15:30:56 2024 +0000

    Fix issue removing docker volumes

    The error response from the docker daemon has changed to:

        Error response from daemon: get opensm_logs: no such volume

    This change should support both old and new responses.

    Change-Id: Ibef1713d7de2e0a2c6fe046f607b2a8e60dae3da
    Closes-Bug: #2050092
    (cherry picked from commit 27ece2622b934eb6f4d9712d1879ac9bab8d236e)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kayobe 16.0.0.0rc1

This issue was fixed in the openstack/kayobe 16.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kayobe 14.6.0

This issue was fixed in the openstack/kayobe 14.6.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kayobe 15.4.0

This issue was fixed in the openstack/kayobe 15.4.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.