tripleo

fedora_infos.containers.0.State.Healthcheck.Status): ''dict object'' has no attribute ''Healthcheck

Bug #1979810 reported by chandan kumar
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Cédric Jeanneret
tripleo zed-1

Bug Description

https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ansible-centos-stream-molecule-tripleo_container_manage&project=openstack/tripleo-ansible is failing consistently while porting it to CentOS Stream 9.

tripleo_container_manage role is tested with delegated driver and below is the failure [1].
```
TASK [Assert that tripleo_fedora systemd service is active] ********************
2022-06-24 08:44:15.732335 | centos-9-stream | ok: [instance] => changed=false
2022-06-24 08:44:15.732347 | centos-9-stream | msg: tripleo_fedora systemd service is active
2022-06-24 08:44:15.732358 | centos-9-stream |
2022-06-24 08:44:15.732369 | centos-9-stream | TASK [Check if tripleo_fedora healthcheck is active and healthy] ***************
2022-06-24 08:44:15.732380 | centos-9-stream | fatal: [instance]: FAILED! =>
2022-06-24 08:44:15.732391 | centos-9-stream | msg: 'The conditional check ''''healthy'' in fedora_infos.containers.0.State.Healthcheck.Status'' failed. The error was: error while evaluating conditional (''healthy'' in fedora_infos.containers.0.State.Healthcheck.Status): ''dict object'' has no attribute ''Healthcheck'''
```

[1]. https://5357c71da65fa095806b-bd1ccbb46e138d06309351c369577ba2.ssl.cf2.rackcdn.com/839688/82/check/tripleo-ansible-centos-stream-molecule-tripleo_container_manage/25c071d/job-output.txt

Here is the codebase at which it is failing: https://opendev.org/openstack/tripleo-ansible/src/branch/master/tripleo_ansible/roles/tripleo_container_manage/molecule/default/converge.yml#L94

It is blocking cs9 porting work. So I am capturing the failure in the bug and marking the job non-voting to unblock.

Cédric Jeanneret (cjeanner)
Changed in tripleo:
assignee: nobody → Cédric Jeanneret (cjeanner)
Revision history for this message
Cédric Jeanneret (cjeanner) wrote :

Notes:
- underlying OS is also updated, from cs8 to cs9
- this means podman has also an update

I'm suspecting a change in the way podman exposes its data.

Chandan, is there a way to know the podman version, as well as its source repository? That would make things easier in order to debug that one.

Cheers,

C.

Revision history for this message
Cédric Jeanneret (cjeanner) wrote :

podman version: podman-4.1.1-1.el9.src.rpm
From repo : appstream

Guess I'll be able to reproduce this locally.

Revision history for this message
Cédric Jeanneret (cjeanner) wrote :

Bingo

        State:
            CgroupPath: /machine.slice/libpod-8b4dae9522d92989cc7a88da3842d79a93a67716fa8494377514d59990671dbf.scope
            CheckpointedAt: '0001-01-01T00:00:00Z'
            ConmonPid: 5550
            Dead: false
            Error: ''
            ExitCode: 0
            FinishedAt: '0001-01-01T00:00:00Z'
            Health:
                FailingStreak: 0
                Log:
                - End: '2022-06-27T07:55:37.216833031-04:00'
                    ExitCode: 0
                    Output: test
                    Start: '2022-06-27T07:55:37.166444845-04:00'
                Status: healthy

It's now named Health.... "yay".

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-ansible (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ansible/+/847774

Changed in tripleo:
status: Triaged → In Progress
Revision history for this message
Cédric Jeanneret (cjeanner) wrote :

We may need this as well:
https://github.com/containers/ansible-podman-collections/pull/442

Revision history for this message
Cédric Jeanneret (cjeanner) wrote :

Closed the PR against ansible-podman-collections. It's mostly useless at this point:
the Config part of the container-inspect stills shows the "right" Healthcheck key; There's more than probably another difference nested in that block, I'm deploying wallaby against cs8 and will compare with a wallaby against cs9, for instance for the mysql container.

I'll come back once I get all the needed data, and update my tripleo-ansible patch accordingly.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-ansible (master)

Reviewed: https://review.opendev.org/c/openstack/tripleo-ansible/+/839688
Committed: https://opendev.org/openstack/tripleo-ansible/commit/4c9233d07c0886871719b46b2801c7b0565f26cc
Submitter: "Zuul (22348)"
Branch: master

commit 4c9233d07c0886871719b46b2801c7b0565f26cc
Author: Chandan Kumar (raukadah) <email address hidden>
Date: Mon May 30 09:45:42 2022 +0530

    Use ubi9-init image as base

    CentOS-8 no longer exists and existing jobs are running
    on centos-stream-8. There will be no openstack master
    rpm packages for centos-stream-8.

    In order to use ubi9-init image in molecule jobs,
    we need to include following workarounds:
    - redhat-release package(RH specific package) is installed in ubi-init
      which does not work with centos-stream package.
      On removing redhat-release, dnf ansible breaks as they did not find
      --releasever during package installation.
      - By adding releasever in the package module fixes the issue.
    - In Molecule job, we can use any hostname either of centos or ubi both
      will work.
    - It changes the nodeset and image to ubi9.
    - Install pip < 20.3.0 to avoid https://pip.pypa.io/en/stable/topics/dependency-resolution/#backtracking
      and upstream requirements file to avoid cyclic dependency
      issue.
    - Set enable_fips: true in collectd jobs as in molecule it is not
      needed otherwise job will enter into retry_limit.

    We cannot use centos:stream9 image as, init support is removed
    from there.

    Below is the list of following jobs marked non-voting
    due to following bugs.
    - tripleo-ansible-centos-stream-molecule-tripleo_update_trusted_cas
      - https://bugs.launchpad.net/tripleo/+bug/1979807
    - tripleo-ansible-centos-stream-molecule-tripleo-modules
      - https://bugs.launchpad.net/tripleo/+bug/1979807
    - tripleo-ansible-centos-stream-molecule-tripleo_container_manage
      - https://bugs.launchpad.net/tripleo/+bug/1979810
    - tripleo-ansible-centos-stream-molecule-tripleo_redhat_enforce
      - https://bugs.launchpad.net/tripleo/+bug/1979971
    - tripleo-ansible-centos-stream-molecule-tripleo_collectd
      - https://bugs.launchpad.net/tripleo/+bug/1979972
    - tripleo-ansible-centos-stream-molecule-backup_and_restore
      - https://bugs.launchpad.net/tripleo/+bug/1980198

    Related-Bug: #1979807
    Related-Bug: #1979810
    Related-Bug: #1979971
    Related-Bug: #1979972
    Related-Bug: #1980198

    Signed-off-by: Chandan Kumar (raukadah) <email address hidden>
    Change-Id: I75a27ca496fb7670ba22a6a22f62f001c8afe6a9

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-ansible (master)

Reviewed: https://review.opendev.org/c/openstack/tripleo-ansible/+/847774
Committed: https://opendev.org/openstack/tripleo-ansible/commit/b9694f360ebeb9d4d2dcd99b05b6d34f8c14828f
Submitter: "Zuul (22348)"
Branch: master

commit b9694f360ebeb9d4d2dcd99b05b6d34f8c14828f
Author: Cédric Jeanneret <email address hidden>
Date: Mon Jun 27 14:07:53 2022 +0200

    Update for new podman output

    With podman 4.1.1 (as shipped in Stream 9), the "Healthcheck" key is
    now called "Health" in `podman inspect <container>'.

    This patch therefor affects the doc, molecule testing and modules in
    order to make it aware of the new naming.

    This patch also switches back the container_manage molecule job to
    voting.

    Change-Id: Iad3054bd384f7875f111dc0a72830c9e0e9fda9a
    Closes-Bug: #1979810

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-ansible (stable/wallaby)

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-ansible/+/848810

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-ansible (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/tripleo-ansible/+/848810
Committed: https://opendev.org/openstack/tripleo-ansible/commit/b666c78954fbbe7edfd50b6b949c4921fc6c5518
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit b666c78954fbbe7edfd50b6b949c4921fc6c5518
Author: Chandan Kumar (raukadah) <email address hidden>
Date: Mon May 30 09:45:42 2022 +0530

    Use ubi9-init image as base

    CentOS-8 no longer exists and existing jobs are running
    on centos-stream-8. There will be no openstack master
    rpm packages for centos-stream-8.

    In order to use ubi9-init image in molecule jobs,
    we need to include following workarounds:
    - redhat-release package(RH specific package) is installed in ubi-init
      which does not work with centos-stream package.
      On removing redhat-release, dnf ansible breaks as they did not find
      --releasever during package installation.
      - By adding releasever in the package module fixes the issue.
    - In Molecule job, we can use any hostname either of centos or ubi both
      will work.
    - It changes the nodeset and image to ubi9.
    - Install pip < 20.3.0 to avoid https://pip.pypa.io/en/stable/topics/dependency-resolution/#backtracking
      and upstream requirements file to avoid cyclic dependency
      issue.
    - Set enable_fips: true in collectd jobs as in molecule it is not
      needed otherwise job will enter into retry_limit.

    We cannot use centos:stream9 image as, init support is removed
    from there.

    Below is the list of following jobs marked non-voting
    due to following bugs.
    - tripleo-ansible-centos-stream-molecule-tripleo_update_trusted_cas
      - https://bugs.launchpad.net/tripleo/+bug/1979807
    - tripleo-ansible-centos-stream-molecule-tripleo-modules
      - https://bugs.launchpad.net/tripleo/+bug/1979807
    - tripleo-ansible-centos-stream-molecule-tripleo_container_manage
      - https://bugs.launchpad.net/tripleo/+bug/1979810
    - tripleo-ansible-centos-stream-molecule-tripleo_redhat_enforce
      - https://bugs.launchpad.net/tripleo/+bug/1979971
    - tripleo-ansible-centos-stream-molecule-tripleo_collectd
      - https://bugs.launchpad.net/tripleo/+bug/1979972
    - tripleo-ansible-centos-stream-molecule-backup_and_restore
      - https://bugs.launchpad.net/tripleo/+bug/1980198

    Note: It is not a clean cherry pick.
    - Remove cap from molecule dependency
      to fix No module named 'ansiblelint.prerun'
    - Populate wallaby repos using tripleo-repos tool
    - This includes the commit 0bc2fa808d34235ffe7cf8d36920f6d96f0685b5
      which fixed the issue with the original change in master.
    - Marking tripleo-ansible-centos-stream-molecule-tripleo_packages
      to non-voting due to
      https://bugs.launchpad.net/tripleo/+bug/1983347

    Related-Bug: #1979807
    Related-Bug: #1979810
    Related-Bug: #1979971
    Related-Bug: #1979972
    Related-Bug: #1980198
    Related-Bug: #1983347

    Signed-off-by: Chandan Kumar (raukadah) <email address hidden>
    Change-Id: I75a27ca496fb7670ba22a6a22f62f001c8afe6a9
    (cherry picked from commit 4c9233d07c0886871719b46b2801c7b0565f26cc)

tags: added: in-stable-wallaby
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-ansible 5.0.0

This issue was fixed in the openstack/tripleo-ansible 5.0.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.