tripleo-ansible molecule jobs are failing with Error: invalid config provided: CapAdd and privileged are mutually exclusive options'

Bug #1910970 reported by chandan kumar
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
High
Unassigned

Bug Description

I am working on merging two backport patches on stable ussuri and victoria branches against tripleo-ansible.

* https://review.opendev.org/c/openstack/tripleo-ansible/+/764360 and https://review.opendev.org/c/openstack/tripleo-ansible/+/764361

Below is the list of molecule jobs are failing with the following errors on master, victoria, ussuri also.

```
          - /etc/pki/rpm-gpg:/etc/pki/rpm-gpg
          - /opt/yum.repos.d:/etc/yum.repos.d:rw
        results_file: /home/zuul/.ansible_async/920632886046.32781
        started: 1
      msg: non-zero return code
      rc: 125
      start: '2021-01-11 07:04:17.991603'
      stderr: 'Error: invalid config provided: CapAdd and privileged are mutually exclusive options'
      stderr_lines: <omitted>
      stdout: ''
      stdout_lines: <omitted>

    PLAY RECAP *********************************************************************
    localhost : ok=6 changed=3 unreachable=0 failed=1 skipped=1 rescued=0 ignored=0

An error occurred during the test sequence action: 'create'. Cleaning up.
--> Scenario: 'default'
--> Action: 'cleanup'
Skipping, cleanup playbook not configured.
--> Scenario: 'default'
--> Action: 'destroy'

    PLAY [Destroy] *****************************************************************

    TASK [Destroy molecule instance(s)] ********************************************
    changed: [localhost] => (item={'capabilities': ['ALL'], 'command': '/sbin/init', 'dockerfile': 'Dockerfile', 'environment': {'http_proxy': "{{ lookup('env', 'http_proxy') }}", 'https_proxy': "{{ lookup('env', 'https_proxy') }}"}, 'hostname': 'ubi8', 'image': 'ubi8/ubi-init', 'name': 'ubi8', 'pkg_extras': 'python*setuptools', 'privileged': True, 'registry': {'url': 'registry.access.redhat.com'}, 'ulimits': ['host'], 'volumes': ['/etc/ci/mirror_info.sh:/etc/ci/mirror_info.sh:ro', '/etc/pki/rpm-gpg:/etc/pki/rpm-gpg', '/opt/yum.repos.d:/etc/yum.repos.d:rw']})

    TASK [Wait for instance(s) deletion to complete] *******************************
    changed: [localhost] => (item={'started': 1, 'finished': 0, 'ansible_job_id': '789622559526.32840', 'results_file': '/home/zuul/.ansible_async/789622559526.32840', 'changed': True, 'failed': False, 'item': {'capabilities': ['ALL'], 'command': '/sbin/init', 'dockerfile': 'Dockerfile', 'environment': {'http_proxy': "{{ lookup('env', 'http_proxy') }}", 'https_proxy': "{{ lookup('env', 'https_proxy') }}"}, 'hostname': 'ubi8', 'image': 'ubi8/ubi-init', 'name': 'ubi8', 'pkg_extras': 'python*setuptools', 'privileged': True, 'registry': {'url': 'registry.access.redhat.com'}, 'ulimits': ['host'], 'volumes': ['/etc/ci/mirror_info.sh:/etc/ci/mirror_info.sh:ro', '/etc/pki/rpm-gpg:/etc/pki/rpm-gpg', '/opt/yum.repos.d:/etc/yum.repos.d:rw']}, 'ansible_loop_var': 'item'})

    PLAY RECAP *********************************************************************
    localhost : ok=2 changed=2 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

```

Failed job links:
* https://af0eb05e30f4936061b6-4a299ff78fba409899c97719d91e9af9.ssl.cf2.rackcdn.com/764360/4/check/tripleo-ansible-centos-8-molecule-tripleo_image_serve/e727e6f/reports.html

* https://2a85af6f167b6dd82922-ab20a40ae0deb3e2878c195f0f6c0d73.ssl.cf2.rackcdn.com/764360/4/check/tripleo-ansible-centos-8-molecule-tripleo_ovs_dpdk/ed39b4b/reports.html

* https://09a0361b964056038e95-77898a6eec3a425de21ff1866f14fdfb.ssl.cf2.rackcdn.com/764360/4/check/tripleo-ansible-centos-8-molecule-tripleo_packages/48eb0ad/reports.html

* https://bc90d6750c30bb4706d3-111c5415117c6f7e8c7678caa672fcb9.ssl.cf1.rackcdn.com/764360/4/check/tripleo-ansible-centos-8-molecule-tripleo_podman/feacf3a/reports.html

* https://7082585bebb095b4b71f-5541c36990a7c89f041bed0b72004574.ssl.cf2.rackcdn.com/764360/4/check/tripleo-ansible-centos-8-molecule-tripleo_ptp/38ecb86/reports.html

* https://3aef5f653c0b89dd4915-58cf4a3a4fc59e9f2b77716e9b3e3ff8.ssl.cf5.rackcdn.com/764360/4/check/tripleo-ansible-centos-8-molecule-tripleo_puppet_cache/28ccc8d/reports.html

* https://9f672e0630f459ee81cb-e4093b1756a9a5a7c7d28e6575b4af7f.ssl.cf5.rackcdn.com/764360/4/check/tripleo-ansible-centos-8-molecule-tripleo_sshd/598a9d0/reports.html

* https://0bae10ce328e8dc46074-2f7fc4cc5295aef11715d5c702aba78c.ssl.cf2.rackcdn.com/764360/4/check/tripleo-ansible-centos-8-molecule-tripleo_systemd_wrapper/4bfc11c/reports.html

* https://1710be50f200489ecc9d-9f3c64f3d814a9e4d0be6167f83e8b00.ssl.cf5.rackcdn.com/764360/4/check/tripleo-ansible-centos-8-molecule-tripleo_timezone/73633ba/reports.html

* https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e7f/764360/4/check/tripleo-ansible-centos-8-molecule-tripleo_transfer/e7f5b42/reports.html

* https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_a62/764360/4/check/tripleo-ansible-centos-8-molecule-tripleo_upgrade_hiera/a62d200/reports.html

* https://4d3799361c00ee7c2de1-f89a5ef8d1bd221399a14347eddacf71.ssl.cf5.rackcdn.com/764360/4/check/tripleo-ansible-centos-8-molecule-tripleo_validations_package/a3fcb8a/reports.html

It is also failing in periodic job also.

From the logs, it is using podman-2.0.5-5.module_el8.3.0+512+b3b58dca.x86_64 is used.

Need to investigate whether it is a podman bug or some config issue.

tags: added: promotion-blocker
Revision history for this message
Sagi (Sergey) Shnaidman (sshnaidm) wrote :
Revision history for this message
Sagi (Sergey) Shnaidman (sshnaidm) wrote :

While removing caps, I hit another problem - systemd doesn't work in podman 2.0.5 inside rootless containers, opened an issue in podman: https://github.com/containers/podman/issues/8965
Seems like this problem was resolved in 2.0.6 only: https://github.com/containers/podman/commit/4da3677e3f243de1cedca11976f156f980e132dc
So all molecule jobs that involves start/stop systemd services will fail on podman 2.0.5 (using podman driver).
I see here a few workarounds:
1) To use 1.6.4 version for molecule podman driver
2) To use root containers instead of rootless, not sure it works in molecule now.

Revision history for this message
Sagi (Sergey) Shnaidman (sshnaidm) wrote :

Should be solved in patch: https://review.opendev.org/c/openstack/tripleo-ansible/+/770102

One of possible solutions is:
https://review.opendev.org/c/openstack/tripleo-ansible/+/771064
Run molecule in systemd scope user

wes hayutin (weshayutin)
Changed in tripleo:
status: Triaged → Fix Released
Revision history for this message
Michele Baldessari (michele) wrote :
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-ansible 3.0.0

This issue was fixed in the openstack/tripleo-ansible 3.0.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-ansible 2.2.0

This issue was fixed in the openstack/tripleo-ansible 2.2.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-ansible 1.5.3

This issue was fixed in the openstack/tripleo-ansible 1.5.3 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.