Undercloud upgrade to podman fails with bind mount read errors

Bug #1802203 reported by Jill Rouleau on 2018-11-08
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cédric Jeanneret

Bug Description

Undercloud upgrade fails when switching from docker to podman as containers (at least rmq/memcached) started via podman can not read/copy files from /var/lib/config-data/puppet-generated/$service/.

type=AVC msg=audit(1541628356.826:5369): avc: denied { read } for pid=452886 comm="python" name="rabbitmq-env.conf" dev="dm-0" ino=100801360 scontext=system_u:system_r:container_t:s0:c41,c44 tcontext=unconfined_u:object_r:container_var_lib_t:s0 tclass=file

type=AVC msg=audit(1541628334.756:5302): avc: denied { read } for pid=450657 comm="bash" name="memcached" dev="dm-0" ino=102946033 scontext=system_u:system_r:container_t:s0:c143,c723 tcontext=unconfined_u:object_r:container_var_lib_t:s0 tclass=file

failure in openstack undercloud upgrade output: http://paste.openstack.org/show/734383/

[root@undercloud ~]# podman container ls -a
212e550c9dd9 docker.io/tripleomaster/centos-binary-rabbitmq:3ed8ac0e93367a02ad53d9fa93467057724b6621_fd8eb74b kolla_start 12 minutes ago Exited (2) 12 minutes ago rabbitmq
0b74f790a30d docker.io/tripleomaster/centos-binary-haproxy:3ed8ac0e93367a02ad53d9fa93467057724b6621_fd8eb74b /docker_puppet_ap... 13 minutes ago Exited (0) 12 minutes ago haproxy_firewall
fdeb43cca6ac docker.io/tripleomaster/centos-binary-rabbitmq:3ed8ac0e93367a02ad53d9fa93467057724b6621_fd8eb74b kolla_start 13 minutes ago Exited (2) 13 minutes ago rabbitmq_bootstrap
a0c4d8900973 docker.io/tripleomaster/centos-binary-haproxy:3ed8ac0e93367a02ad53d9fa93467057724b6621_fd8eb74b kolla_start 13 minutes ago Up 13 minutes ago haproxy
711c5a85e14b docker.io/tripleomaster/centos-binary-memcached:3ed8ac0e93367a02ad53d9fa93467057724b6621_fd8eb74b /bin/bash -c sour... 13 minutes ago Exited (64) 12 minutes ago memcached
7750a5469261 docker.io/tripleomaster/centos-binary-rabbitmq:3ed8ac0e93367a02ad53d9fa93467057724b6621_fd8eb74b /bin/bash -c chow... 13 minutes ago Exited (0) 13 minutes ago rabbitmq_init_logs
a6acaadcf640 docker.io/tripleomaster/centos-binary-keepalived:3ed8ac0e93367a02ad53d9fa93467057724b6621_fd8eb74b /usr/local/bin/ko... 13 minutes ago Up 13 minutes ago keepalived
8e4ff80b7818 docker.io/tripleomaster/centos-binary-mariadb:3ed8ac0e93367a02ad53d9fa93467057724b6621_fd8eb74b /bin/bash -c chow... 13 minutes ago Exited (0) 13 minutes ago mysql_init_logs

To reproduce: openstack undercloud upgrade existing deploy to latest with container_cli = docker, edit undercloud.conf to container_cli = podman and rerun upgrade.

Changed in tripleo:
importance: Undecided → High
milestone: none → stein-2
status: New → Triaged
Cédric Jeanneret (cjeanner) wrote :


Will try to work on this today. We might need some tasks in the "upgrade" part in order to ensure the directories have the correct SELinux context.

Changed in tripleo:
assignee: nobody → Cédric Jeanneret (cjeanner)
Cédric Jeanneret (cjeanner) wrote :


did you apply any patches? Apparently I can't go that far:
TASK [Ensure all online data migrations for Ironic have been applied] *************************************************************************************************************************************************************************
fatal: [undercloud]: FAILED! => {"changed": true, "cmd": ["podman", "exec", "ironic_api", "ironic-dbsync", "--config-file", "/etc/ironic/ironic.conf", "online_data_migrations"], "delta": "0:00:00.159337", "end": "2018-11-08 09:07:31.896824
", "msg": "non-zero return code", "rc": 125, "start": "2018-11-08 09:07:31.737487", "stderr": "unable to exec into ironic_api: no container with name or ID ironic_api found: no such container", "stderr_lines": ["unable to exec into ironic_
api: no container with name or ID ironic_api found: no such container"], "stdout": "", "stdout_lines": []}

PLAY RECAP ************************************************************************************************************************************************************************************************************************************
undercloud : ok=31 changed=12 unreachable=0 failed=1

Cédric Jeanneret (cjeanner) wrote :

So I think a selinux policy would not be bad for that specific case - I've opened an issue against libpod here: https://github.com/containers/libpod/issues/1782

Stay tuned!

Cédric Jeanneret (cjeanner) wrote :

Back on that one.

So, in order to reproduce this one, we have to:
- deploy an undercloud that does not involve patches for the directory creation/setype (that's old, prior August, maybe even prior June)
- update to the latest version
- migrate from docker -> podman

We might want to involve "upgrade" people for that, in order to get some thoughts from them.

For now, if I deploy an undercloud with docker, using master, I get the following setype on the directories:
drwxr-xr-x. root root unconfined_u:object_r:container_file_t:s0 /var/lib/kolla/
drwxr-xr-x. root root unconfined_u:object_r:container_file_t:s0 /var/lib/kolla/config_files/
-rw-------. root root system_u:object_r:container_file_t:s0 glance_api.json

Those rights are OK for both docker and podman. Would be really great to get a reproducer on that, Jill :).

Cédric Jeanneret (cjeanner) wrote :

So, apparently we'll need to add some upgrade task somewhere just before the podman containers are started. Either a "chcon -R", or, cleaner:
semanage fcontext -a -t container_share_t "/var/lib/kolla/config_files(/.*)?"
restorecon -R -v /var/lib/kolla

The correct location has to be defined though, with the support of the Upgrade Team :).

Changed in tripleo:
milestone: stein-2 → stein-3
Changed in tripleo:
milestone: stein-3 → stein-rc1
Changed in tripleo:
milestone: stein-rc1 → train-1
Changed in tripleo:
milestone: train-1 → train-2
Changed in tripleo:
milestone: train-2 → train-3
tags: added: upgrade
Changed in tripleo:
milestone: train-3 → ussuri-1
Changed in tripleo:
milestone: ussuri-1 → ussuri-2
wes hayutin (weshayutin) on 2020-02-10
Changed in tripleo:
milestone: ussuri-2 → ussuri-3
wes hayutin (weshayutin) on 2020-04-07
Changed in tripleo:
status: Triaged → Incomplete
wes hayutin (weshayutin) on 2020-04-13
Changed in tripleo:
milestone: ussuri-3 → ussuri-rc3
wes hayutin (weshayutin) on 2020-05-26
Changed in tripleo:
milestone: ussuri-rc3 → victoria-1
Changed in tripleo:
milestone: victoria-1 → victoria-3
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.