Undercloud upgrade to podman fails with bind mount read errors

[Jill Rouleau]

Undercloud upgrade fails when switching from docker to podman as containers (at least rmq/memcached) started via podman can not read/copy files from /var/lib/config-data/puppet-generated/$service/.

type=AVC msg=audit(1541628356.826:5369): avc: denied { read } for pid=452886 comm="python" name="rabbitmq-env.conf" dev="dm-0" ino=100801360 scontext=system_u:system_r:container_t:s0:c41,c44 tcontext=unconfined_u:object_r:container_var_lib_t:s0 tclass=file

type=AVC msg=audit(1541628334.756:5302): avc: denied { read } for pid=450657 comm="bash" name="memcached" dev="dm-0" ino=102946033 scontext=system_u:system_r:container_t:s0:c143,c723 tcontext=unconfined_u:object_r:container_var_lib_t:s0 tclass=file

failure in openstack undercloud upgrade output: http://paste.openstack.org/show/734383/

[root@undercloud ~]# podman container ls -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
212e550c9dd9 docker.io/tripleomaster/centos-binary-rabbitmq:3ed8ac0e93367a02ad53d9fa93467057724b6621_fd8eb74b kolla_start 12 minutes ago Exited (2) 12 minutes ago rabbitmq
0b74f790a30d docker.io/tripleomaster/centos-binary-haproxy:3ed8ac0e93367a02ad53d9fa93467057724b6621_fd8eb74b /docker_puppet_ap... 13 minutes ago Exited (0) 12 minutes ago haproxy_firewall
fdeb43cca6ac docker.io/tripleomaster/centos-binary-rabbitmq:3ed8ac0e93367a02ad53d9fa93467057724b6621_fd8eb74b kolla_start 13 minutes ago Exited (2) 13 minutes ago rabbitmq_bootstrap
a0c4d8900973 docker.io/tripleomaster/centos-binary-haproxy:3ed8ac0e93367a02ad53d9fa93467057724b6621_fd8eb74b kolla_start 13 minutes ago Up 13 minutes ago haproxy
711c5a85e14b docker.io/tripleomaster/centos-binary-memcached:3ed8ac0e93367a02ad53d9fa93467057724b6621_fd8eb74b /bin/bash -c sour... 13 minutes ago Exited (64) 12 minutes ago memcached
7750a5469261 docker.io/tripleomaster/centos-binary-rabbitmq:3ed8ac0e93367a02ad53d9fa93467057724b6621_fd8eb74b /bin/bash -c chow... 13 minutes ago Exited (0) 13 minutes ago rabbitmq_init_logs
a6acaadcf640 docker.io/tripleomaster/centos-binary-keepalived:3ed8ac0e93367a02ad53d9fa93467057724b6621_fd8eb74b /usr/local/bin/ko... 13 minutes ago Up 13 minutes ago keepalived
8e4ff80b7818 docker.io/tripleomaster/centos-binary-mariadb:3ed8ac0e93367a02ad53d9fa93467057724b6621_fd8eb74b /bin/bash -c chow... 13 minutes ago Exited (0) 13 minutes ago mysql_init_logs

To reproduce: openstack undercloud upgrade existing deploy to latest with container_cli = docker, edit undercloud.conf to container_cli = podman and rerun upgrade.

[Cédric Jeanneret (cjeanner)]

Hey,

Will try to work on this today. We might need some tasks in the "upgrade" part in order to ensure the directories have the correct SELinux context.

[Cédric Jeanneret (cjeanner)]

Hey,

did you apply any patches? Apparently I can't go that far:
TASK [Ensure all online data migrations for Ironic have been applied] *************************************************************************************************************************************************************************
fatal: [undercloud]: FAILED! => {"changed": true, "cmd": ["podman", "exec", "ironic_api", "ironic-dbsync", "--config-file", "/etc/ironic/ironic.conf", "online_data_migrations"], "delta": "0:00:00.159337", "end": "2018-11-08 09:07:31.896824
", "msg": "non-zero return code", "rc": 125, "start": "2018-11-08 09:07:31.737487", "stderr": "unable to exec into ironic_api: no container with name or ID ironic_api found: no such container", "stderr_lines": ["unable to exec into ironic_
api: no container with name or ID ironic_api found: no such container"], "stdout": "", "stdout_lines": []}

PLAY RECAP ************************************************************************************************************************************************************************************************************************************
undercloud : ok=31 changed=12 unreachable=0 failed=1

[Cédric Jeanneret (cjeanner)]

Funky... apparently it's not an isolated bug I hit by chance:
http://logs.openstack.org/56/615956/5/check/tripleo-ci-centos-7-containerized-undercloud-upgrades/3443e6f/logs/undercloud/home/zuul/undercloud_upgrade.log.txt.gz#_2018-11-08_08_27_44

:/

[Cédric Jeanneret (cjeanner)]

So I think a selinux policy would not be bad for that specific case - I've opened an issue against libpod here: https://github.com/containers/libpod/issues/1782

Stay tuned!

[Cédric Jeanneret (cjeanner)]

Back on that one.

So, in order to reproduce this one, we have to:
- deploy an undercloud that does not involve patches for the directory creation/setype (that's old, prior August, maybe even prior June)
- update to the latest version
- migrate from docker -> podman

We might want to involve "upgrade" people for that, in order to get some thoughts from them.

For now, if I deploy an undercloud with docker, using master, I get the following setype on the directories:
drwxr-xr-x. root root unconfined_u:object_r:container_file_t:s0 /var/lib/kolla/
drwxr-xr-x. root root unconfined_u:object_r:container_file_t:s0 /var/lib/kolla/config_files/
-rw-------. root root system_u:object_r:container_file_t:s0 glance_api.json

Those rights are OK for both docker and podman. Would be really great to get a reproducer on that, Jill :).

[Cédric Jeanneret (cjeanner)]

So, apparently we'll need to add some upgrade task somewhere just before the podman containers are started. Either a "chcon -R", or, cleaner:
semanage fcontext -a -t container_share_t "/var/lib/kolla/config_files(/.*)?"
restorecon -R -v /var/lib/kolla

The correct location has to be defined though, with the support of the Upgrade Team :).