ceph-ansible preparation tasks with no permission to read files in /var/lib/mistral

[Jose Luis Franco]

Launchpad bug based on https://bugzilla.redhat.com/show_bug.cgi?id=1824266

When upgrading the controllers nodes from Queens to Train we first need to run a external-upgrade step to change all cheph systemd units to run from Docker to Podman. The command used to achieve this is the following:

openstack overcloud external-upgrade run \
        --stack qe-Cloud-0 \
        --tags ceph_systemd \
        -e ceph_ansible_limit=controller-0 2>&1

However, as we come from an OSP13 installation, /var/lib/mistral doesn't have execution rights at world level, having 750 rights:

(qe-Cloud-0) [stack@undercloud-0 ~]$ sudo ls -larth /var/lib/mistral/
total 44K
drwx------. 2 42430 42430 31 Apr 13 16:51 .ssh
-r--r--r--. 1 42430 42430 1001 Apr 13 16:51 undercloud.conf
drwxr-xr-x. 3 42430 42430 78 Apr 13 17:59 .novaclient
drwxr-xr-x. 77 root root 4.0K Apr 13 18:00 ..
drwxr-xr-x. 12 42430 42430 4.0K Apr 14 10:48 4405e3f5-0ae1-40b3-95c8-a496cde410fb
drwxr-xr-x. 2 42430 42430 4.0K Apr 14 11:01 ansible_fact_cache
drwxr-xr-x. 13 42430 42430 4.0K Apr 14 11:02 6321adf8-5417-4113-bdd4-03206a1d987e
drwxr-xr-x. 12 42430 42430 4.0K Apr 14 12:43 239d393a-fc93-44f5-8000-a5d701eced8f
drwxr-xr-x. 12 42430 42430 4.0K Apr 14 14:03 4003182b-47b6-45ad-8c83-83e675071fb7
drwxr-xr-x. 13 42430 42430 4.0K Apr 14 14:10 33704f84-99ee-442c-b500-e82bca250c42
drwxr-xr-x. 12 42430 42430 4.0K Apr 15 10:29 eae3da95-e0a1-427c-be87-70887b264c78
lrwxrwxrwx. 1 42430 42430 53 Apr 15 10:34 config-download-latest -> /var/lib/mistral/1f0536f5-fd4b-45d8-b970-6fa69ba22144
drwxr-x---. 12 42430 42430 4.0K Apr 15 10:34 .
drwxr-xr-x. 13 42430 42430 4.0K Apr 15 10:35 1f0536f5-fd4b-45d8-b970-6fa69ba22144

This causes the external-upgrade (which connects as tripleo-admin user into the Undercloud) to fail when accessing any file inside /var/lib/mistral:

2020-04-15 10:35:58 | TASK [tripleo-ceph-common : set calling_ansible_environment_variables] *********
2020-04-15 10:35:58 | Wednesday 15 April 2020 10:35:47 -0400 (0:00:00.863) 0:01:04.770 *******
2020-04-15 10:35:58 | skipping: [undercloud] => {"changed": false, "skip_reason": "Conditional result was False"}
2020-04-15 10:35:58 |
2020-04-15 10:35:58 | TASK [create ceph-ansible working direcotry] ***********************************
2020-04-15 10:35:58 | Wednesday 15 April 2020 10:35:48 -0400 (0:00:00.946) 0:01:05.717 *******
2020-04-15 10:35:58 |
2020-04-15 10:35:58 | TASK [tripleo-ceph-work-dir : create ceph-ansible temp dirs] *******************
2020-04-15 10:35:58 | Wednesday 15 April 2020 10:35:51 -0400 (0:00:03.146) 0:01:08.864 *******
2020-04-15 10:35:58 | changed: [undercloud] => (item=/var/lib/mistral/1f0536f5-fd4b-45d8-b970-6fa69ba22144/ceph-ansible) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/var/lib/mistral/1f0536f5-fd4b-45d8-b970-6fa69ba22144/ceph-ansible", "mode": "0755", "owner": "tripleo-admin", "path": "/var/lib/mistral/1f0536f5-fd4b-45d8-b970-6fa69ba22144/ceph-ansible", "secontext": "unconfined_u:object_r:container_file_t:s0", "size": 6, "state": "directory", "uid": 1003}
2020-04-15 10:35:58 | changed: [undercloud] => (item=/var/lib/mistral/1f0536f5-fd4b-45d8-b970-6fa69ba22144/ceph-ansible/group_vars) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/var/lib/mistral/1f0536f5-fd4b-45d8-b970-6fa69ba22144/ceph-ansible/group_vars", "mode": "0755", "owner": "tripleo-admin", "path": "/var/lib/mistral/1f0536f5-fd4b-45d8-b970-6fa69ba22144/ceph-ansible/group_vars", "secontext": "unconfined_u:object_r:container_file_t:s0", "size": 6, "state": "directory", "uid": 1003}
2020-04-15 10:35:58 | changed: [undercloud] => (item=/var/lib/mistral/1f0536f5-fd4b-45d8-b970-6fa69ba22144/ceph-ansible/host_vars) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/var/lib/mistral/1f0536f5-fd4b-45d8-b970-6fa69ba22144/ceph-ansible/host_vars", "mode": "0755", "owner": "tripleo-admin", "path": "/var/lib/mistral/1f0536f5-fd4b-45d8-b970-6fa69ba22144/ceph-ansible/host_vars", "secontext": "unconfined_u:object_r:container_file_t:s0", "size": 6, "state": "directory", "uid": 1003}
2020-04-15 10:35:58 | changed: [undercloud] => (item=/var/lib/mistral/1f0536f5-fd4b-45d8-b970-6fa69ba22144/ceph-ansible/fetch_dir) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/var/lib/mistral/1f0536f5-fd4b-45d8-b970-6fa69ba22144/ceph-ansible/fetch_dir", "mode": "0755", "owner": "tripleo-admin", "path": "/var/lib/mistral/1f0536f5-fd4b-45d8-b970-6fa69ba22144/ceph-ansible/fetch_dir", "secontext": "unconfined_u:object_r:container_file_t:s0", "size": 6, "state": "directory", "uid": 1003}
2020-04-15 10:35:58 |
2020-04-15 10:35:58 | TASK [tripleo-ceph-work-dir : symbolic link to tripleo inventory from ceph-ansible work directory] ***
2020-04-15 10:35:58 | Wednesday 15 April 2020 10:35:55 -0400 (0:00:04.092) 0:01:12.957 *******
2020-04-15 10:35:58 | fatal: [undercloud]: FAILED! => {"changed": false, "msg": "Error while linking: [Errno 13] Permission denied: b'/var/lib/mistral/1f0536f5-fd4b-45d8-b970-6fa69ba22144/inventory.yaml' -> b'/var/lib/mistral/1f0536f5-fd4b-45d8-b970-6fa69ba22144/ceph-ansible/inventory.yml'", "path": "/var/lib/mistral/1f0536f5-fd4b-45d8-b970-6fa69ba22144/ceph-ansible/inventory.yml"}

The ceph-ansible creation is allowed though, because become: true is used in the task:

[root@undercloud-0 stack]# ls -larth /var/lib/mistral/config-download-latest/ [20/1951]
total 1.1M
drwxr-xr-x. 7 42430 42430 128 Apr 15 10:34 .git
-rw-r--r--. 1 42430 42430 9 Apr 15 10:34 .gitignore
drwxr-xr-x. 2 42430 42430 4.0K Apr 15 10:34 BlockStorage
drwxr-xr-x. 5 42430 42430 4.0K Apr 15 10:34 CephStorage
drwxr-xr-x. 4 42430 42430 4.0K Apr 15 10:34 Compute
drwxr-xr-x. 5 42430 42430 4.0K Apr 15 10:34 Controller
drwxr-xr-x. 2 42430 42430 4.0K Apr 15 10:34 ObjectStorage
-rw-r--r--. 1 42430 42430 3.8K Apr 15 10:34 all_nodes_validation_script.sh
-rw-r--r--. 1 42430 42430 1.9K Apr 15 10:34 common_deploy_steps_playbooks.yaml
-rw-r--r--. 1 42430 42430 8.7K Apr 15 10:34 common_deploy_steps_tasks.yaml
-rw-r--r--. 1 42430 42430 14K Apr 15 10:34 common_deploy_steps_tasks_step_1.yaml
-rw-r--r--. 1 42430 42430 6.8K Apr 15 10:34 container_puppet_script.yaml
-rw-r--r--. 1 42430 42430 573 Apr 15 10:34 container_startup_configs_tasks.yaml
-rw-r--r--. 1 42430 42430 805 Apr 15 10:34 deploy-artifacts.sh
-rw-r--r--. 1 42430 42430 37K Apr 15 10:34 deploy_steps_playbook.yaml
-rw-r--r--. 1 42430 42430 2.3K Apr 15 10:34 deploy_steps_tasks_step_0.yaml
-rw-r--r--. 1 42430 42430 8.5K Apr 15 10:34 deployments.yaml
-rw-r--r--. 1 42430 42430 21K Apr 15 10:34 docker_puppet_script.yaml
-rw-r--r--. 1 42430 42430 40K Apr 15 10:34 external_deploy_steps_tasks.yaml
-rw-r--r--. 1 42430 42430 1.1K Apr 15 10:34 external_post_deploy_steps_tasks.yaml
-rw-r--r--. 1 42430 42430 5.5K Apr 15 10:34 external_update_steps_playbook.yaml
-rw-r--r--. 1 42430 42430 217 Apr 15 10:34 external_update_steps_tasks.yaml
-rw-r--r--. 1 42430 42430 7.6K Apr 15 10:34 external_upgrade_steps_playbook.yaml
-rw-r--r--. 1 42430 42430 8.9K Apr 15 10:34 external_upgrade_steps_tasks.yaml
-rw-r--r--. 1 42430 42430 796 Apr 15 10:34 fast_forward_upgrade_bootstrap_role_tasks.yaml
-rw-r--r--. 1 42430 42430 130 Apr 15 10:34 fast_forward_upgrade_bootstrap_tasks.yaml
-rw-r--r--. 1 42430 42430 515 Apr 15 10:34 fast_forward_upgrade_playbook.yaml
-rw-r--r--. 1 42430 42430 922 Apr 15 10:34 fast_forward_upgrade_post_role_tasks.yaml
-rw-r--r--. 1 42430 42430 621 Apr 15 10:34 fast_forward_upgrade_prep_role_tasks.yaml
-rw-r--r--. 1 42430 42430 3.6K Apr 15 10:34 fast_forward_upgrade_prep_tasks.yaml
-rw-r--r--. 1 42430 42430 113 Apr 15 10:34 fast_forward_upgrade_release_tasks.yaml
-rw-r--r--. 1 42430 42430 4.6K Apr 15 10:34 generate-config-tasks.yaml
-rw-r--r--. 1 42430 42430 13K Apr 15 10:34 global_vars.yaml
-rw-r--r--. 1 42430 42430 679 Apr 15 10:34 hiera_steps_tasks.yaml
-rw-r--r--. 1 42430 42430 3.7K Apr 15 10:34 host-container-puppet-tasks.yaml
drwxr-xr-x. 2 42430 42430 142 Apr 15 10:34 host_vars
-rw-r--r--. 1 42430 42430 575 Apr 15 10:34 post_update_steps_tasks.yaml
-rw-r--r--. 1 42430 42430 611 Apr 15 10:34 post_upgrade_steps_playbook.yaml
-rw-r--r--. 1 42430 42430 581 Apr 15 10:34 post_upgrade_steps_tasks.yaml
-rw-r--r--. 1 42430 42430 2.2K Apr 15 10:34 pre_upgrade_rolling_steps_playbook.yaml
-rw-r--r--. 1 42430 42430 616 Apr 15 10:34 pre_upgrade_rolling_steps_tasks.yaml
-rw-r--r--. 1 42430 42430 703K Apr 15 10:34 qe-Cloud-0-config.tar.gz
-rw-r--r--. 1 42430 42430 2.1K Apr 15 10:34 scale_playbook.yaml
-rw-r--r--. 1 42430 42430 2.2K Apr 15 10:34 scale_steps_tasks.yaml
drwxr-xr-x. 2 42430 42430 28 Apr 15 10:34 templates
-rw-r--r--. 1 42430 42430 6.8K Apr 15 10:34 update_steps_playbook.yaml
-rw-r--r--. 1 42430 42430 551 Apr 15 10:34 update_steps_tasks.yaml
-rw-r--r--. 1 42430 42430 7.3K Apr 15 10:34 upgrade_steps_playbook.yaml
drwxr-x---. 12 42430 42430 4.0K Apr 15 10:34 ..
-rw-r--r--. 1 42430 42430 13K Apr 15 10:34 inventory.yaml
-rw-------. 1 42430 42430 1.7K Apr 15 10:34 ssh_private_key
-rw-r--r--. 1 42430 42430 2.1K Apr 15 10:34 ansible.cfg
-rwxr-x---. 1 42430 42430 758 Apr 15 10:34 ansible-playbook-command.sh
drwxr-xr-x. 2 42430 42430 80 Apr 15 10:35 group_vars
drwxr-xr-x. 5 tripleo-admin root 58 Apr 15 10:35 ceph-ansible
drwxr-xr-x. 13 42430 42430 4.0K Apr 15 10:35 .
drwx------. 2 42430 42430 6 Apr 15 11:05 ansible-ssh

[root@undercloud-0 stack]# su - tripleo-admin
[tripleo-admin@undercloud-0 ~]$ ls -larth /var/lib/mistral/config-download-latest/ceph-ansible
ls: cannot access '/var/lib/mistral/config-download-latest/ceph-ansible': Permission denied

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/717320
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=7170a5d527158e4af6f82d82e2001d1314d2a379
Submitter: Zuul
Branch: master

commit 7170a5d527158e4af6f82d82e2001d1314d2a379
Author: Giulio Fidente <email address hidden>
Date: Fri Apr 3 17:01:09 2020 +0200

    Make /var/lib/mistral traversable by all users

    Some tasks in tripleo-ansible (like tripleo_ceph_work_dir/tasks/prepare.yml)
    need to be able to run the global inventory from /var/lib/mistral/$id and
    refer to it using {{ playbook_dir }} so we need to make the directory
    traversable by the ansible_ssh user

    Closes-Bug: #1873081
    Change-Id: I388b353e5a1f0b79a711ef5c97d0f3ae8b0de44c

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/725884

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/725884
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=d56dcc61bc74e6150d5cef8db4cd94164cca3d2c
Submitter: Zuul
Branch: stable/train

commit d56dcc61bc74e6150d5cef8db4cd94164cca3d2c
Author: Giulio Fidente <email address hidden>
Date: Fri Apr 3 17:01:09 2020 +0200

    Make /var/lib/mistral traversable by all users

    Some tasks in tripleo-ansible (like tripleo_ceph_work_dir/tasks/prepare.yml)
    need to be able to run the global inventory from /var/lib/mistral/$id and
    refer to it using {{ playbook_dir }} so we need to make the directory
    traversable by the ansible_ssh user

    Closes-Bug: #1873081
    Change-Id: I388b353e5a1f0b79a711ef5c97d0f3ae8b0de44c
    (cherry picked from commit 7170a5d527158e4af6f82d82e2001d1314d2a379)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 11.4.0 release.