selinux ssh denieals on CentOS/RHEL 8

[Michele Baldessari]

Observed the following on the undercloud while doing an overcloud deployment:
type=AVC msg=audit(1553623415.058:12945): avc: denied { read } for pid=127833 comm="sshd" name="authorized_keys" dev="vda1" ino=100688482 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:container_file_t:s0:c85,c195 tclass=file permissive=0
type=AVC msg=audit(1553623415.152:12959): avc: denied { read } for pid=127843 comm="sshd" name="authorized_keys" dev="vda1" ino=100688482 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:container_file_t:s0:c85,c195 tclass=file permissive=0
type=AVC msg=audit(1553623416.237:12985): avc: denied { read } for pid=128104 comm="sshd" name="authorized_keys" dev="vda1" ino=100688482 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:container_file_t:s0:c85,c195 tclass=file permissive=0
type=AVC msg=audit(1553623419.327:12999): avc: denied { read } for pid=128112 comm="sshd" name="authorized_keys" dev="vda1" ino=100688482 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:container_file_t:s0:c85,c195 tclass=file permissive=0
type=AVC msg=audit(1553623426.425:13013): avc: denied { read } for pid=128125 comm="sshd" name="authorized_keys" dev="vda1" ino=100688482 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:container_file_t:s0:c85,c195 tclass=file permissive=0
type=AVC msg=audit(1553623441.526:13040): avc: denied { read } for pid=128416 comm="sshd" name="authorized_keys" dev="vda1" ino=100688482 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:container_file_t:s0:c85,c195 tclass=file permissive=0

The deploy CLI fails with:
2019-03-26 18:02:25Z [overcloud.AllNodesDeploySteps.ComputeArtifactsDeploy]: CREATE_COMPLETE Stack CREATE completed successfullyGenerating public/private rsa key pair.
Your identification has been saved in /tmp/tmptj7cv8yb/id_rsa.
Your public key has been saved in /tmp/tmptj7cv8yb/id_rsa.pub.
The key fingerprint is:
SHA256:uAHU3YJq2WUyiDiIvPC1obOmCf0+iL7PB0u3bnHY/Hw TripleO split stack short term key
The key's randomart image is:
+---[RSA 4096]----+
|+. ..o o . |
|*...+ = = . |
|.o.o.* = . |
| .+ =... |
| . + * S |
|. +o + * |
|.=.o+ = o |
|+ o.o+ o E |
|.oo+=o . |
+----[SHA256]-----+
Warning: Permanently added '192.168.24.24' (ECDSA) to the list of known hosts.^M
Warning: Permanently added '192.168.24.21' (ECDSA) to the list of known hosts.^M
Warning: Permanently added '192.168.24.10' (ECDSA) to the list of known hosts.^M
Warning: Permanently added '192.168.24.22' (ECDSA) to the list of known hosts.^M
Warning: Permanently added '192.168.24.24' (ECDSA) to the list of known hosts.^M
Warning: Permanently added '192.168.24.21' (ECDSA) to the list of known hosts.^M
Warning: Permanently added '192.168.24.10' (ECDSA) to the list of known hosts.^M
Warning: Permanently added '192.168.24.22' (ECDSA) to the list of known hosts.^M
Waiting for messages on queue 'tripleo' with no timeout.
Exception occured while running the command
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 29, in run
    super(Command, self).run(parsed_args)
  File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python3.6/site-packages/cliff/command.py", line 184, in run
    return_code = self.take_action(parsed_args) or 0
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 949, in take_action
    verbosity=self.app_args.verbose_level)
  File "/usr/lib/python3.6/site-packages/tripleoclient/workflows/deployment.py", line 327, in config_download
    raise exceptions.DeploymentError("Overcloud configuration failed.")
tripleoclient.exceptions.DeploymentError: Overcloud configuration failed.
Overcloud configuration failed.

2019-03-26 18:02:25Z [overcloud.AllNodesDeploySteps.ControllerExtraConfigPost]: CREATE_IN_PROGRESS state changed
2019-03-26 18:02:26Z [overcloud.AllNodesDeploySteps.ComputeArtifactsDeploy]: CREATE_COMPLETE state changed
2019-03-26 18:02:26Z [overcloud.AllNodesDeploySteps.ControllerArtifactsDeploy.2]: CREATE_COMPLETE state changed
2019-03-26 18:02:26Z [overcloud.AllNodesDeploySteps.ControllerArtifactsDeploy.0]: CREATE_COMPLETE state changed
2019-03-26 18:02:27Z [overcloud.AllNodesDeploySteps.ControllerExtraConfigPost]: CREATE_COMPLETE state changed
2019-03-26 18:02:27Z [overcloud.AllNodesDeploySteps.ComputePostConfig]: CREATE_IN_PROGRESS state changed
2019-03-26 18:02:27Z [overcloud.AllNodesDeploySteps.ControllerPostConfig]: CREATE_IN_PROGRESS state changed
2019-03-26 18:02:27Z [overcloud.AllNodesDeploySteps.ComputePostConfig]: CREATE_COMPLETE state changed
2019-03-26 18:02:27Z [overcloud.AllNodesDeploySteps.ControllerPostConfig]: CREATE_COMPLETE state changed
2019-03-26 18:02:28Z [overcloud.AllNodesDeploySteps.ControllerArtifactsDeploy.1]: CREATE_COMPLETE state changed
2019-03-26 18:02:28Z [overcloud.AllNodesDeploySteps.ControllerArtifactsDeploy]: CREATE_COMPLETE Stack CREATE completed successfully
2019-03-26 18:02:28Z [overcloud.AllNodesDeploySteps.ControllerArtifactsDeploy]: CREATE_COMPLETE state changed
2019-03-26 18:02:28Z [overcloud.AllNodesDeploySteps]: CREATE_COMPLETE Stack CREATE completed successfully
2019-03-26 18:02:29Z [overcloud.AllNodesDeploySteps]: CREATE_COMPLETE state changed
2019-03-26 18:02:29Z [overcloud]: CREATE_COMPLETE Stack CREATE completed successfully

 Stack overcloud/bb60cfcf-c70d-4cf8-8074-cc13daf5456a CREATE_COMPLETE

Deploying overcloud configuration
Enabling ssh admin (tripleo-admin) for hosts:
192.168.24.24 192.168.24.21 192.168.24.10 192.168.24.22
Using ssh user heat-admin for initial connection.
Using ssh key at /home/stack/.ssh/id_rsa for initial connection.
Inserting TripleO short term key for 192.168.24.24
Inserting TripleO short term key for 192.168.24.21
Inserting TripleO short term key for 192.168.24.10
Inserting TripleO short term key for 192.168.24.22
Starting ssh admin enablement workflow
ssh admin enablement workflow - RUNNING.
ssh admin enablement workflow - RUNNING.
ssh admin enablement workflow - RUNNING.
ssh admin enablement workflow - COMPLETE.
Removing TripleO short term key from 192.168.24.24
Removing TripleO short term key from 192.168.24.21
Removing TripleO short term key from 192.168.24.10
Removing TripleO short term key from 192.168.24.22
Removing short term keys locally
Enabling ssh admin - COMPLETE.
Config downloaded at /var/lib/mistral/overcloud
Inventory generated at /var/lib/mistral/overcloud/tripleo-ansible-inventory.yaml
Running ansible playbook at /var/lib/mistral/overcloud/deploy_steps_playbook.yaml. See log file at /var/lib/mistral/overcloud/ansible.log for progress. ...

Using /var/lib/mistral/overcloud/ansible.cfg as config file
/var/lib/mistral/overcloud/tripleo-ansible-inventory.yaml did not meet host_list requirements, check plugin documentation if this is unexpected
/var/lib/mistral/overcloud/tripleo-ansible-inventory.yaml did not meet script requirements, check plugin documentation if this is unexpected

PLAY [Gather facts from undercloud] ********************************************

TASK [Gathering Facts] *********************************************************
Tuesday 26 March 2019 18:03:34 +0000 (0:00:00.032) 0:00:00.032 *********
fatal: [undercloud]: UNREACHABLE! => {"changed": false, "msg": "SSH Error: data could not be sent to remote host \"localhost\". Make sure this host can be reached over ssh", "unreachable": true}

PLAY RECAP *********************************************************************
undercloud : ok=0 changed=0 unreachable=1 failed=0

Tuesday 26 March 2019 18:05:31 +0000 (0:01:57.193) 0:01:57.226 *********
===============================================================================

Ansible failed, check log at /var/lib/mistral/overcloud/ansible.log.

[Michele Baldessari (michele)]

Seems the issue is that ~tripleo-admin/.ssh/authorized_keys gets created with the container_t context and then ssh to localhost then fails:
[root@undercloud-0 ~]# ls -laZR /home/tripleo-admin/.ssh/
/home/tripleo-admin/.ssh/:
total 12
drwx------. 2 tripleo-admin tripleo-admin unconfined_u:object_r:ssh_home_t:s0 61 Mar 26 18:02 .
drwx------. 3 tripleo-admin tripleo-admin unconfined_u:object_r:user_home_dir_t:s0 74 Mar 26 17:12 ..
-rw-------. 1 tripleo-admin tripleo-admin system_u:object_r:container_file_t:s0:c85,c195 830 Mar 26 18:02 authorized_keys
-rw-------. 1 tripleo-admin tripleo-admin unconfined_u:object_r:ssh_home_t:s0 1864 Mar 26 17:12 id_rsa
-rw-r--r--. 1 tripleo-admin tripleo-admin unconfined_u:object_r:ssh_home_t:s0 428 Mar 26 17:12 id_rsa.pub

After I run a forced relabel:
[root@undercloud-0 ~]# restorecon -Rv /home/tripleo-admin/.ssh/
/home/tripleo-admin/.ssh/authorized_keys not reset as customized by admin to system_u:object_r:container_file_t:s0:c85,c195
[root@undercloud-0 ~]# restorecon -RvF /home/tripleo-admin/.ssh/
Relabeled /home/tripleo-admin/.ssh/authorized_keys from system_u:object_r:container_file_t:s0:c85,c195 to unconfined_u:object_r:ssh_home_t:s0

The deploy continued

[Michele Baldessari (michele)]

So the problem is that we create the user via a host_prep_task in deployment/mistral/mistral-executor-container-puppet.yaml (see previous comment: happened at around 17:12), but then a container starts and fiddles with authorized_keys (see later timestamp at 18:02) and that changes the selinux context and messes things up for us.

[Michele Baldessari (michele)]

Likely https://review.openstack.org/#/c/638323/ will avoid this completely?

[Michele Baldessari (michele)]

Indeed https://review.openstack.org/#/c/638323/ fixes it for me