containers are not working when the host has selinux in enforcing mode

[Michele Baldessari]

Spotted this while doing some containers experimenting:
Apr 12 16:16:11 overcloud-controller-0 dockerd-current[15253]: INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
Apr 12 16:16:11 overcloud-controller-0 dockerd-current[15253]: ERROR:__main__:Unexpected error:
Apr 12 16:16:11 overcloud-controller-0 dockerd-current[15253]: Traceback (most recent call last):
Apr 12 16:16:11 overcloud-controller-0 dockerd-current[15253]: File "/usr/local/bin/kolla_set_configs", line 412, in <module>
Apr 12 16:16:11 overcloud-controller-0 dockerd-current[15253]: main()
Apr 12 16:16:11 overcloud-controller-0 dockerd-current[15253]: File "/usr/local/bin/kolla_set_configs", line 401, in main
Apr 12 16:16:11 overcloud-controller-0 dockerd-current[15253]: config = load_config()
Apr 12 16:16:11 overcloud-controller-0 dockerd-current[15253]: File "/usr/local/bin/kolla_set_configs", line 316, in load_config
Apr 12 16:16:11 overcloud-controller-0 dockerd-current[15253]: config = load_from_file()
Apr 12 16:16:11 overcloud-controller-0 dockerd-current[15253]: File "/usr/local/bin/kolla_set_configs", line 304, in load_from_file
Apr 12 16:16:11 overcloud-controller-0 dockerd-current[15253]: with open(config_file) as f:
Apr 12 16:16:11 overcloud-controller-0 dockerd-current[15253]: IOError: [Errno 13] Permission denied: '/var/lib/kolla/config_files/config.json'

[Martin André (mandre)]

Related BZ, https://bugzilla.redhat.com/show_bug.cgi?id=1448482

[Emilien Macchi (emilienm)]

There are no currently open reviews on this bug, changing the status back to the previous state and unassigning. If there are active reviews related to this bug, please include links in comments.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/513357

[Bogdan Dobrelya (bogdando)]

I'd raise this bug to High, as it is related to security trade offs.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/513669

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/513746

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/513774

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/513357
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=47ce82003ee8d9b2d73f1c9451cfb07591807aff
Submitter: Zuul
Branch: master

commit 47ce82003ee8d9b2d73f1c9451cfb07591807aff
Author: Bogdan Dobrelya <email address hidden>
Date: Thu Oct 19 13:16:23 2017 +0200

    Allow containerized undercloud deploy with SELinux

    When SELinux is enforcing, use :Z flag for the heat_all
    container's volumes. Note, if a volume mount with a Z,
    then the label will be specific to the container, and
    not be able to be shared between containers.

    Partial-bug: #1682179
    Related-bug: #1723003

    Change-Id: Ib4022e022eb2b757591635c362b572ab06f65ed8
    Signed-off-by: Bogdan Dobrelya <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/513746
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=fe2afcb87218af6a3523be5a885d260ec54d24a5
Submitter: Zuul
Branch: stable/pike

commit fe2afcb87218af6a3523be5a885d260ec54d24a5
Author: Bogdan Dobrelya <email address hidden>
Date: Thu Oct 19 13:16:23 2017 +0200

    Allow containerized undercloud deploy with SELinux

    When SELinux is enforcing, use :Z flag for the heat_all
    container's volumes. Note, if a volume mount with a Z,
    then the label will be specific to the container, and
    not be able to be shared between containers.

    Partial-bug: #1682179
    Related-bug: #1723003

    Change-Id: Ib4022e022eb2b757591635c362b572ab06f65ed8
    Signed-off-by: Bogdan Dobrelya <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/517383

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/513669
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=1fc928512590b119318e7089e5f3d45f8839b385
Submitter: Zuul
Branch: master

commit 1fc928512590b119318e7089e5f3d45f8839b385
Author: Bogdan Dobrelya <email address hidden>
Date: Fri Oct 20 11:00:18 2017 +0200

    Allow containerized undercloud deploy with SELinux

    When SELinux is enforcing, use the docker volume mount flag
    :z for the docker-puppet tool's bind-mounted volumes in RW mode.
    Note, if a volume mount with a Z, then the label will be specific
    to the container, and not be able to be shared between containers.

    Volumes from /etc/pki mounted RO do not require the context changes.
    For those RO volumes that do require it, use :ro,z.

    For deploy-steps, make sure ansible file resources in /var/lib/
    are enforced the same SELinux context attributes what docker's :z
    provides.

    Partial-bug: #1682179
    Related-bug: #1723003

    Change-Id: Idc0caa49573bd88e8410d3d4217fd39e9aabf8f2
    Signed-off-by: Bogdan Dobrelya <email address hidden>

[OpenStack Infra (hudson-openstack)]

Change abandoned by Bogdan Dobrelya (<email address hidden>) on branch: master
Review: https://review.openstack.org/517383
Reason: let's catch up it later may be

[Alex Schultz (alex-schultz)]

We're actually running the containerized undercloud enforcing in CI right now and no longer seeing this issue. I believe it to be fixed. Please reopen if the issue pops up again

[Bogdan Dobrelya (bogdando)]

Unless you are trying to launch an instance (like stantalone AIO), SELinux works for undercloud.

For AIO, testing shows problems with that
/var/log/containers/nova/nova-conductor.log:2018-07-04 14:53:30.153 24 ERROR nova.scheduler.utils [req-73bd35c6-a29a-4d39-948b-0fd6bd4b864e 8b4d0b6b0e0543d280b96e34649500b8 7c06624ac5e84f129b8f3a52e700d43a - default default] [instance: 2b4f6d11-aa19-4a84-838a-441520fea06f] Error from last host: standalone-0.localdomain (node standalone-0.localdomain): [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1819, in _do_build_and_run_instance\n filter_properties, request_spec)\n', u' File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2108, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance 2b4f6d11-aa19-4a84-838a-441520fea06f was re-scheduled: internal error: process exited while connecting to monitor: libvirt: error : cannot execute binary /usr/libexec/qemu-kvm: Permission denied\n

after setenforce 0 - problem goes away.

Reopening as we need to double check containerized compute nodes work with enforcing SELinux overcloud.

[Martin Schuppert (mschuppert)]

which virt_type is being used ? qemu or kvm?
https://bugzilla.redhat.com/show_bug.cgi?id=1538651

[Bogdan Dobrelya (bogdando)]

@Martin, it is kvm, the default t-h-t value http://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/puppet/services/nova-libvirt.yaml#n58 and it seem not overriden in CI (quickstart)

[Bogdan Dobrelya (bogdando)]

hm, at the 2nd sight, it **is** overriden in CI, just not via t-h-t but the tripleoclient!
http://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/tree/roles/overcloud-deploy/defaults/main.yml#n40

[Bogdan Dobrelya (bogdando)]

Related change to change CI defaults https://review.openstack.org/#/c/584380

[Alex Schultz (alex-schultz)]

Closing this bug out as we've got this working.