containers are not working when the host has selinux in enforcing mode

Bug #1682179 reported by Michele Baldessari on 2017-04-12
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
High
Unassigned

Bug Description

Spotted this while doing some containers experimenting:
Apr 12 16:16:11 overcloud-controller-0 dockerd-current[15253]: INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
Apr 12 16:16:11 overcloud-controller-0 dockerd-current[15253]: ERROR:__main__:Unexpected error:
Apr 12 16:16:11 overcloud-controller-0 dockerd-current[15253]: Traceback (most recent call last):
Apr 12 16:16:11 overcloud-controller-0 dockerd-current[15253]: File "/usr/local/bin/kolla_set_configs", line 412, in <module>
Apr 12 16:16:11 overcloud-controller-0 dockerd-current[15253]: main()
Apr 12 16:16:11 overcloud-controller-0 dockerd-current[15253]: File "/usr/local/bin/kolla_set_configs", line 401, in main
Apr 12 16:16:11 overcloud-controller-0 dockerd-current[15253]: config = load_config()
Apr 12 16:16:11 overcloud-controller-0 dockerd-current[15253]: File "/usr/local/bin/kolla_set_configs", line 316, in load_config
Apr 12 16:16:11 overcloud-controller-0 dockerd-current[15253]: config = load_from_file()
Apr 12 16:16:11 overcloud-controller-0 dockerd-current[15253]: File "/usr/local/bin/kolla_set_configs", line 304, in load_from_file
Apr 12 16:16:11 overcloud-controller-0 dockerd-current[15253]: with open(config_file) as f:
Apr 12 16:16:11 overcloud-controller-0 dockerd-current[15253]: IOError: [Errno 13] Permission denied: '/var/lib/kolla/config_files/config.json'

Changed in tripleo:
milestone: none → pike-2
Martin André (mandre) wrote :
Changed in tripleo:
assignee: nobody → Martin André (mandre)
Changed in tripleo:
milestone: pike-2 → pike-3
Emilien Macchi (emilienm) wrote :

There are no currently open reviews on this bug, changing the status back to the previous state and unassigning. If there are active reviews related to this bug, please include links in comments.

Changed in tripleo:
assignee: Martin André (mandre) → nobody
Changed in tripleo:
milestone: pike-3 → pike-rc1
Changed in tripleo:
milestone: pike-rc1 → queens-1
Changed in tripleo:
milestone: queens-1 → queens-2
tags: added: pike-backport-potential

Fix proposed to branch: master
Review: https://review.openstack.org/513357

Changed in tripleo:
assignee: nobody → Bogdan Dobrelya (bogdando)
status: Triaged → In Progress
Bogdan Dobrelya (bogdando) wrote :

I'd raise this bug to High, as it is related to security trade offs.

Changed in tripleo:
importance: Medium → High

Reviewed: https://review.openstack.org/513357
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=47ce82003ee8d9b2d73f1c9451cfb07591807aff
Submitter: Zuul
Branch: master

commit 47ce82003ee8d9b2d73f1c9451cfb07591807aff
Author: Bogdan Dobrelya <email address hidden>
Date: Thu Oct 19 13:16:23 2017 +0200

    Allow containerized undercloud deploy with SELinux

    When SELinux is enforcing, use :Z flag for the heat_all
    container's volumes. Note, if a volume mount with a Z,
    then the label will be specific to the container, and
    not be able to be shared between containers.

    Partial-bug: #1682179
    Related-bug: #1723003

    Change-Id: Ib4022e022eb2b757591635c362b572ab06f65ed8
    Signed-off-by: Bogdan Dobrelya <email address hidden>

Reviewed: https://review.openstack.org/513746
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=fe2afcb87218af6a3523be5a885d260ec54d24a5
Submitter: Zuul
Branch: stable/pike

commit fe2afcb87218af6a3523be5a885d260ec54d24a5
Author: Bogdan Dobrelya <email address hidden>
Date: Thu Oct 19 13:16:23 2017 +0200

    Allow containerized undercloud deploy with SELinux

    When SELinux is enforcing, use :Z flag for the heat_all
    container's volumes. Note, if a volume mount with a Z,
    then the label will be specific to the container, and
    not be able to be shared between containers.

    Partial-bug: #1682179
    Related-bug: #1723003

    Change-Id: Ib4022e022eb2b757591635c362b572ab06f65ed8
    Signed-off-by: Bogdan Dobrelya <email address hidden>

tags: added: in-stable-pike

Reviewed: https://review.openstack.org/513669
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=1fc928512590b119318e7089e5f3d45f8839b385
Submitter: Zuul
Branch: master

commit 1fc928512590b119318e7089e5f3d45f8839b385
Author: Bogdan Dobrelya <email address hidden>
Date: Fri Oct 20 11:00:18 2017 +0200

    Allow containerized undercloud deploy with SELinux

    When SELinux is enforcing, use the docker volume mount flag
    :z for the docker-puppet tool's bind-mounted volumes in RW mode.
    Note, if a volume mount with a Z, then the label will be specific
    to the container, and not be able to be shared between containers.

    Volumes from /etc/pki mounted RO do not require the context changes.
    For those RO volumes that do require it, use :ro,z.

    For deploy-steps, make sure ansible file resources in /var/lib/
    are enforced the same SELinux context attributes what docker's :z
    provides.

    Partial-bug: #1682179
    Related-bug: #1723003

    Change-Id: Idc0caa49573bd88e8410d3d4217fd39e9aabf8f2
    Signed-off-by: Bogdan Dobrelya <email address hidden>

Changed in tripleo:
milestone: queens-2 → queens-3
Changed in tripleo:
status: In Progress → Triaged
assignee: Bogdan Dobrelya (bogdando) → nobody
Changed in tripleo:
milestone: queens-3 → queens-rc1

Change abandoned by Bogdan Dobrelya (<email address hidden>) on branch: master
Review: https://review.openstack.org/517383
Reason: let's catch up it later may be

Changed in tripleo:
milestone: queens-rc1 → rocky-1
Changed in tripleo:
milestone: rocky-1 → rocky-2
Changed in tripleo:
milestone: rocky-2 → rocky-3
Alex Schultz (alex-schultz) wrote :

We're actually running the containerized undercloud enforcing in CI right now and no longer seeing this issue. I believe it to be fixed. Please reopen if the issue pops up again

Changed in tripleo:
status: Triaged → Fix Released
Bogdan Dobrelya (bogdando) wrote :

Unless you are trying to launch an instance (like stantalone AIO), SELinux works for undercloud.

For AIO, testing shows problems with that
/var/log/containers/nova/nova-conductor.log:2018-07-04 14:53:30.153 24 ERROR nova.scheduler.utils [req-73bd35c6-a29a-4d39-948b-0fd6bd4b864e 8b4d0b6b0e0543d280b96e34649500b8 7c06624ac5e84f129b8f3a52e700d43a - default default] [instance: 2b4f6d11-aa19-4a84-838a-441520fea06f] Error from last host: standalone-0.localdomain (node standalone-0.localdomain): [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1819, in _do_build_and_run_instance\n filter_properties, request_spec)\n', u' File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2108, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance 2b4f6d11-aa19-4a84-838a-441520fea06f was re-scheduled: internal error: process exited while connecting to monitor: libvirt: error : cannot execute binary /usr/libexec/qemu-kvm: Permission denied\n

after setenforce 0 - problem goes away.

Reopening as we need to double check containerized compute nodes work with enforcing SELinux overcloud.

Changed in tripleo:
status: Fix Released → Triaged
Changed in tripleo:
status: Triaged → Incomplete
Changed in tripleo:
status: Incomplete → In Progress
assignee: nobody → Martin Schuppert (mschuppert)
Changed in tripleo:
status: In Progress → Incomplete
assignee: Martin Schuppert (mschuppert) → nobody
Martin Schuppert (mschuppert) wrote :

which virt_type is being used ? qemu or kvm?
https://bugzilla.redhat.com/show_bug.cgi?id=1538651

Bogdan Dobrelya (bogdando) wrote :

@Martin, it is kvm, the default t-h-t value http://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/puppet/services/nova-libvirt.yaml#n58 and it seem not overriden in CI (quickstart)

tags: added: quickstart
Bogdan Dobrelya (bogdando) wrote :

hm, at the 2nd sight, it **is** overriden in CI, just not via t-h-t but the tripleoclient!
http://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/tree/roles/overcloud-deploy/defaults/main.yml#n40

Bogdan Dobrelya (bogdando) wrote :

Related change to change CI defaults https://review.openstack.org/#/c/584380

Changed in tripleo:
milestone: rocky-3 → rocky-rc1
Changed in tripleo:
milestone: rocky-rc1 → stein-1
Alex Schultz (alex-schultz) wrote :

Closing this bug out as we've got this working.

Changed in tripleo:
status: Incomplete → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.