tripleo

dbus selinux denials are harmful

Bug #1883849 reported by Michele Baldessari
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
High
Michele Baldessari
tripleo wallaby-rc1

Bug Description

First spotted via https://bugzilla.redhat.com/show_bug.cgi?id=1846020

Containers that are not privileged and that bind mount /run (nova_scheduler and swift_proxy) will trigger selinux denials like the following:
type=USER_AVC msg=audit(1592337899.069:74465): pid=1284 uid=81 auid=4294967295 ses=4294967295 subj=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 msg='avc: denied { send_msg } for msgtype=method_call interface=org.freedesktop.DBus member=Hello dest=org.freedesktop.DBus spid=417423 scontext=system_u:system_r:container_t:s0:c162,c886 tcontext=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 tclass=dbus permissive=0 exe="/usr/bin/dbus-daemon" sauid=81 hostname=? addr=? terminal=?'UID="dbus" AUID="unset" SAUID="dbus"

The reason for those is that by bind-mounting /run from the host, the container has access to /run/dbus/system_bus_socket and any NSS lookup (getent passwd/group) will take 30seconds until the systemd module will timeout:
$ time podman -u root -it nova_scheduler sh -c 'getent passwd &> /dev/null'
real 0m30.378s

The timeout does not happen if the container bind-mounts /run and is privileged because selinux does not block those. The implications of this is that any command that runs sudo/su/etc inside those two containers will take at least 30 seconds. In the environment I was given this actually caused messaging timeouts while nova_api was waiting for nova_scheduler. Fixing this issue made the environment functional again.

Revision history for this message
Bogdan Dobrelya (bogdando) wrote :

I have an idea how to mitigate that

Changed in tripleo:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (master)

Fix proposed to branch: master
Review: https://review.opendev.org/736128

Changed in tripleo:
assignee: Michele Baldessari (michele) → Bogdan Dobrelya (bogdando)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-common (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/736134

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tripleo-heat-templates (master)

Change abandoned by Bogdan Dobrelya (bogdando) (<email address hidden>) on branch: master
Review: https://review.opendev.org/736128

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-common (master)

Reviewed: https://review.opendev.org/736134
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=db71bdb684ac347cd2f915a39e7e99a95958606c
Submitter: Zuul
Branch: master

commit db71bdb684ac347cd2f915a39e7e99a95958606c
Author: Michele Baldessari <email address hidden>
Date: Wed Jun 17 10:39:19 2020 +0200

    Drop systemd support from nsswitch.conf

    A bit like we did for I3e0e86026f5a4a78473bed824cd1682d3a020cd5 we
    should remove the nss-systemd lookup from containers. The reasons for
    this are as follows:
    1) Just like for I3e0e86026f5a4a78473bed824cd1682d3a020cd5
    when this nss module is triggered it tries to talk to dbus.
    It triggers a bunch of selinux denials and it makes little sense
    to open all containers to talk to dbus.
    In particular, if a container is run as non-privileged and bind-mounts
    /run from the host, we will hit selinux denials like the following:

      type=USER_AVC msg=audit(1592337775.860:74119): pid=1284 uid=81 auid=4294967295 ses=4294967295 subj=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 msg='avc: denied { send_msg } for msgtype=method_call interface=org.freedesktop.DBus member=Hello dest=org.freedesktop.DBus spid=406228 scontext=system_u:system_r:container_t:s0:c162,c886 tcontext=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 tclass=dbus permissive=0 exe="/usr/bin/dbus-daemon" sauid=81 hostname=? addr=? terminal=?'UID="dbus" AUID="unset" SAUID="dbus"

    2) It just makes little sense in a kolla-world to have containers
    talk to dbus/systemd and it saves us some time when a lookup triggers
    the systemd module for whatever reason. Especially because the
    nss-systemd module does a few things which are not useful in a container
    (ensures that the root and nobody users and groups remain resolvable,
    SystemD's DynamicUser= feature, provide Lookup API via Varlink)

    The sed regex gives us the wanted results:
    $ diff -u /etc/nsswitch.conf.orig /etc/nsswitch.conf
    --- /etc/nsswitch.conf.orig 2020-06-17 08:13:20.108520894 +0000
    +++ /etc/nsswitch.conf 2020-06-17 08:13:27.627069541 +0000
    @@ -19,8 +19,8 @@
     # passwd: sss files # from profile
     # hosts: files dns # from user file

    -passwd: sss files systemd
    -group: sss files systemd
    +passwd: sss files
    +group: sss files
     netgroup: sss files
     automount: sss files
     services: sss files

    NB: This is the tcib versionb of the kolla change at I81e5b7abf4571fece13a029e25911e9e4dece673
    Related-Bug: #1883849

    Change-Id: I29d3a2ed0bdfce412ebef549c29053754763e555

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-common (stable/ussuri)

Related fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/739744

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/739950

Bogdan Dobrelya (bogdando)
Changed in tripleo:
assignee: Bogdan Dobrelya (bogdando) → Michele Baldessari (michele)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-heat-templates (master)

Reviewed: https://review.opendev.org/739950
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=bd4b57c2690e7e341691bcf4e3aa15af36978936
Submitter: Zuul
Branch: master

commit bd4b57c2690e7e341691bcf4e3aa15af36978936
Author: Michele Baldessari <email address hidden>
Date: Wed Jul 8 13:04:24 2020 +0200

    Remove /run from some services

    redis(non-pcmk), nova-scheduler and swift-proxy do not need /run bind
    mounted from the host. As a matter of fact bind-mounting /run is
    problematic due to a number of reasons (see LP#1883849 for more
    background). In particular swift-proxy is the only swift container
    (out of 9) that has /run bind-mounted.

    These three services always had /run from the very beginning:
    - redis -> Ie750caa34c6fa22ca6eae6834b9ca20e15d97f7f
    - nova-scheduler -> I39436783409ed752b08619b07b0a0c592bce0456
    - swift-proxy -> I2d96514fb7aa51dffe8fe293bc950e0e99df5e94

    Tested this by applying this patch on a train deployment and
    deployed an undercloud and an overcloud with it.
    Verified that:
    A) /run:/run is not present in the three containers
    B) Deploy of UC and OC worked correctly
    C) Tempest still works
    D) Restarting the swift_proxy and nova_scheduler works correctly
    E) Reboot the overcloud worked okay and tempest still works after the
       full overcloud reboot
    F) Ran a minor UC update
    G) Ran a minor update on all nodes and tempest still worked
    H) Ran a redeploy on all nodes and tempest still worked

    NB: I did not investigate other containers that bind mount /run
    because they 1) seem to need it and 2) had no means to do proper
    testing.

    NB2: Note that while once we rebuild containers with
    I81e5b7abf4571fece13a029e25911e9e4dece673 this change here is not
    strictly needed for the LP bug, but is a nice cleanup nonetheless.
    So this is to be backported only if rebuilding containers is
    a problematic/costly move

    Change-Id: Ic1a892a7f78a54b5e149f5ce52cb9db68ebc9529
    Related-Bug: #1883849

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-common (stable/train)

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/741214

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-common (stable/ussuri)

Reviewed: https://review.opendev.org/739744
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=5af8278c5967cc5ddc4e779d8e39f94483e12164
Submitter: Zuul
Branch: stable/ussuri

commit 5af8278c5967cc5ddc4e779d8e39f94483e12164
Author: Michele Baldessari <email address hidden>
Date: Wed Jun 17 10:39:19 2020 +0200

    Drop systemd support from nsswitch.conf

    A bit like we did for I3e0e86026f5a4a78473bed824cd1682d3a020cd5 we
    should remove the nss-systemd lookup from containers. The reasons for
    this are as follows:
    1) Just like for I3e0e86026f5a4a78473bed824cd1682d3a020cd5
    when this nss module is triggered it tries to talk to dbus.
    It triggers a bunch of selinux denials and it makes little sense
    to open all containers to talk to dbus.
    In particular, if a container is run as non-privileged and bind-mounts
    /run from the host, we will hit selinux denials like the following:

      type=USER_AVC msg=audit(1592337775.860:74119): pid=1284 uid=81 auid=4294967295 ses=4294967295 subj=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 msg='avc: denied { send_msg } for msgtype=method_call interface=org.freedesktop.DBus member=Hello dest=org.freedesktop.DBus spid=406228 scontext=system_u:system_r:container_t:s0:c162,c886 tcontext=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 tclass=dbus permissive=0 exe="/usr/bin/dbus-daemon" sauid=81 hostname=? addr=? terminal=?'UID="dbus" AUID="unset" SAUID="dbus"

    2) It just makes little sense in a kolla-world to have containers
    talk to dbus/systemd and it saves us some time when a lookup triggers
    the systemd module for whatever reason. Especially because the
    nss-systemd module does a few things which are not useful in a container
    (ensures that the root and nobody users and groups remain resolvable,
    SystemD's DynamicUser= feature, provide Lookup API via Varlink)

    The sed regex gives us the wanted results:
    $ diff -u /etc/nsswitch.conf.orig /etc/nsswitch.conf
    --- /etc/nsswitch.conf.orig 2020-06-17 08:13:20.108520894 +0000
    +++ /etc/nsswitch.conf 2020-06-17 08:13:27.627069541 +0000
    @@ -19,8 +19,8 @@
     # passwd: sss files # from profile
     # hosts: files dns # from user file

    -passwd: sss files systemd
    -group: sss files systemd
    +passwd: sss files
    +group: sss files
     netgroup: sss files
     automount: sss files
     services: sss files

    NB: This is the tcib versionb of the kolla change at I81e5b7abf4571fece13a029e25911e9e4dece673
    Related-Bug: #1883849

    Change-Id: I29d3a2ed0bdfce412ebef549c29053754763e555
    (cherry picked from commit db71bdb684ac347cd2f915a39e7e99a95958606c)

tags: added: in-stable-ussuri
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (stable/ussuri)

Related fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/742009

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (stable/train)

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/742010

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-heat-templates (stable/ussuri)

Reviewed: https://review.opendev.org/742009
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=6358fd4a1aa47971363fa25949cb1b85ab8090d7
Submitter: Zuul
Branch: stable/ussuri

commit 6358fd4a1aa47971363fa25949cb1b85ab8090d7
Author: Michele Baldessari <email address hidden>
Date: Wed Jul 8 13:04:24 2020 +0200

    Remove /run from some services

    redis(non-pcmk), nova-scheduler and swift-proxy do not need /run bind
    mounted from the host. As a matter of fact bind-mounting /run is
    problematic due to a number of reasons (see LP#1883849 for more
    background). In particular swift-proxy is the only swift container
    (out of 9) that has /run bind-mounted.

    These three services always had /run from the very beginning:
    - redis -> Ie750caa34c6fa22ca6eae6834b9ca20e15d97f7f
    - nova-scheduler -> I39436783409ed752b08619b07b0a0c592bce0456
    - swift-proxy -> I2d96514fb7aa51dffe8fe293bc950e0e99df5e94

    Tested this by applying this patch on a train deployment and
    deployed an undercloud and an overcloud with it.
    Verified that:
    A) /run:/run is not present in the three containers
    B) Deploy of UC and OC worked correctly
    C) Tempest still works
    D) Restarting the swift_proxy and nova_scheduler works correctly
    E) Reboot the overcloud worked okay and tempest still works after the
       full overcloud reboot
    F) Ran a minor UC update
    G) Ran a minor update on all nodes and tempest still worked
    H) Ran a redeploy on all nodes and tempest still worked

    NB: I did not investigate other containers that bind mount /run
    because they 1) seem to need it and 2) had no means to do proper
    testing.

    NB2: Note that while once we rebuild containers with
    I81e5b7abf4571fece13a029e25911e9e4dece673 this change here is not
    strictly needed for the LP bug, but is a nice cleanup nonetheless.
    So this is to be backported only if rebuilding containers is
    a problematic/costly move

    Change-Id: Ic1a892a7f78a54b5e149f5ce52cb9db68ebc9529
    Related-Bug: #1883849
    (cherry picked from commit bd4b57c2690e7e341691bcf4e3aa15af36978936)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-heat-templates (stable/train)

Reviewed: https://review.opendev.org/742010
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=31d8500d96f77ffc1e034a9a427cd0daaa416d7f
Submitter: Zuul
Branch: stable/train

commit 31d8500d96f77ffc1e034a9a427cd0daaa416d7f
Author: Michele Baldessari <email address hidden>
Date: Wed Jul 8 13:04:24 2020 +0200

    Remove /run from some services

    redis(non-pcmk), nova-scheduler and swift-proxy do not need /run bind
    mounted from the host. As a matter of fact bind-mounting /run is
    problematic due to a number of reasons (see LP#1883849 for more
    background). In particular swift-proxy is the only swift container
    (out of 9) that has /run bind-mounted.

    These three services always had /run from the very beginning:
    - redis -> Ie750caa34c6fa22ca6eae6834b9ca20e15d97f7f
    - nova-scheduler -> I39436783409ed752b08619b07b0a0c592bce0456
    - swift-proxy -> I2d96514fb7aa51dffe8fe293bc950e0e99df5e94

    Tested this by applying this patch on a train deployment and
    deployed an undercloud and an overcloud with it.
    Verified that:
    A) /run:/run is not present in the three containers
    B) Deploy of UC and OC worked correctly
    C) Tempest still works
    D) Restarting the swift_proxy and nova_scheduler works correctly
    E) Reboot the overcloud worked okay and tempest still works after the
       full overcloud reboot
    F) Ran a minor UC update
    G) Ran a minor update on all nodes and tempest still worked
    H) Ran a redeploy on all nodes and tempest still worked

    NB: I did not investigate other containers that bind mount /run
    because they 1) seem to need it and 2) had no means to do proper
    testing.

    NB2: Note that while once we rebuild containers with
    I81e5b7abf4571fece13a029e25911e9e4dece673 this change here is not
    strictly needed for the LP bug, but is a nice cleanup nonetheless.
    So this is to be backported only if rebuilding containers is
    a problematic/costly move

    Change-Id: Ic1a892a7f78a54b5e149f5ce52cb9db68ebc9529
    Related-Bug: #1883849
    (cherry picked from commit bd4b57c2690e7e341691bcf4e3aa15af36978936)

tags: added: in-stable-train
Emilien Macchi (emilienm)
Changed in tripleo:
milestone: victoria-1 → victoria-3
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-common (stable/train)

Reviewed: https://review.opendev.org/741214
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=38a3a5d14cb91d4b85af66b9f15197a36322ccbc
Submitter: Zuul
Branch: stable/train

commit 38a3a5d14cb91d4b85af66b9f15197a36322ccbc
Author: Michele Baldessari <email address hidden>
Date: Wed Jun 17 10:39:19 2020 +0200

    Drop systemd support from nsswitch.conf

    A bit like we did for I3e0e86026f5a4a78473bed824cd1682d3a020cd5 we
    should remove the nss-systemd lookup from containers. The reasons for
    this are as follows:
    1) Just like for I3e0e86026f5a4a78473bed824cd1682d3a020cd5
    when this nss module is triggered it tries to talk to dbus.
    It triggers a bunch of selinux denials and it makes little sense
    to open all containers to talk to dbus.
    In particular, if a container is run as non-privileged and bind-mounts
    /run from the host, we will hit selinux denials like the following:

      type=USER_AVC msg=audit(1592337775.860:74119): pid=1284 uid=81 auid=4294967295 ses=4294967295 subj=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 msg='avc: denied { send_msg } for msgtype=method_call interface=org.freedesktop.DBus member=Hello dest=org.freedesktop.DBus spid=406228 scontext=system_u:system_r:container_t:s0:c162,c886 tcontext=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 tclass=dbus permissive=0 exe="/usr/bin/dbus-daemon" sauid=81 hostname=? addr=? terminal=?'UID="dbus" AUID="unset" SAUID="dbus"

    2) It just makes little sense in a kolla-world to have containers
    talk to dbus/systemd and it saves us some time when a lookup triggers
    the systemd module for whatever reason. Especially because the
    nss-systemd module does a few things which are not useful in a container
    (ensures that the root and nobody users and groups remain resolvable,
    SystemD's DynamicUser= feature, provide Lookup API via Varlink)

    The sed regex gives us the wanted results:
    $ diff -u /etc/nsswitch.conf.orig /etc/nsswitch.conf
    --- /etc/nsswitch.conf.orig 2020-06-17 08:13:20.108520894 +0000
    +++ /etc/nsswitch.conf 2020-06-17 08:13:27.627069541 +0000
    @@ -19,8 +19,8 @@
     # passwd: sss files # from profile
     # hosts: files dns # from user file

    -passwd: sss files systemd
    -group: sss files systemd
    +passwd: sss files
    +group: sss files
     netgroup: sss files
     automount: sss files
     services: sss files

    NB: This is the tcib versionb of the kolla change at I81e5b7abf4571fece13a029e25911e9e4dece673
    Related-Bug: #1883849

    Change-Id: I29d3a2ed0bdfce412ebef549c29053754763e555
    (cherry picked from commit db71bdb684ac347cd2f915a39e7e99a95958606c)
    (cherry picked from commit fa9d0097322ac463a6448087a07960bb38d019b6)

Marios Andreou (marios-b)
Changed in tripleo:
milestone: victoria-3 → wallaby-1
Marios Andreou (marios-b)
Changed in tripleo:
milestone: wallaby-1 → wallaby-2
Marios Andreou (marios-b)
Changed in tripleo:
milestone: wallaby-2 → wallaby-3
Marios Andreou (marios-b)
Changed in tripleo:
milestone: wallaby-3 → wallaby-rc1
Michele Baldessari (michele)
Changed in tripleo:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.