IPA may unexpectedly stop in a CoreOS-based ramdisk in certain circumstances

[Jay Faulkner]

In some scenarios, such as:

1) Imaging a disk with a recent CoreOS image, or any other image with a partition labelled OEM
2) During a cleaning step refreshing or changing partition tables that include a partition labelled OEM (no shipped cleaning step does this today).

Will cause a udev event to trigger, causing systemd to attempt to mount the new partition at /usr/share/oem. This mount will not succeed, due to ConditionPathExists=!/usr/.noupdate not passing, however, when it fails to mount, systemd will unexpectedly unmount /usr/share/oem, causing ironic-python-agent to unexpectedly stop and not complete whatever action was in progress.

The most common impact is that it becomes impossible to deploy a CoreOS image using a CoreOS based IPA ramdisk because IPA will stop before completing the deployment.

[Jay Faulkner (jason-oldos)]

Apparently, there are other additional units which can automount directories based on udev triggers. When we figure out the full list, I'll update the bug as we'll also need to mask those to avoid this case.

[Jim Rollenhagen (jim-rollenhagen)]

Going to add Brad Morgan (co-reported) once I confirm his launchpad account. Adding Deva now.

[Jim Rollenhagen (jim-rollenhagen)]

Did not add VMT yet but we'll need to once we confirm this.

[Jay Faulkner (jason-oldos)]

Reproduction steps for functionality issue (I've made no attempt to POC an attack):

Install and configure devstack for CoreOS images. I followed the Ironic devstack guide but set these additional settings in local.conf:

IRONIC_VM_SPECS_RAM=8192 # IDK if it requires this much, but 2GB was not enough
IRONIC_RAMDISK_TYPE=coreos
IRONIC_BUILD_DEPLOY_RAMDISK=True

# Additional thing I added to help with troubleshooting + log verbosity
IRONIC_EXTRA_PXE_PARAMS="systemd.log_level=debug"

Once you've stacked with these configs, download the latest CoreOS image as provided for OpenStack:
wget https://stable.release.core-os.net/amd64-usr/current/coreos_production_openstack_image.img.bz2
bunzip2 coreos_production_openstack_image.img.bz2

Then install that image into glance:
glance image-create --name CoreOS --container-format bare --disk-format qcow2 --file files/coreos_production_openstack_image.img

Then perform a boot:
nova boot --flavor baremetal --image CoreOS --key-name default testing

You can see from the console output that as soon as the partitions are reloaded (by write_configdrive_to_disk), systemd sees the new partitions on disk, attempts to mount them, and in the process kills IPA (I will provide debug level logs in a later update):
[ 71.938657] chroot[446]: 2016-06-14 21:06:59.797 446 INFO ironic_python_agent.extensions.standby [-] Attempting to download image from http://172.99.85.138:8080/v1/AUTH_bcbd9322211b46ff864b0cac66efac2b/glance/7c5ae4e9-d19c-47e5-b27d-6947912900bd?temp_url_sig=2fbc8480d722fc13b12f4297819a3f8c64e91cb5&temp_url_expires=1465941868
[ 89.174992] chroot[446]: 2016-06-14 21:07:17.033 446 INFO ironic_python_agent.extensions.standby [-] Image downloaded from /tmp/7c5ae4e9-d19c-47e5-b27d-6947912900bd in 17.2361268997 seconds
[ 89.181958] chroot[446]: 2016-06-14 21:07:17.041 446 DEBUG ironic_python_agent.extensions.standby [-] Verifying image at /tmp/7c5ae4e9-d19c-47e5-b27d-6947912900bd against MD5 checksum 93a511c4293c9d596471ca936c45a935 _verify_image /usr/local/lib/python2.7/dist-packages/ironic_python_agent/extensions/standby.py:352
[ 89.189561] chroot[446]: 2016-06-14 21:07:17.049 446 INFO ironic_python_agent.extensions.standby [-] Writing image with command: /bin/bash /usr/local/lib/python2.7/dist-packages/ironic_python_agent/extensions/../shell/write_image.sh /tmp/7c5ae4e9-d19c-47e5-b27d-6947912900bd /dev/vda
[ 89.196322] chroot[446]: 2016-06-14 21:07:17.055 446 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): /bin/bash /usr/local/lib/python2.7/dist-packages/ironic_python_agent/extensions/../shell/write_image.sh /tmp/7c5ae4e9-d19c-47e5-b27d-6947912900bd /dev/vda execute /usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:344
[ 89.483299] systemd-udevd[315]: IMPORT{builtin}: 'uaccess' unknown /usr/lib64/udev/rules.d/73-seat-late.rules:15
[ 89.547930] chroot[446]: 2016-06-14 21:07:17.406 446 INFO ironic_python_agent.agent [-] heartbeat successful
[ 89.553071] chroot[446]: 2016-06-14 21:07:17.412 446 INFO ironic_python_agent.agent [-] sleeping before next heartbeat, interval: 11.4053231288
[ 89.574524] chroot[446]: 172.24.4.1 - - [14/Jun/2016 21:07:17] "GET /v1/com...

[Jay Faulkner (jason-oldos)]

systemd.log_level=debug apparently only works on newer systemd/coreos images. In search of the proper flag for the older image IPA ships...

[Jay Faulkner (jason-oldos)]

We've now successfully reproduced the issue in latest CoreOS as well, 1068.0.0. I will be responsibly disclosing the vulnerability in their shipped PXE image using their documented process.

[Jay Faulkner (jason-oldos)]

Upon further investigation; it appears this is not exploitable for RCE:

1) systemd is blocked from completely executing any of these mount units by the ConditionFileExists=!/usr/.noupdate being false.

HOWEVER, the udev event still triggers the mount job to attempt to start, so it begins starting, is blocked from completing by systemd, and then automatically stopped; summarily unmounting the dir out from under IPA, and since we properly declare dependencies for our chroot environment, systemd enforces those dependencies and stops the IPA chroot.

I believe this downgrade this from a major vulnerability to a very minor DoS vuln at worst.

[Jay Faulkner (jason-oldos)]

After research proving this was not exploitable, opening this bug up after consensus from myself, Brad (co-reporter), Jim (PTL), and Devananda (VMT liason).

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/330789

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/330789
Committed: https://git.openstack.org/cgit/openstack/ironic-python-agent/commit/?id=da0a8aff88e67fce7603b03a220b77b451349e7d
Submitter: Jenkins
Branch: master

commit da0a8aff88e67fce7603b03a220b77b451349e7d
Author: Brad Morgan <email address hidden>
Date: Thu Jun 16 13:30:34 2016 -0700

    Fix unwanted unmounting of /usr/share/oem

    Creating specific partitions will cause a udev event to trigger,
    causing systemd to attempt to mount the new partition at /usr/share/oem.

    This mount will not succeed, but will unmount /usr/share/oem,
    causing ironic-python-agent to die.

    Masking this unit disables this behavior.

    Change-Id: Ifea3482f36a354cbf988e6f6003f4284c5c2a21d
    Closes-bug: #1592163

[Jay Faulkner (jason-oldos)]

This needs to be backported to stable/mitaka before it's closed.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/332471

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/332472

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/332471
Committed: https://git.openstack.org/cgit/openstack/ironic-python-agent/commit/?id=d19558e7262c26559cef862c4d750756b37ada3c
Submitter: Jenkins
Branch: stable/mitaka

commit d19558e7262c26559cef862c4d750756b37ada3c
Author: Brad Morgan <email address hidden>
Date: Thu Jun 16 13:30:34 2016 -0700

    Fix unwanted unmounting of /usr/share/oem

    Creating specific partitions will cause a udev event to trigger,
    causing systemd to attempt to mount the new partition at /usr/share/oem.

    This mount will not succeed, but will unmount /usr/share/oem,
    causing ironic-python-agent to die.

    Masking this unit disables this behavior.

    Change-Id: Ifea3482f36a354cbf988e6f6003f4284c5c2a21d
    Closes-bug: #1592163
    (cherry picked from commit da0a8aff88e67fce7603b03a220b77b451349e7d)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/332472
Committed: https://git.openstack.org/cgit/openstack/ironic-python-agent/commit/?id=b1471eaf0eaff4187097da2db862b1971c9c67f1
Submitter: Jenkins
Branch: stable/liberty

commit b1471eaf0eaff4187097da2db862b1971c9c67f1
Author: Brad Morgan <email address hidden>
Date: Thu Jun 16 13:30:34 2016 -0700

    Fix unwanted unmounting of /usr/share/oem

    Creating specific partitions will cause a udev event to trigger,
    causing systemd to attempt to mount the new partition at /usr/share/oem.

    This mount will not succeed, but will unmount /usr/share/oem,
    causing ironic-python-agent to die.

    Masking this unit disables this behavior.

    Change-Id: Ifea3482f36a354cbf988e6f6003f4284c5c2a21d
    Closes-bug: #1592163
    (cherry picked from commit da0a8aff88e67fce7603b03a220b77b451349e7d)

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/ironic-python-agent 1.3.0 release.

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/ironic-python-agent 1.2.3 release.

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/ironic-python-agent 1.0.4 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/ironic-python-agent 1.2.3 release.