ironic-python-agent

  1. Series mitaka
  2. Bug #1592163
  3. Comment #4

Comment 4 for bug 1592163

Revision history for this message
Jay Faulkner (jason-oldos) wrote : Re: IPA CoreOS Image mounts GPT disks during cleaning

Reproduction steps for functionality issue (I've made no attempt to POC an attack):

Install and configure devstack for CoreOS images. I followed the Ironic devstack guide but set these additional settings in local.conf:

IRONIC_VM_SPECS_RAM=8192 # IDK if it requires this much, but 2GB was not enough
IRONIC_RAMDISK_TYPE=coreos
IRONIC_BUILD_DEPLOY_RAMDISK=True

# Additional thing I added to help with troubleshooting + log verbosity
IRONIC_EXTRA_PXE_PARAMS="systemd.log_level=debug"

Once you've stacked with these configs, download the latest CoreOS image as provided for OpenStack:
wget https://stable.release.core-os.net/amd64-usr/current/coreos_production_openstack_image.img.bz2
bunzip2 coreos_production_openstack_image.img.bz2

Then install that image into glance:
glance image-create --name CoreOS --container-format bare --disk-format qcow2 --file files/coreos_production_openstack_image.img

Then perform a boot:
nova boot --flavor baremetal --image CoreOS --key-name default testing

You can see from the console output that as soon as the partitions are reloaded (by write_configdrive_to_disk), systemd sees the new partitions on disk, attempts to mount them, and in the process kills IPA (I will provide debug level logs in a later update):
[ 71.938657] chroot[446]: 2016-06-14 21:06:59.797 446 INFO ironic_python_agent.extensions.standby [-] Attempting to download image from http://172.99.85.138:8080/v1/AUTH_bcbd9322211b46ff864b0cac66efac2b/glance/7c5ae4e9-d19c-47e5-b27d-6947912900bd?temp_url_sig=2fbc8480d722fc13b12f4297819a3f8c64e91cb5&temp_url_expires=1465941868
[ 89.174992] chroot[446]: 2016-06-14 21:07:17.033 446 INFO ironic_python_agent.extensions.standby [-] Image downloaded from /tmp/7c5ae4e9-d19c-47e5-b27d-6947912900bd in 17.2361268997 seconds
[ 89.181958] chroot[446]: 2016-06-14 21:07:17.041 446 DEBUG ironic_python_agent.extensions.standby [-] Verifying image at /tmp/7c5ae4e9-d19c-47e5-b27d-6947912900bd against MD5 checksum 93a511c4293c9d596471ca936c45a935 _verify_image /usr/local/lib/python2.7/dist-packages/ironic_python_agent/extensions/standby.py:352
[ 89.189561] chroot[446]: 2016-06-14 21:07:17.049 446 INFO ironic_python_agent.extensions.standby [-] Writing image with command: /bin/bash /usr/local/lib/python2.7/dist-packages/ironic_python_agent/extensions/../shell/write_image.sh /tmp/7c5ae4e9-d19c-47e5-b27d-6947912900bd /dev/vda
[ 89.196322] chroot[446]: 2016-06-14 21:07:17.055 446 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): /bin/bash /usr/local/lib/python2.7/dist-packages/ironic_python_agent/extensions/../shell/write_image.sh /tmp/7c5ae4e9-d19c-47e5-b27d-6947912900bd /dev/vda execute /usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:344
[ 89.483299] systemd-udevd[315]: IMPORT{builtin}: 'uaccess' unknown /usr/lib64/udev/rules.d/73-seat-late.rules:15
[ 89.547930] chroot[446]: 2016-06-14 21:07:17.406 446 INFO ironic_python_agent.agent [-] heartbeat successful
[ 89.553071] chroot[446]: 2016-06-14 21:07:17.412 446 INFO ironic_python_agent.agent [-] sleeping before next heartbeat, interval: 11.4053231288
[ 89.574524] chroot[446]: 172.24.4.1 - - [14/Jun/2016 21:07:17] "GET /v1/commands HTTP/1.1" 200 8765
[ 89.620770] chroot[446]: 172.24.4.1 - - [14/Jun/2016 21:07:17] "GET /v1/commands HTTP/1.1" 200 8765
[ 89.652336] chroot[446]: 172.24.4.1 - - [14/Jun/2016 21:07:17] "GET /v1/commands HTTP/1.1" 200 8765
[ 90.629087] vda: unknown partition table
[ 101.057090] chroot[446]: 2016-06-14 21:07:28.911 446 INFO ironic_python_agent.agent [-] heartbeat successful
[ 101.063686] chroot[446]: 2016-06-14 21:07:28.923 446 INFO ironic_python_agent.agent [-] sleeping before next heartbeat, interval: 10.7901295871
[ 101.102089] chroot[446]: 172.24.4.1 - - [14/Jun/2016 21:07:28] "GET /v1/commands HTTP/1.1" 200 8765
[ 101.149408] chroot[446]: 172.24.4.1 - - [14/Jun/2016 21:07:29] "GET /v1/commands HTTP/1.1" 200 8765
[ 101.202996] chroot[446]: 172.24.4.1 - - [14/Jun/2016 21:07:29] "GET /v1/commands HTTP/1.1" 200 8765
[ 111.955794] chroot[446]: 2016-06-14 21:07:39.815 446 INFO ironic_python_agent.agent [-] heartbeat successful
[ 111.962565] chroot[446]: 2016-06-14 21:07:39.822 446 INFO ironic_python_agent.agent [-] sleeping before next heartbeat, interval: 16.615013086
[ 111.968991] chroot[446]: 172.24.4.1 - - [14/Jun/2016 21:07:39] "GET /v1/commands HTTP/1.1" 200 8765
[ 112.020445] chroot[446]: 172.24.4.1 - - [14/Jun/2016 21:07:39] "GET /v1/commands HTTP/1.1" 200 8765
[ 112.070479] chroot[446]: 172.24.4.1 - - [14/Jun/2016 21:07:39] "GET /v1/commands HTTP/1.1" 200 8765
[ 128.918253] chroot[446]: 2016-06-14 21:07:56.776 446 INFO ironic_python_agent.agent [-] heartbeat successful
[ 128.932644] chroot[446]: 2016-06-14 21:07:56.792 446 INFO ironic_python_agent.agent [-] sleeping before next heartbeat, interval: 16.2791435636
[ 128.942546] chroot[446]: 172.24.4.1 - - [14/Jun/2016 21:07:56] "GET /v1/commands HTTP/1.1" 200 8765
[ 129.057330] chroot[446]: 172.24.4.1 - - [14/Jun/2016 21:07:56] "GET /v1/commands HTTP/1.1" 200 8765
[ 129.090760] chroot[446]: 172.24.4.1 - - [14/Jun/2016 21:07:56] "GET /v1/commands HTTP/1.1" 200 8765
[ 146.135642] chroot[446]: 2016-06-14 21:08:13.909 446 INFO ironic_python_agent.agent [-] heartbeat successful
[ 146.166623] chroot[446]: 2016-06-14 21:08:14.001 446 INFO ironic_python_agent.agent [-] sleeping before next heartbeat, interval: 15.2491563698
[ 146.247543] chroot[446]: 172.24.4.1 - - [14/Jun/2016 21:08:14] "GET /v1/commands HTTP/1.1" 200 8765
[ 146.394902] chroot[446]: 172.24.4.1 - - [14/Jun/2016 21:08:14] "GET /v1/commands HTTP/1.1" 200 8765
[ 146.475559] chroot[446]: 172.24.4.1 - - [14/Jun/2016 21:08:14] "GET /v1/commands HTTP/1.1" 200 8765
[ 152.119313] chroot[446]: 2016-06-14 21:08:19.977 446 DEBUG oslo_concurrency.processutils [-] CMD "/bin/bash /usr/local/lib/python2.7/dist-packages/ironic_python_agent/extensions/../shell/write_image.sh /tmp/7c5ae4e9-d19c-47e5-b27d-6947912900bd /dev/vda" returned: 0 in 62.923s execute /usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:374
[ 152.126149] chroot[446]: 2016-06-14 21:08:19.985 446 DEBUG ironic_python_agent.utils [-] Execution completed, command line is "/bin/bash /usr/local/lib/python2.7/dist-packages/ironic_python_agent/extensions/../shell/write_image.sh /tmp/7c5ae4e9-d19c-47e5-b27d-6947912900bd /dev/vda" execute /usr/local/lib/python2.7/dist-packages/ironic_python_agent/utils.py:63
[ 152.132043] chroot[446]: 2016-06-14 21:08:19.991 446 DEBUG ironic_python_agent.utils [-] Command stdout is: "write_image.sh: Erasing existing GPT and MBR data structures from /dev/vda
[ 152.134746] chroot[446]: Creating new GPT entries.
[ 152.137110] chroot[446]: GPT data structures destroyed! You may now partition the disk using fdisk or
[ 152.139802] chroot[446]: other utilities.
[ 152.142155] chroot[446]: write_image.sh: Imaging /tmp/7c5ae4e9-d19c-47e5-b27d-6947912900bd to /dev/vda
[ 152.144636] chroot[446]: write_image.sh: /dev/vda imaged successfully!
[ 152.147224] chroot[446]: " execute /usr/local/lib/python2.7/dist-packages/ironic_python_agent/utils.py:65
[ 152.152027] chroot[446]: 2016-06-14 21:08:20.011 446 DEBUG ironic_python_agent.utils [-] Command stderr is: "" execute /usr/local/lib/python2.7/dist-packages/ironic_python_agent/utils.py:66
[ 152.162447] chroot[446]: 2016-06-14 21:08:20.017 446 INFO ironic_python_agent.extensions.standby [-] Image /tmp/7c5ae4e9-d19c-47e5-b27d-6947912900bd written to device /dev/vda in 62.969547987 seconds
[ 152.165095] chroot[446]: 2016-06-14 21:08:20.019 446 DEBUG ironic_python_agent.extensions.standby [-] Writing configdrive to /tmp/configdrive _write_configdrive_to_file /usr/local/lib/python2.7/dist-packages/ironic_python_agent/extensions/standby.py:176
[ 152.193002] chroot[446]: 2016-06-14 21:08:20.052 446 INFO ironic_python_agent.extensions.standby [-] copying configdrive to disk with command /bin/bash /usr/local/lib/python2.7/dist-packages/ironic_python_agent/extensions/../shell/copy_configdrive_to_disk.sh /tmp/configdrive /dev/vda
[ 152.205341] chroot[446]: 2016-06-14 21:08:20.064 446 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): /bin/bash /usr/local/lib/python2.7/dist-packages/ironic_python_agent/extensions/../shell/copy_configdrive_to_disk.sh /tmp/configdrive /dev/vda execute /usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:344
[ 156.067901] vda: vda1 vda2 vda3 vda4 vda6 vda7 vda9
[ 156.102305] systemd[1]: Stopping Ironic Python Agent...
[ 156.286481] systemd[1]: Stopped Ironic Python Agent.
[ 156.289182] systemd[1]: Unmounting /opt/ironic-python-agent/mnt...
[ 156.722348] systemd[1]: Unmounted /opt/ironic-python-agent/mnt.
[ 156.728639] systemd[1]: Unmounting /usr/share/oem...
[ 157.030346] systemd[1]: Unmounted /usr/share/oem.