disk-image-create deleting system /dev during image create

[John-Paul Robinson]

The disk-image-create run by nodepool is causing problems by deleting the contents of the /dev directory. This appears to happen as a result of /dev getting bind mounted to the working space of the image getting created and then a bulk mv command moving all the staged input files onto the target disk image. This swoops up /dev's content in the move.

The command triggered by nodepool (observed via ps):

nodepool 10846 4465 0 20:23 pts/3 00:00:01 /usr/bin/python /usr/local/bin/nodepool image-build trustyvm
nodepool 10890 8727 0 20:23 ? 00:00:00 /bin/bash /usr/local/bin/disk-image-create -x -t --no-tmpfs -o /opt/nodepool_dib/5 ubuntu vm

(Note, -t is missing a qcow2 arg. This appears to result from a nodepool config parse bug which will be submitted separately.)

The image.log file shows the last actions of the build as:

2016-01-25 20:25:57,037 INFO nodepool.image.build.trustyvm: tune2fs 1.42.9 (4-Feb-2014)
2016-01-25 20:25:57,040 INFO nodepool.image.build.trustyvm: + mkdir /opt/dib_tmp/image.1UCql7CH/mnt
2016-01-25 20:25:57,043 INFO nodepool.image.build.trustyvm: + sudo mount /dev/loop0p1 /opt/dib_tmp/image.1UCql7CH/mnt
2016-01-25 20:25:57,071 INFO nodepool.image.build.trustyvm: + sudo mv -t /opt/dib_tmp/image.1UCql7CH/mnt /opt/dib_tmp/image.1UCql7CH/built/bin /opt/dib_tmp/image.1UCql7CH/built/boot /opt/dib_tmp/image.1UCql7CH/built/dev /opt/dib_tmp/image.1UCql7CH/built/etc /opt/dib_tmp/image.1UCql7CH/built/home /opt/dib_tmp/image.1UCql7CH/built/initrd.img /opt/dib_tmp/image.1UCql7CH/built/lib /opt/dib_tmp/image.1UCql7CH/built/lib64 /opt/dib_tmp/image.1UCql7CH/built/media /opt/dib_tmp/image.1UCql7CH/built/mnt /opt/dib_tmp/image.1UCql7CH/built/opt /opt/dib_tmp/image.1UCql7CH/built/proc /opt/dib_tmp/image.1UCql7CH/built/root /opt/dib_tmp/image.1UCql7CH/built/run /opt/dib_tmp/image.1UCql7CH/built/sbin /opt/dib_tmp/image.1UCql7CH/built/srv /opt/dib_tmp/image.1UCql7CH/built/sys /opt/dib_tmp/image.1UCql7CH/built/tmp /opt/dib_tmp/image.1UCql7CH/built/usr /opt/dib_tmp/image.1UCql7CH/built/var /opt/dib_tmp/image.1UCql7CH/built/vmlinuz
2016-01-25 20:25:57,599 INFO nodepool.image.build.trustyvm: mv: cannot remove '/opt/dib_tmp/image.1UCql7CH/built/dev/pts/0': Operation not permitted
2016-01-25 20:25:57,600 INFO nodepool.image.build.trustyvm: mv: cannot remove '/opt/dib_tmp/image.1UCql7CH/built/dev/pts/6': Operation not permitted
2016-01-25 20:25:57,600 INFO nodepool.image.build.trustyvm: mv: cannot remove '/opt/dib_tmp/image.1UCql7CH/built/dev/pts/5': Operation not permitted
2016-01-25 20:25:57,600 INFO nodepool.image.build.trustyvm: mv: cannot remove '/opt/dib_tmp/image.1UCql7CH/built/dev/pts/4': Operation not permitted
2016-01-25 20:25:57,601 INFO nodepool.image.build.trustyvm: mv: cannot remove '/opt/dib_tmp/image.1UCql7CH/built/dev/pts/3': Operation not permitted
2016-01-25 20:25:57,601 INFO nodepool.image.build.trustyvm: mv: cannot remove '/opt/dib_tmp/image.1UCql7CH/built/dev/pts/2': Operation not permitted
2016-01-25 20:25:57,601 INFO nodepool.image.build.trustyvm: mv: cannot remove '/opt/dib_tmp/image.1UCql7CH/built/dev/pts/1': Operation not permitted
2016-01-25 20:25:57,602 INFO nodepool.image.build.trustyvm: mv: cannot remove '/opt/dib_tmp/image.1UCql7CH/built/dev/pts/ptmx': Operation not permitted
2016-01-25 20:25:59,369 INFO nodepool.image.build.trustyvm: mv: error reading '/opt/dib_tmp/image.1UCql7CH/built/proc/sysrq-trigger': Input/output error
2016-01-25 20:25:59,369 INFO nodepool.image.build.trustyvm: mv: failed to extend '/opt/dib_tmp/image.1UCql7CH/mnt/proc/sysrq-trigger': Input/output error

After this point the build system is hosed. A reboot returns normal operation.

Here are the bind mounts inplace at the time of the mv:

root@ci:~# mount | grep bind
/mnt/opt/dib_cache/ccache on /mnt/opt/dib_tmp/image.OYzc7Uy3/mnt/tmp/ccache type none (rw,bind)
/mnt/opt/dib_cache/apt/ubuntu on /mnt/opt/dib_tmp/image.OYzc7Uy3/mnt/var/cache/apt/archives type none (rw,bind)
/dev on /mnt/opt/dib_tmp/image.OYzc7Uy3/mnt/dev type none (rw,bind)
/dev/pts on /mnt/opt/dib_tmp/image.OYzc7Uy3/mnt/dev/pts type none (rw,bind)

The system image on which the build is occurring is defined by the openstackci configuration
http://docs.openstack.org/infra/openstackci/third_party_ci.html

If disk-image-create is called by hand via the nodepool user account (with -t fixed) the same problem occurs:

TMPDIR=/opt/dib_tmp DIB_IMAGE_CACHE=/opt/dib_cache disk-image-create -x -t qcow2 --no-tmpfs -o /opt/nodepool_dib/6 ubuntu vm

Interestingly, if called from an ordinary user account the image build succeeds without breaking /dev:

# as ubuntu user
disk-image-create ubuntu vm

[John-Paul Robinson (jprorama)]

This is the log from the /var/log/nodepool/image/image.log for a build triggered by

nodepool image-build trustyvm

run from the nodepool account on the ci node defined by openstackci.

This triggers a background process from nodepoold:

/bin/bash /usr/local/bin/disk-image-create -x -t qcow2 --no-tmpfs --qemu-img-options compat=0.10 -o /opt/nodepool_dib/4 ubuntu vm

[John-Paul Robinson (jprorama)]

Here's a build from the command line as the ubuntu user (not nodepool) on the same box using the command line the is created with nodepoold triggers the build:

TMPDIR=/mnt/ubuntu/dib_tmp DIB_IMAGE_CACHE=/mnt/ubuntu/dib_cache /usr/local/bin/disk-image-create -x -t qcow2 --no-tmpfs -o /opt/nodepool_dib/6 ubuntu vm > dib-cli-image.log 2>&1

This build succeeds without impacting the /dev .

[John-Paul Robinson (jprorama)]

correction to last post. This is a better log of a successful build because I fixed the -o argument to write to place the user had permissions to write. Command that generated the log:

TMPDIR=/mnt/ubuntu/dib_tmp DIB_IMAGE_CACHE=/mnt/ubuntu/dib_cache /usr/local/bin/disk-image-create -x -t qcow2 --no-tmpfs -o ubuntuvm.qcow2 ubuntu vm > dib-cli-image-2.log 2>&1

[John-Paul Robinson (jprorama)]

Here is log that can be diff'd against dib-cli-image-2.log.gz above. This is produced my a functionally identical command line run of dib from the nodepool account.

TMPDIR=/opt/dib_tmp DIB_IMAGE_CACHE=/opt/dib_cache /usr/local/bin/disk-image-create -x -t qcow2 --no-tmpfs -o /opt/nodepool_dib/6 ubuntu vm > nodepool-dib-cli-image.log 2>&1

This command fails. The differences are predictable (path uniqueness) up until the sequence of mount commands and population the image.

I haven't pinpointed the difference but this diff should help

diff dib-cli-image-2.log nodepool-dib-cli-image.log

[John-Paul Robinson (jprorama)]

Here is a repeat of the nodepool account run with -x added to the disk-image-create script.

[John-Paul Robinson (jprorama)]

I'm working on a rebuild of my openstackci node to confirm the behavior across installs of the third party ci.

[John-Paul Robinson (jprorama)]

I've confirmed that a rebuilt ci node from a pristine openstack-ci setup produces the same odd behavior. The disk image create works when run under the stock ubuntu account but fails when running under the nodepool account. Confusing.

The ci image is built with the user-data file: http://paste.openstack.org/show/485097/

Launched with the command:

nova boot --flavor m1.large --image "ubuntu-14.04.3-server-cloudimg-amd64-disk1" --key-name mykey --security-groups default,ssh --user-data ./openstackci-base ci

Then follow the steps for setting up the ci node from http://docs.openstack.org/infra/openstackci/third_party_ci.html#configure-masterless-puppet

sudo su -
cp /etc/puppet/modules/openstackci/contrib/hiera.yaml /etc/puppet

cp /etc/puppet/modules/openstackci/contrib/single_node_ci_site.pp /etc/puppet/manifests/site.pp
cp /etc/puppet/modules/openstackci/contrib/single_node_ci_data.yaml /etc/puppet/environments/common.yaml

Edit common.yaml with the local config. Reference the project config:

https://gitlab.uabgrid.uab.edu/jpr/project-config-ci-test

Run puppet apply: http://docs.openstack.org/infra/openstackci/third_party_ci.html#run-masterless-puppet

Move the cache to /mnt (80GB) for more space:

sudo su -
cd /
rsync -a /opt /mnt/opt
mv opt opt-orig
ln -s /mnt/opt
mkdir /mnt/ubuntu
chown ubuntu.ubuntu /mnt/ubuntu
mkdir /mnt/ubuntu/dib_tmp

Start zuul.

Then test the build of a simple image:

As the Ubuntu user it succeeds:

TMPDIR=/mnt/ubuntu/dib_tmp DIB_IMAGE_CACHE=/mnt/ubuntu/dib_cache disk-image-create -x -t qcow2 --no-tmpfs -o ubuntuvm.qcow2 ubuntu vm

As the nodepool user it fails, breaking /dev:

TMPDIR=/opt/dib_tmp DIB_IMAGE_CACHE=/opt/dib_cache disk-image-create -x -t qcow2 --no-tmpfs -o ubuntuvm.qcow2 ubuntu vm

[John-Paul Robinson (jprorama)]

Additional log from run with diskimage-builder debug patch installed via:

git clone https://git.openstack.org/openstack/diskimage-builder
cd diskimage-builder/
git review -s
git review -d 273171
sudo -H pip install .

Command run under nodepool:

TMPDIR=/opt/dib_tmp DIB_IMAGE_CACHE=/opt/dib_cache disk-image-create -x -t qcow2 --no-tmpfs -o ubuntuvm.qcow2 ubuntu vm > debug-nodepool-cli-image-ci2.log 2>&1

[John-Paul Robinson (jprorama)]

After applying the debug build of dib, it reported the current mount pattern as /opt/dib_tmp/image.3uTEKgO2/mnt but /proc/mounts has the absolute path of the mount recorded as /mnt/opt/dib_tmp/image.3uTEKgO2/mnt/dev. This fails the pattern match check in the code:

http://git.openstack.org/cgit/openstack/diskimage-builder/tree/lib/common-functions#n286

This symlink comes from the use of the local x1.large instance that provides the 80G drive on /mnt and moving /opt the /mnt/opt via a symlink as described in comment #7 above.

The fix/workaround is to remount the 80G drive under /opt. The dib create that succeeds under the nodepool account. The reason for earlier success under ubuntu was that the dib cache/tmp dirs were specified with the full path under /mnt/ubuntu so the umount command succeeded.

The relevant part of the debug output is:

+ unmount_dir /opt/dib_tmp/image.3uTEKgO2/mnt
+ local pattern=/opt/dib_tmp/image.3uTEKgO2/mnt mnts=
+ echo 'Unmounting dirs matching pattern '\''/opt/dib_tmp/image.3uTEKgO2/mnt'\'''
Unmounting dirs matching pattern '/opt/dib_tmp/image.3uTEKgO2/mnt'
+ echo 'Current mounts:'
Current mounts:
+ cat /proc/mounts
rootfs / rootfs rw 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
udev /dev devtmpfs rw,relatime,size=4082832k,nr_inodes=1020708,mode=755 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=817680k,mode=755 0 0
/dev/disk/by-uuid/ee1a8759-f2e2-4924-a2a3-505c432325bc / ext4 rw,relatime,data=ordered 0 0
none /sys/fs/cgroup tmpfs rw,relatime,size=4k,mode=755 0 0
none /sys/fs/fuse/connections fusectl rw,relatime 0 0
none /sys/kernel/debug debugfs rw,relatime 0 0
none /sys/kernel/security securityfs rw,relatime 0 0
none /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0
none /run/shm tmpfs rw,nosuid,nodev,relatime 0 0
none /run/user tmpfs rw,nosuid,nodev,noexec,relatime,size=102400k,mode=755 0 0
none /sys/fs/pstore pstore rw,relatime 0 0
/dev/vdb /mnt ext3 rw,relatime,data=ordered 0 0
systemd /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,name=systemd 0 0
/dev/vdb /mnt/opt/dib_tmp/image.iYEI5csR/mnt/tmp/ccache ext3 rw,relatime,data=ordered 0 0
/dev/vdb /mnt/opt/dib_tmp/image.3uTEKgO2/mnt/tmp/ccache ext3 rw,relatime,data=ordered 0 0
/dev/vdb /mnt/opt/dib_tmp/image.3uTEKgO2/mnt/var/cache/apt/archives ext3 rw,relatime,data=ordered 0 0
none /mnt/opt/dib_tmp/image.3uTEKgO2/mnt/proc proc rw,relatime 0 0
udev /mnt/opt/dib_tmp/image.3uTEKgO2/mnt/dev devtmpfs rw,relatime,size=4082832k,nr_inodes=1020708,mode=755 0 0
devpts /mnt/opt/dib_tmp/image.3uTEKgO2/mnt/dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
none /mnt/opt/dib_tmp/image.3uTEKgO2/mnt/sys sysfs rw,relatime 0 0
+ '[' -n /opt/dib_tmp/image.3uTEKgO2/mnt ']'
+ awk '{print $2}'
+ grep '^/opt/dib_tmp/image.3uTEKgO2/mnt'
+ '[' -n '' ']'
+ '[' -n '' ']'
+ mv /opt/dib_tmp/image.3uTEKgO2/mnt /opt/dib_tmp/image.3uTEKgO2/built