extlinux cores during build of centos

[Amrith Kumar]

On Ubuntu 16.04 machine I attempted to build a centos image

disk-image-create -o centos64 -t qcow2 -a amd64 -x centos vm

It fails with the following (full log is attached):

2017-06-17 10:37:08.508 INFO diskimage_builder.block_device.utils [-] Calling [sudo umount /tmp/dib_build.ngDM1r2M/mnt/]
2017-06-17 10:37:08.515 DEBUG diskimage_builder.block_device.utils [-] exec_sudo: umount: /tmp/dib_build.ngDM1r2M/mnt/: target is busy exec_sudo /usr/local/lib/python2.7/dist-packages/diskimage_builder/block_device/utils.py:120
2017-06-17 10:37:08.516 DEBUG diskimage_builder.block_device.utils [-] exec_sudo: (In some cases useful info about processes that exec_sudo /usr/local/lib/python2.7/dist-packages/diskimage_builder/block_device/utils.py:120
2017-06-17 10:37:08.516 DEBUG diskimage_builder.block_device.utils [-] exec_sudo: use the device is found by lsof(8) or fuser(1).) exec_sudo /usr/local/lib/python2.7/dist-packages/diskimage_builder/block_device/utils.py:120
Traceback (most recent call last):
  File "/usr/local/bin/dib-block-device", line 10, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/diskimage_builder/block_device/cmd.py", line 120, in main
    return bdc.main()
  File "/usr/local/lib/python2.7/dist-packages/diskimage_builder/block_device/cmd.py", line 115, in main
    self.args.func()
  File "/usr/local/lib/python2.7/dist-packages/diskimage_builder/block_device/cmd.py", line 39, in cmd_umount
    self.bd.cmd_umount()
  File "/usr/local/lib/python2.7/dist-packages/diskimage_builder/block_device/blockdevice.py", line 420, in cmd_umount
    node.umount()
  File "/usr/local/lib/python2.7/dist-packages/diskimage_builder/block_device/level3/mount.py", line 98, in umount
    exec_sudo(["umount", self.state['mount'][self.mount_point]['path']])
  File "/usr/local/lib/python2.7/dist-packages/diskimage_builder/block_device/utils.py", line 125, in exec_sudo
    ' '.join(sudo_cmd))
subprocess.CalledProcessError: Command 'sudo umount /tmp/dib_build.ngDM1r2M/mnt/' returned non-zero exit status 32

[Andreas Florath (ansreas)]

Thanks for the report.

The failure are the following lines:

+ extlinux --install /boot/extlinux\
/boot/extlinux is device /dev/mapper/loop0p1\
/tmp/in_target.d/finalise.d/50-bootloader: line 18: 31100 Segmentation fault (core dumped) extlinux --install $EXTDIR\

(The failing umount logs looks like a follow up error.)

[Ian Wienand (iwienand)]

Actually this is a red herring, the problem is before this at

/tmp/in_target.d/finalise.d/50-bootloader: line 18: 31100 Segmentation fault (core dumped) extlinux --install $EXTDIR\

do you need extlinux? i haven't seen that issue before, but extlinux is probably not well tested

[Ian Wienand (iwienand)]

Oh, also, you probably want "centos7" ... i admit it's very confusing

[Amrith Kumar (amrith)]

And that worked, with the centos7 that is.

Thanks Ian!

[Luigi Toscano (ltoscano)]

Apart from normally using centos7, the bug still applies to CentOS 6. In Sahara we still support CentOS 6 images with Hadoop stuff; if this is not addressed we should end up deprecating them, I guess (at least removing from the CI).

[Clenimar Filemon (clenimar-filemon)]

Just hit this when trying to build a fedora-atomic image.

disk-image-create fedora-atomic -o fedora-atomic

Following the instructions on magnum repo: https://github.com/openstack/magnum/tree/master/magnum/drivers/common/image/fedora-atomic

Fails with:

2017-10-31 00:10:56.032 INFO diskimage_builder.block_device.level3.mount [-] Called for [mount_mkfs_root]
2017-10-31 00:10:56.032 INFO diskimage_builder.block_device.utils [-] Calling [sudo umount /tmp/dib_build.Ov21ervR/mnt/]
Traceback (most recent call last):
  File "/home/fedora/bin/dib-block-device", line 11, in <module>
    sys.exit(main())
  File "/home/fedora/lib/python2.7/site-packages/diskimage_builder/block_device/cmd.py", line 120, in main
    return bdc.main()
  File "/home/fedora/lib/python2.7/site-packages/diskimage_builder/block_device/cmd.py", line 115, in main
    self.args.func()
  File "/home/fedora/lib/python2.7/site-packages/diskimage_builder/block_device/cmd.py", line 39, in cmd_umount
    self.bd.cmd_umount()
  File "/home/fedora/lib/python2.7/site-packages/diskimage_builder/block_device/blockdevice.py", line 420, in cmd_umount
    node.umount()
  File "/home/fedora/lib/python2.7/site-packages/diskimage_builder/block_device/level3/mount.py", line 98, in umount
    exec_sudo(["umount", self.state['mount'][self.mount_point]['path']])
  File "/home/fedora/lib/python2.7/site-packages/diskimage_builder/block_device/utils.py", line 125, in exec_sudo
    ' '.join(sudo_cmd))
subprocess.CalledProcessError: Command 'sudo umount /tmp/dib_build.Ov21ervR/mnt/' returned non-zero exit status 32

[Clenimar Filemon (clenimar-filemon)]

I could reproduce it in virtual machines running Ubuntu 16.04, Fedora 25 and Fedora 26. Any hint?

[Ian Wienand (iwienand)]

@Clenimar -- the real error will be well before this. Please use --logfile (on master) or just redirect to a logfile and send us the full output

[Andreas Florath (ansreas)]

@Clenimar:

1 - This is a completely different problem than the initial: please file a separate bug.

2 - IMHO the problem is in magnum/../finalize.d/80-fedora-atomic
    In my environment this terminates with:
       error: opendir(ostree/repo): No such file or directory
    Please see
       https://review.openstack.org/#/c/516886/
    for a proposed fix

    Nevertheless there seams to be an additional problem with the file hierarchy:

       2017-11-01 08:12:41.548 | + ostree remote add --set=gpg-verify=false fedora-atomic https://kojipkgs.fedoraproject.org/atomic/26/
       2017-11-01 08:12:41.554 | + ostree pull fedora-atomic '<!DOCTYPE' HTML PUBLIC '"-//IETF//DTD' HTML '2.0//EN">' '<html><head>' '<title>404' Not 'Found</title>' '</head><body>' '<h1>Not' 'Found</h1>' '<p>The' requested URL /atomic/26//refs/heads/fedora-atomic/26/x86_64/docker-host was not found on this 'server.</p>' '</body></html>'
       2017-11-01 08:12:42.598 | error: Invalid ref name <!DOCTYPE

[Clenimar Filemon (clenimar-filemon)]

Thanks for replying, Ian and Andreas.

@Andreas: yes you're right. #1719102 is more suitable in this case. I'll take the log file and post there.

Regarding magnum, I think those problems are happening because you're using DIB_RELEASE=26. The docker_host ref magnum relies on is not available on the Fedora 26 repo. Use the 25 repo and it will be good to go (no need to initfs).

[Andreas Florath (ansreas)]

Sorry, but IMHO also #1719102 is something completely different; I'd open a new bug.

In the meantime:

When using fedora 25, it fails in first place with:

2017-11-01 19:04:44.530 | + ostree admin os-init fedora-atomic
2017-11-01 19:04:44.538 | ostree/deploy/fedora-atomic initialized as OSTree root
2017-11-01 19:04:44.539 | + ostree remote add --set=gpg-verify=false fedora-atomic https://kojipkgs.fedoraproject.org/atomic/25/
2017-11-01 19:04:44.548 | + ostree pull fedora-atomic 721fddd24ca1e4cd7b03ca5c7a9382dbb8adfd23eb97cf603d1091c24d1822cb
2017-11-01 19:09:24.669 | error: min-free-space-percent '3%' would be exceeded, 13.7 MB more required

The problem is, that the size computation is IMHO done 'too early' (see https://review.openstack.org/#/c/512085) - and here some disk space is used during the finalize-phase.

As a (temporary) work-around I'd suggest using 'disk-image-create --image-size 5 ...' or similar.

After fixing this, the image creation fails with:

2017-11-01 19:38:08.814 | /tmp/in_target.d/finalise.d/80-fedora-atomic: line 29: DIB_IMAGE_ROOT_FS_UUID: unbound variable

The 'new' block level layer does currently not provide the UUID of the root partition. For other proposes the LABEL is used which is stored in DIB_ROOT_LABEL.

If you really need the UUID here, you need to patch
    diskimage-builder/diskimage_builder/block_device/blockdevice.py line 285ff
and add a block similar to root-label; adapt
    diskimage-builder/diskimage_builder/lib/disk-image-create
similar to line 309ff to store the result in a variable.