The libguestfs versions < 1.37.22 don't work on s390x, as they don't contain the patches which include the s390x specifics to instruct qemu-kvm correctly. Ubuntu 16.04 contains an older version than that, which makes libguestfs not usable on s390x. Steps to reproduce ================== A chronological list of steps which will bring off the issue I noticed: mz@s390xhost$ docker run -it --privileged --rm ubuntu:xenial bash root@5dde0aef5d2e:/# root@5dde0aef5d2e:/# root@5dde0aef5d2e:/# apt update root@5dde0aef5d2e:/# DEBIAN_FRONTEND=noninteractive apt install -y \ libguestfs-tools \ python-libguestfs \ linux-image-generic \ qemu root@5dde0aef5d2e:/# root@5dde0aef5d2e:/# root@5dde0aef5d2e:/# export LIBGUESTFS_DEBUG=1 root@5dde0aef5d2e:/# export LIBGUESTFS_TRACE=1 root@5dde0aef5d2e:/# libguestfs-test-tool Expected result =============== The libguestfs-test-tool passes and I can use it on s390x. To check if qemu on s390x works independently from libguestfs, I use this command: root@5dde0aef5d2e:/# qemu-system-s390x \ -enable-kvm \ -nographic \ -kernel /boot/vmlinuz-4.4.0-119-generic \ -initrd /boot/initrd.img-4.4.0-119-generic \ -m 1G \ -M s390-ccw-virtio [...] # boot BusyBox v1.22.1 (Ubuntu 1:1.22.0-15ubuntu1) built-in shell (ash) Enter 'help' for a list of built-in commands. (initramfs) Actual result ============= libguestfs instructs qemu to use pci devices and serial consoles which doesn't work on s390x, e.g.: root@5dde0aef5d2e:/# libguestfs-test-tool ************************************************************ * IMPORTANT NOTICE * * When reporting bugs, include the COMPLETE, UNEDITED * output below in your bug report. * ************************************************************ libguestfs: trace: set_verbose true libguestfs: trace: set_verbose = 0 libguestfs: trace: set_verbose true libguestfs: trace: set_verbose = 0 LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin SELinux: sh: 1: getenforce: not found libguestfs: trace: add_drive_scratch 104857600 libguestfs: trace: get_tmpdir libguestfs: trace: get_tmpdir = "/tmp" libguestfs: trace: disk_create "/tmp/libguestfs82Gj1d/scratch.1" "raw" 104857600 libguestfs: trace: disk_create = 0 libguestfs: trace: add_drive "/tmp/libguestfs82Gj1d/scratch.1" "format:raw" "cachemode:unsafe" libguestfs: trace: add_drive = 0 libguestfs: trace: add_drive_scratch = 0 libguestfs: trace: get_append libguestfs: trace: get_append = "NULL" guestfs_get_append: (null) libguestfs: trace: get_autosync libguestfs: trace: get_autosync = 1 guestfs_get_autosync: 1 libguestfs: trace: get_backend libguestfs: trace: get_backend = "direct" guestfs_get_backend: direct libguestfs: trace: get_backend_settings libguestfs: trace: get_backend_settings = [] guestfs_get_backend_settings: [] libguestfs: trace: get_cachedir libguestfs: trace: get_cachedir = "/var/tmp" guestfs_get_cachedir: /var/tmp libguestfs: trace: get_direct libguestfs: trace: get_direct = 0 guestfs_get_direct: 0 libguestfs: trace: get_hv libguestfs: trace: get_hv = "/usr/bin/qemu-system-s390x" guestfs_get_hv: /usr/bin/qemu-system-s390x libguestfs: trace: get_memsize libguestfs: trace: get_memsize = 500 guestfs_get_memsize: 500 libguestfs: trace: get_network libguestfs: trace: get_network = 0 guestfs_get_network: 0 libguestfs: trace: get_path libguestfs: trace: get_path = "/usr/lib/s390x-linux-gnu/guestfs" guestfs_get_path: /usr/lib/s390x-linux-gnu/guestfs libguestfs: trace: get_pgroup libguestfs: trace: get_pgroup = 0 guestfs_get_pgroup: 0 libguestfs: trace: get_program libguestfs: trace: get_program = "libguestfs-test-tool" guestfs_get_program: libguestfs-test-tool libguestfs: trace: get_recovery_proc libguestfs: trace: get_recovery_proc = 1 guestfs_get_recovery_proc: 1 libguestfs: trace: get_selinux libguestfs: trace: get_selinux = 0 guestfs_get_selinux: 0 libguestfs: trace: get_smp libguestfs: trace: get_smp = 1 guestfs_get_smp: 1 libguestfs: trace: get_tmpdir libguestfs: trace: get_tmpdir = "/tmp" guestfs_get_tmpdir: /tmp libguestfs: trace: get_trace libguestfs: trace: get_trace = 1 guestfs_get_trace: 1 libguestfs: trace: get_verbose libguestfs: trace: get_verbose = 1 guestfs_get_verbose: 1 host_cpu: s390x Launching appliance, timeout set to 600 seconds. libguestfs: trace: launch libguestfs: trace: version libguestfs: trace: version = libguestfs: trace: get_backend libguestfs: trace: get_backend = "direct" libguestfs: launch: program=libguestfs-test-tool libguestfs: launch: version=1.32.2 libguestfs: launch: backend registered: unix libguestfs: launch: backend registered: uml libguestfs: launch: backend registered: libvirt libguestfs: launch: backend registered: direct libguestfs: launch: backend=direct libguestfs: launch: tmpdir=/tmp/libguestfs82Gj1d libguestfs: launch: umask=0022 libguestfs: launch: euid=0 libguestfs: trace: get_backend_setting "force_tcg" libguestfs: trace: get_backend_setting = NULL (error) libguestfs: trace: get_cachedir libguestfs: trace: get_cachedir = "/var/tmp" libguestfs: [00000ms] begin building supermin appliance libguestfs: [00000ms] run supermin libguestfs: command: run: /usr/bin/supermin libguestfs: command: run: \ --build libguestfs: command: run: \ --verbose libguestfs: command: run: \ --if-newer libguestfs: command: run: \ --lock /var/tmp/.guestfs-0/lock libguestfs: command: run: \ --copy-kernel libguestfs: command: run: \ -f ext2 libguestfs: command: run: \ --host-cpu s390x libguestfs: command: run: \ /usr/lib/s390x-linux-gnu/guestfs/supermin.d libguestfs: command: run: \ -o /var/tmp/.guestfs-0/appliance.d supermin: version: 5.1.14 supermin: package handler: debian/dpkg supermin: acquiring lock on /var/tmp/.guestfs-0/lock supermin: build: /usr/lib/s390x-linux-gnu/guestfs/supermin.d supermin: reading the supermin appliance supermin: build: visiting /usr/lib/s390x-linux-gnu/guestfs/supermin.d/base.tar.gz type gzip base image (tar) supermin: build: visiting /usr/lib/s390x-linux-gnu/guestfs/supermin.d/daemon.tar.gz type gzip base image (tar) supermin: build: visiting /usr/lib/s390x-linux-gnu/guestfs/supermin.d/excludefiles type uncompressed excludefiles supermin: build: visiting /usr/lib/s390x-linux-gnu/guestfs/supermin.d/hostfiles type uncompressed hostfiles supermin: build: visiting /usr/lib/s390x-linux-gnu/guestfs/supermin.d/init.tar.gz type gzip base image (tar) supermin: build: visiting /usr/lib/s390x-linux-gnu/guestfs/supermin.d/packages type uncompressed packages supermin: build: visiting /usr/lib/s390x-linux-gnu/guestfs/supermin.d/packages-hfsplus type uncompressed packages supermin: build: visiting /usr/lib/s390x-linux-gnu/guestfs/supermin.d/packages-reiserfs type uncompressed packages supermin: build: visiting /usr/lib/s390x-linux-gnu/guestfs/supermin.d/packages-xfs type uncompressed packages supermin: build: visiting /usr/lib/s390x-linux-gnu/guestfs/supermin.d/udev-rules.tar.gz type gzip base image (tar) supermin: mapping package names to installed packages supermin: resolving full list of package dependencies supermin: build: 194 packages, including dependencies supermin: build: 6751 files supermin: build: 3694 files, after matching excludefiles supermin: build: 3696 files, after adding hostfiles supermin: build: 3696 files, after removing unreadable files supermin: build: 3700 files, after munging supermin: kernel: picked kernel vmlinuz-4.4.0-119-generic supermin: kernel: picked modules path /lib/modules/4.4.0-119-generic supermin: kernel: kernel_version 4.4.0-119-generic supermin: kernel: modules /lib/modules/4.4.0-119-generic supermin: ext2: creating empty ext2 filesystem '/var/tmp/.guestfs-0/appliance.d.s5hkw6ja/root' supermin: ext2: populating from base image supermin: ext2: copying files from host filesystem supermin: ext2: copying kernel modules supermin: ext2: creating minimal initrd '/var/tmp/.guestfs-0/appliance.d.s5hkw6ja/initrd' supermin: ext2: wrote 9 modules to minimal initrd supermin: renaming /var/tmp/.guestfs-0/appliance.d.s5hkw6ja to /var/tmp/.guestfs-0/appliance.d libguestfs: [01090ms] finished building supermin appliance libguestfs: [01090ms] begin testing qemu features libguestfs: command: run: /usr/bin/qemu-system-s390x libguestfs: command: run: \ -display none libguestfs: command: run: \ -help libguestfs: command: run: /usr/bin/qemu-system-s390x libguestfs: command: run: \ -display none libguestfs: command: run: \ -version libguestfs: qemu version 2.5 libguestfs: command: run: /usr/bin/qemu-system-s390x libguestfs: command: run: \ -display none libguestfs: command: run: \ -machine accel=kvm:tcg libguestfs: command: run: \ -device ? libguestfs: [01166ms] finished testing qemu features libguestfs: trace: get_backend_setting "gdb" libguestfs: trace: get_backend_setting = NULL (error) [01172ms] /usr/bin/qemu-system-s390x \ -global virtio-blk-pci.scsi=off \ -nodefconfig \ -enable-fips \ -nodefaults \ -display none \ -machine accel=kvm:tcg \ -cpu host \ -m 500 \ -no-reboot \ -rtc driftfix=slew \ -global kvm-pit.lost_tick_policy=discard \ -kernel /var/tmp/.guestfs-0/appliance.d/kernel \ -initrd /var/tmp/.guestfs-0/appliance.d/initrd \ -device virtio-scsi-pci,id=scsi \ -drive file=/tmp/libguestfs82Gj1d/scratch.1,cache=unsafe,format=raw,id=hd0,if=none \ -device scsi-hd,drive=hd0 \ -drive file=/var/tmp/.guestfs-0/appliance.d/root,snapshot=on,id=appliance,cache=unsafe,if=none \ -device scsi-hd,drive=appliance \ -device virtio-serial-pci \ -serial stdio \ -chardev socket,path=/tmp/libguestfs82Gj1d/guestfsd.sock,id=channel0 \ -device virtserialport,chardev=channel0,name=org.libguestfs.channel.0 \ -append 'panic=1 console=ttyS0 udevtimeout=6000 udev.event-timeout=6000 no_timer_check acpi=off printk.time=1 cgroup_disable=memory root=/dev/sdb selinux=0 guestfs_verbose=1 TERM=xterm' qemu-system-s390x: Warning: global kvm-pit.lost_tick_policy has invalid class name Alarm clock Environment =========== This affects ubuntu 16.04 (see below) host ---- $ lsb_release -a LSB Version: core-9.20160110ubuntu0.2-noarch:core-9.20160110ubuntu0.2-s390x:printing-9.20160110ubuntu0.2-noarch:printing-9.20160110ubuntu0.2-s390x:security-9.20160110ubuntu0.2-noarch:security-9.20160110ubuntu0.2-s390x Distributor ID: Ubuntu Description: Ubuntu 16.04.4 LTS Release: 16.04 Codename: xenial container --------- root@5dde0aef5d2e:/# lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 16.04.3 LTS Release: 16.04 Codename: xenial root@5dde0aef5d2e:/# apt-cache policy libguestfs-tools libguestfs-tools: Installed: 1:1.32.2-4ubuntu2 Candidate: 1:1.32.2-4ubuntu2 Version table: *** 1:1.32.2-4ubuntu2 500 500 http://ports.ubuntu.com/ubuntu-ports xenial/universe s390x Packages 100 /var/lib/dpkg/status Additional information ====================== Please be aware that there this is a known libguestfs upstream bug [1] which is already fixed and backported by RedHat [2][3] and was the trigger for a patch series on the libguestfs mailing list [4]. As an example, the latest known needed patch is [5]. As a smoke test, in case the libguestfs-test-tool result was a false positive, I used the python binding code in Appendix A below, which is based on [6]. FWIW, I checked Ubuntu 18.04 (beta), and it doesn't work there too. The packaged version of libguestfs is v1.36 in 18.04 bionic beaver, which is too old (and/or doesn't contain the needed backports). References ========== [1] Bug 1376547 - qemu-system-s390x: -device isa-serial,chardev=charserial0, id=serial0: 'isa-serial' is not a valid device model name https://bugzilla.redhat.com/show_bug.cgi?id=1376547 [2] [Private] Bug 1479526 - [RFE] libguestfs for z Systems https://bugzilla.redhat.com/show_bug.cgi?id=1479526 [3] The RHEL7.5 branch of libguestfs with backports to v1.36 https://github.com/libguestfs/libguestfs/tree/rhel-7.5 [4] The patch series of libguestfs on s390x: https://www.redhat.com/archives/libguestfs/2017-May/msg00066.html [5] Fix of an x86 specific item to enable s390x in v1.37.22 https://github.com/libguestfs/libguestfs/commit/5b60dd4eff02f48d344bcdad0d4bad4676ca9168 [6] https://rwmj.wordpress.com/2010/02/01/libguestfs-python-example/ Appendix A ========== Smoke test, based on the example from the author of libguestfs [6]: root@26248e573a87:/# cat smoke.py #!/usr/bin/python # To see the Python docs, do: help (guestfs) # To see the general docs http://libguestfs.org/guestfs.3.html # For FAQ see http://libguestfs.org/FAQ.html import os import guestfs g = guestfs.GuestFS() f = open("/tmp/test.img", "w") f.truncate(500 * 1024 * 1024) f.close() g.add_drive("/tmp/test.img") g.launch() g.part_disk("/dev/sda", "mbr") g.mkfs("ext2", "/dev/sda1") g.mount("/dev/sda1", "/") f = open("/tmp/ks.cfg", "w") f.write("# this would be the kickstart file\n") f.write("#\n") f.write ("# end of file\n") f.close() g.upload("/tmp/ks.cfg", "/ks.cfg") g.sync() g.umount_all() == Comment: #1 - Markus Zoeller