Adding s390x support to DIB

[Andreas Scheuring]

This is a feature request to add the ability to build s390x images with diskimage-builder.

Use Cases
---------
* Allow users to create qcow2 image that can be used on a s390x nova libvirt/kvm node
* Enable nodepool image build for third party CI

Supported Distros
-----------------

The following Distros are officially supported on s390x architecture.

* RHEL - will not be supported by DIB, as rhel is not supported as a kvm guest on s390x and therefore there's no s390x cloud image like it is for other arches (this is required by the rhel7 DIB element)
* SLES - no support for SLES in DIB
* Ubuntu - full support in DIB

Testing
-------

There are limitation with building s390x images on other architectures. Therfore the plan is to provide a Third Party CI job that builds the s390x images and provides the logfiles. A similar CI already exists for nova [1]

Technical Details
-----------------

s390x does not support the grub2 bootloader. Instead of grub "zipl" [2] must be used.

new zipl element vs. existing bootloader element
++++++++++++++++++++++++++++++++++++++++++++++++

The challenge is, that the bootloader element (which contains the grub setup) is dependency of the vm element [3]. It will be called implicitly everytime the element "vm" is specified. Now 2 options exist

#1 extend the bootloader element to support zipl
#2 Implement a new element that "element-provides" bootloader. When explicitly specified as element in the commandline, the zipl element will override the "bootloader" element that is pulled in as dependency. For example:

    diskimage-create ubuntu-minimal vm zipl

The second approach has been chosen (not sure about the exact reason - the patch has a long history)

zipl configuration
++++++++++++++++++

Zipl requires a zipl.conf file being present at "/etc/zipl.conf". More details about this file see [4]. Basically it contains the location of the kernel and initramfs as well as the parmline.

* The location of the kernel and the initramfs is hardcoded to /boot/*.
* A default cmdline will be implemented, with the option for an overrride

zipl advanced block/sector settings
+++++++++++++++++++++++++++++++++++

The patch has some history [5] (outdated) [6] (current). Some versions of the patch where using a zipl helper file (see [7][8] for more information on zipl helper files) which did some magic to the block/sector settings. BTW, all those settings can also be done via zipl command options [9].

Let's understand it:

Disk image builder uses the following defaults
* Disk logical sector = physical sector size: 512B [10]
* Alignment: 1MiB [11] (configurable via [14])
(* File system block size: 4096B [12])

Alignment means, that a partition starts at the 1 MiB boundary.

The following parameters can be specified to zipl [8]

* targetbase -> The device that holds the boot volume
* targetblocksize -> the blocksize of the target device (in DIB always 512B) [10]
* targetoffset -> the block at which the boot partition starts
* targettype -> always "scsi" for s390x KVM VMs

We need to distinguish between a partitioned setup and a non partitioned setup [15].

Partitioned setup
~~~~~~~~~~~~~~~~~

Only MBR is supported (no GPT partitioning) [13].
* MBR supports up to 4 primary partitions.
* One of the primary partitions can become a extended partition.
* In an extended partition many logical partitions can be created.

MBR consumes 512 Byte. Due to the alignment the next (1MiB -512 Bytes) are unused. The first partition starts at 1 MiB which is block 2048 ((1024 * 1024)B / 512 Bytes per Block).

-> zipls targetoffset should be set to 2048 when the first partition is the boot partition.

Luckily zipl is able to calculate the starting block for primary and logical partitions as well as for an lvm setup and all the other parameters correctly. It uses the zipl_helper.device-mapper file which is shipped with the operating system. In the following example the boot partition is the second partition:

Building a partitioned image:
export DIB_DEV_USER_USERNAME=devuser
export DIB_DEV_USER_PWDLESS_SUDO=1
export DIB_DEV_USER_PASSWORD=devuser

export DIB_BLOCK_DEVICE_CONFIG='
  - local_loop:
      name: image0
      size: 3GB
  - partitioning:
      base: image0
      label: mbr
      partitions:
        - name: root
          flags: [ primary ]
          size: 50%
        - name: boot
          flags: [boot, primary]
          size: 50%
  - mkfs:
      name: root_fs
      base: root
      label: cloudimage-root
      type: ext4
      mount:
        mount_point: /
        fstab:
          options: "defaults"
          fsck-passno: 1
  - mkfs:
      name: boot_fs
      base: boot
      label: cloudimage-boot
      type: ext4
      mount:
        mount_point: /boot
        fstab:
          options: "defaults"
          fsck-passno: 1
'

disk-image-create -x -x ubuntu-minimal vm zipl devuser &> build.log

root@ubuntu:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda 253:0 0 2.8G 0 disk
├─vda1 253:1 0 1.4G 0 part /
└─vda2 253:2 0 714.5M 0 part /boot
root@ubuntu:~# zipl -V
Using config file '/etc/zipl.conf'
Target device information
  Device..........................: fd:00
  Partition.......................: fd:02
  Device name.....................: vda
  Device driver name..............: virtblk
  Type............................: disk partition
  Disk layout.....................: SCSI disk layout
  Geometry - start................: 2930688
  File system block size..........: 4096
  Physical block size.............: 512
  Device size in physical blocks..: 1463319
Building bootmap in '/boot'
Adding IPL section 'ubuntu' (default)
  initial ramdisk...: /boot/initrd.img
  kernel image......: /boot/vmlinuz
  kernel parmline...: 'root=LABEL=cloudimage-root LANG=en_US.UTF-8 console=ttyS0 console=ttyS1'
  component address:
    kernel image....: 0x00010000-0x003897ff
    parmline........: 0x0038a000-0x0038a1ff
    initial ramdisk.: 0x00390000-0x00a215ff
    internal loader.: 0x0000a000-0x0000c3ff
Preparing boot device: vda (0000).
Detected SCSI PCBIOS disk layout.
Writing SCSI master boot record.
Syncing disks...
Done.

-> Geometry - start (targetoffset) is calculated automatically to the (right) first block of the boot partition.

Non-partitioned setup
~~~~~~~~~~~~~~~~~~~~~

In this setup no partitioning (also no MBR) is being created. The
filesystem directly starts at block 0. This can be done by not specifying the "vm" element during image build, e.g.

    diskimage-create ubuntu-minimal zipl

zipl fails during image build if no partition is defined

2017-11-07 13:08:25.013 | + zipl -V
2017-11-07 13:08:25.014 | Error: Could not get disk geometry

The reason is, that the unpartitioned disk is not detected by the device mapper?? At least the existing zipl_helper (/lib/s390-tools/zipl_helper.device-mapper) is not being issued.

A new zipl_helper script that can handle plain loop devices is required during image build with the following parameters

* targetbase: /dev/loopX (the base device)
* targetblocksize: 512B (more details see above)
* targetoffset: 0
* targettype: scsi (more details see above)

This helper file is only required during image build and can therefore be removed again after zipl has been called. When zipl is run inside a unpartitioned running vm, the disk is detected as device mapper device.

Building a non partitioned image:

export DIB_DEV_USER_USERNAME=devuser
export DIB_DEV_USER_PWDLESS_SUDO=1
export DIB_DEV_USER_PASSWORD=devuser

disk-image-create -x -x ubuntu-minimal zipl devuser &> build_no_part.log

When zipl is run inside a VM, the output looks like this:

# zipl -V
Using config file '/etc/zipl.conf'
Target device information
  Device..........................: fd:00
  Device name.....................: vda
  Device driver name..............: virtblk
  Type............................: disk device
  Disk layout.....................: SCSI disk layout
  Geometry - start................: 0
  File system block size..........: 4096
  Physical block size.............: 512
  Device size in physical blocks..: 2302208
Building bootmap in '/boot'
Adding IPL section 'ubuntu' (default)
  initial ramdisk...: /boot/initrd.img
  kernel image......: /boot/vmlinuz
  kernel parmline...: 'root=LABEL=cloudimg-rootfs LANG=en_US.UTF-8 console=ttyS0 console=ttyS1'
  component address:
    kernel image....: 0x00010000-0x003897ff
    parmline........: 0x0038a000-0x0038a1ff
    initial ramdisk.: 0x00390000-0x00a215ff
    internal loader.: 0x0000a000-0x0000c3ff
Preparing boot device: vda (0000).
Detected plain SCSI partition.
Writing SCSI master boot record.
Syncing disks...
Done.

[1] https://wiki.openstack.org/wiki/ThirdPartySystems/IBM_zKVM_CI
[2] https://www.ibm.com/support/knowledgecenter/en/linuxonibm/com.ibm.linux.z.ludd/ludd_r_zipl_cmd.html
[3] https://github.com/openstack/diskimage-builder/blob/906a3f4a57b574d7809531a5d9e9c9f814fd9eab/diskimage_builder/elements/vm/element-deps#L1
[4] https://www.ibm.com/support/knowledgecenter/en/linuxonibm/com.ibm.linux.z.ludd/ludd_r_zipl_conf_struct.html#
[5] https://review.openstack.org/386031
[6] https://review.openstack.org/#/c/443548
[7] https://www.ibm.com/support/knowledgecenter/en/linuxonibm/com.ibm.linux.z.lgdd/lgdd_r_zipl_lboot_script.html
[8] https://www.ibm.com/support/knowledgecenter/en/linuxonibm/com.ibm.linux.z.lgdd/lgdd_r_zipl_lhelp_script.html
[9] https://www.ibm.com/support/knowledgecenter/en/linuxonibm/com.ibm.linux.z.ludd/ludd_r_ipl_cmd_syn.html
[10] https://github.com/openstack/diskimage-builder/blob/906a3f4a57b574d7809531a5d9e9c9f814fd9eab/diskimage_builder/block_device/level1/mbr.py#L115
[11] https://github.com/openstack/diskimage-builder/blob/906a3f4a57b574d7809531a5d9e9c9f814fd9eab/diskimage_builder/block_device/level1/partitioning.py#L64;L75
[12] ?
[13] https://github.com/openstack/diskimage-builder/blob/906a3f4a57b574d7809531a5d9e9c9f814fd9eab/diskimage_builder/block_device/level1/partitioning.py#L61,L62
[14] https://github.com/openstack/diskimage-builder/blame/906a3f4a57b574d7809531a5d9e9c9f814fd9eab/doc/source/user_guide/building_an_image.rst#L281,L294
[15] https://github.com/openstack/diskimage-builder/blame/906a3f4a57b574d7809531a5d9e9c9f814fd9eab/doc/source/user_guide/building_an_image.rst#L70,L74

[Andreas Scheuring (andreas-scheuring)]

Current patch: https://review.openstack.org/#/c/504588/

[Andreas Scheuring (andreas-scheuring)]

Ups, this is the right one: https://review.openstack.org/#/c/443548

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/443548
Committed: https://git.openstack.org/cgit/openstack/diskimage-builder/commit/?id=271dc36f33e1a2e8774167cd592639fbf98597c0
Submitter: Zuul
Branch: master

commit 271dc36f33e1a2e8774167cd592639fbf98597c0
Author: Zhiguo Deng <email address hidden>
Date: Wed May 10 23:09:42 2017 -0400

    Add zipl element as s390x architecture bootloader

    s390x architecture uses zipl as bootloader. When used in combination
    with the vm element it replaces the existing bootloader element.
    It's mandatory for s390x vm images.

    Use cases
    ---------

    * Allow users to create s390x images that run on nova with s390x
      libvirt/kvm backend
    * Building nodepool images for s390x third party CI

    Supported Distros
    -----------------
    The following listing shows all Distros that officially support
    s390x and how those Distros are supported in DIB with this patch.

    * SLES - not supported (SLES is not supported in DIB)
    * RHEL - not suppoprted (RHEL is not supported as KVM guest on s390x,
                             therefore there's no rhel7 qcow image for s390x available
                             like it is for other archictectures)
    * Ubuntu - supported

    Ubuntu images can for example be built using the following commands:

      $ disk-image-create ubuntu-minimal zipl vm
      $ disk-image-create ubuntu-minimal zipl
      $ disk-image-create ubuntu zipl vm

    Testing
    -------

    Cross architecture building of s390x images is not supported so far.

    The plan is to set up a ThirdParty CI that builds the image for s390x and
    provides the logs.

    Co-Authored-By: Andreas Scheuring <email address hidden>
    Co-Authored-By: Holger Smolinsky <email address hidden>
    Co-Authored-By: Zhiguo Deng <email address hidden>
    Co-Authored-By: Arne Recknagel <email address hidden>

    Closes-Bug: #1730641

    Change-Id: I576e7edda68da12e97c60af38f457915efe7b934