systemd 229-4ubuntu21.22 ADT test failure with linux 4.4.0-156.183 (storage)

[Kleber Sacilotto de Souza]

Testing failed on:
    i386: https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-xenial/xenial/i386/s/systemd/20190719_145555_3528c@/log.gz

The storage testcase is failing on i386. Running the testcase manually, it hangs at:

systemd-229/debian/tests$ sudo ./storage
test_luks_by_devname (__main__.CryptsetupTest)
LUKS device by plain device name, empty ... Please enter passphrase for disk testcrypt1!

Looking at the storage testcase, on function start_password_agent(), it expects the 'disk scsi_debug' string on the systemd ask-password file, however the contains:

Message=Please enter passphrase for disk testcrypt1!

That doesn't seems to happen on other architectures. On amd64, the testcase output is:

systemd-229/debian/tests$ sudo ./storage
test_luks_by_devname (__main__.CryptsetupTest)
LUKS device by plain device name, empty ... Please enter passphrase for disk scsi_debug (testcrypt1)!
ok
[...]

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1837235

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Kleber Sacilotto de Souza (kleber-souza)]

systemd relies on some udev properties to craft the message that will be displayed by the cryptsetup service when a password is required.

For this testcase, the scsi_debug module is loaded and the device created by it is used for the tests.

With xenial kernel 4.4.0-154-generic, udevadm returns for the scsi_debug block device:

$ udevadm info /dev/sdb
P: /devices/pseudo_0/adapter0/host2/target2:0:0/2:0:0:0/block/sdb
N: sdb
S: disk/by-id/scsi-35333333000001770
S: disk/by-id/wwn-0x5333333000001770
E: DEVLINKS=/dev/disk/by-id/scsi-35333333000001770 /dev/disk/by-id/wwn-0x5333333000001770
E: DEVNAME=/dev/sdb
E: DEVPATH=/devices/pseudo_0/adapter0/host2/target2:0:0/2:0:0:0/block/sdb
E: DEVTYPE=disk
E: ID_BUS=scsi
E: ID_MODEL=scsi_debug
E: ID_MODEL_ENC=scsi_debug\x20\x20\x20\x20\x20\x20
E: ID_REVISION=0184
E: ID_SCSI=1
E: ID_SCSI_SERIAL=6000
E: ID_SERIAL=35333333000001770
E: ID_SERIAL_SHORT=5333333000001770
E: ID_TARGET_PORT=300
E: ID_TYPE=disk
E: ID_VENDOR=Linux
E: ID_VENDOR_ENC=Linux\x20\x20\x20
E: ID_WWN=0x5333333000001770
E: ID_WWN_WITH_EXTENSION=0x5333333000001770
E: MAJOR=8
E: MINOR=16
E: SUBSYSTEM=block
E: TAGS=:systemd:
E: USEC_INITIALIZED=121942203

However, with kernel 4.4.0-156-generic the output is:

$ udevadm info /dev/sdb
P: /devices/pseudo_0/adapter0/host2/target2:0:0/2:0:0:0/block/sdb
N: sdb
E: DEVNAME=/dev/sdb
E: DEVPATH=/devices/pseudo_0/adapter0/host2/target2:0:0/2:0:0:0/block/sdb
E: DEVTYPE=disk
E: MAJOR=8
E: MINOR=16
E: SUBSYSTEM=block
E: TAGS=:systemd:
E: USEC_INITIALIZED=36737967

systemd expects one of the following properties to be set:
                  "ID_PART_ENTRY_NAME\0"
                  "DM_NAME\0"
                  "ID_MODEL_FROM_DATABASE\0"
                  "ID_MODEL\0";

But none of them is present with the newer kernel.

[Kleber Sacilotto de Souza (kleber-souza)]

The missing properties seems to be caused by this error:

$ sudo /lib/systemd/systemd-udevd --debug
[...]
IMPORT 'scsi_id --export --whitelisted -d /dev/sdb' /lib/udev/rules.d/60-persistent-storage.rules:44
starting 'scsi_id --export --whitelisted -d /dev/sdb'
Process 'scsi_id --export --whitelisted -d /dev/sdb' failed with exit code 1.

The above command was run while running 'modprobe scsi_debug' in another terminal.

[Kleber Sacilotto de Souza (kleber-souza)]

This is a kernel bug, introduced by one of these patches applied to xenial:

commit d823dfc03906fd278ae6f48219c4ed53faee6f9d
Author: Al Viro <email address hidden>
Date: Thu Jun 27 20:16:00 2019 +0200

    bio_copy_from_iter(): get rid of copying iov_iter

commit aceb27a7de5fc65a3fc70ccac344f33099e8a4dc
Author: Keith Busch <email address hidden>
Date: Thu Jun 27 20:16:00 2019 +0200

    block: Clear kernel memory before copying to user

commit a713ff5b13e8fd9e331fd84232806316aa9e9b2a
Author: Keith Busch <email address hidden>
Date: Thu Jun 27 20:16:00 2019 +0200

    block/bio: Do not zero user pages

[Kleber Sacilotto de Souza (kleber-souza)]

Those commits were applied for LP: #1834499.

[Kleber Sacilotto de Souza (kleber-souza)]

How to easily reproduce the issue:

$ sudo modprobe scsi_debug
$ sudo /lib/udev/scsi_id --export --whitelisted -v -d /dev/sdb
8:16: ioctl failed: Invalid argument
8:16: Unable to get INQUIRY vpd 0 page 0x0.

[Dan Streetman (ddstreet)]

i thought for sure it would be a systemd bug and/or testcase bug :)

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-xenial' to 'verification-done-xenial'. If the problem still exists, change the tag 'verification-needed-xenial' to 'verification-failed-xenial'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Kleber Sacilotto de Souza (kleber-souza)]

ADT systemd storage testcase is now successful:

https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-xenial/xenial/i386/s/systemd/20190723_172332_bb2de@/log.gz

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 4.4.0-157.185

---------------
linux (4.4.0-157.185) xenial; urgency=medium

  * linux: 4.4.0-157.185 -proposed tracker (LP: #1837476)

  * systemd 229-4ubuntu21.22 ADT test failure with linux 4.4.0-156.183 (storage)
    (LP: #1837235)
    - Revert "block/bio: Do not zero user pages"
    - Revert "block: Clear kernel memory before copying to user"
    - Revert "bio_copy_from_iter(): get rid of copying iov_iter"

linux (4.4.0-156.183) xenial; urgency=medium

  * linux: 4.4.0-156.183 -proposed tracker (LP: #1836880)

  * BCM43602 802.11ac Wireless regression - PCI ID 14e4:43ba (LP: #1836801)
    - brcmfmac: add eth_type_trans back for PCIe full dongle

linux (4.4.0-155.182) xenial; urgency=medium

  * linux: 4.4.0-155.182 -proposed tracker (LP: #1834918)

  * Geneve tunnels don't work when ipv6 is disabled (LP: #1794232)
    - geneve: correctly handle ipv6.disable module parameter

  * Kernel modules generated incorrectly when system is localized to a non-
    English language (LP: #1828084)
    - scripts: override locale from environment when running recordmcount.pl

  * Handle overflow in proc_get_long of sysctl (LP: #1833935)
    - sysctl: handle overflow in proc_get_long

  * Xenial update: 4.4.181 upstream stable release (LP: #1832661)
    - x86/speculation/mds: Revert CPU buffer clear on double fault exit
    - x86/speculation/mds: Improve CPU buffer clear documentation
    - ARM: exynos: Fix a leaked reference by adding missing of_node_put
    - crypto: vmx - fix copy-paste error in CTR mode
    - crypto: crct10dif-generic - fix use via crypto_shash_digest()
    - crypto: x86/crct10dif-pcl - fix use via crypto_shash_digest()
    - ALSA: usb-audio: Fix a memory leak bug
    - ALSA: hda/hdmi - Consider eld_valid when reporting jack event
    - ALSA: hda/realtek - EAPD turn on later
    - ASoC: max98090: Fix restore of DAPM Muxes
    - ASoC: RT5677-SPI: Disable 16Bit SPI Transfers
    - mm/mincore.c: make mincore() more conservative
    - ocfs2: fix ocfs2 read inode data panic in ocfs2_iget
    - mfd: da9063: Fix OTP control register names to match datasheets for
      DA9063/63L
    - tty/vt: fix write/write race in ioctl(KDSKBSENT) handler
    - ext4: actually request zeroing of inode table after grow
    - ext4: fix ext4_show_options for file systems w/o journal
    - Btrfs: do not start a transaction at iterate_extent_inodes()
    - bcache: fix a race between cache register and cacheset unregister
    - bcache: never set KEY_PTRS of journal key to 0 in journal_reclaim()
    - ipmi:ssif: compare block number correctly for multi-part return messages
    - crypto: gcm - Fix error return code in crypto_gcm_create_common()
    - crypto: gcm - fix incompatibility between "gcm" and "gcm_base"
    - crypto: chacha20poly1305 - set cra_name correctly
    - crypto: salsa20 - don't access already-freed walk.iv
    - crypto: arm/aes-neonbs - don't access already-freed walk.iv
    - writeback: synchronize sync(2) against cgroup writeback membership switches
    - fs/writeback.c: use rcu_barrier() to wait for inflight wb switches going
      into workqueue when umount
    - ALSA: hda/realtek - Fix for Lenovo B...