remount of multilower moved pivoted-root overlayfs root, results in I/O errors on some modified files

[Dimitri John Ledkov]

== SRU Justification Disco, Eoan, Focal ==

Multiple squashfs filesystems with overlayfs cause file corruption issues
when modifying zero sized files

== Fix ==

The current fix is pending in https://github.com/amir73il/linux/commit/b2d4f0ea5af42e16e154254de99da064f3ac551a

== Test case ==

With an Ubuntu ISO on the cdrom drive, use:

#!/bin/bash -x
mkdir -p /cdrom
mount -t iso9660 -o ro,noatime /dev/sr0 /cdrom
sleep 1
mkdir -p /cow
mount -t tmpfs -o 'rw,noatime,mode=755' tmpfs /cow
sleep 1
mkdir -p /cow/upper
mkdir -p /cow/work
modprobe -q -b overlay
sleep 1
modprobe -q -b loop
sleep 1
dev=$(losetup -f)
mkdir -p /filesystem.squashfs
losetup $dev /cdrom/casper/filesystem.squashfs
mount -t squashfs -o ro,noatime $dev /filesystem.squashfs
sleep 1

dev=$(losetup -f)
mkdir -p /installer.squashfs
losetup $dev /cdrom/casper/installer.squashfs
mount -t squashfs -o ro,noatime $dev /installer.squashfs
sleep 1

mkdir -p /root-tmp
mount -t overlay -o 'upperdir=/cow/upper,lowerdir=/installer.squashfs:/filesystem.squashfs,workdir=/cow/work' /cow /root-tmp

FILE=/root-tmp/etc/.pwd.lock

echo foo > $FILE
cat $FILE
sync
#
# dropping caches or remounting causes the bug
#
echo 3 > /proc/sys/vm/drop_caches
cat $FILE

Without the fix the cat of the file will produce an error. With the the cat will work correctly.

== Regression Potential ==

There is an unhandled corner case:
    - two filesystems, A and B, both have null uuid
    - upper layer is on A
    - lower layer 1 is also on A
    - lower layer 2 is on B

However, since this is an issue without the fix and will be addressed later with subsequent fixes once they are OK with upstream I think the risk is minimal considering nobody is complaining about these corner cases with the current broken overlayfs squashfs layering.

-----------------------

1) Download focal subiquity pending image, or eoan release image
2) boot, and press ESC and edit boot command line (F6 in bios, e in UEFI)
3) After --- insert the following options

   break=top debug init=/bin/bash

4) Continue boot (Enter in BIOS, ctrl+x in UEFI)
5) in the initramfs execute:

    rm /scripts/casper-bottom/25adduser
    exit

6) you will be dropped into pivoted root filesystem, before systemd is execed as pid one
7) /run/initramfs/ will contain a debug log, showing how everything was mounted. Ie. cdrom mounted, squashfs losetup from there, then multilower overlay setup from them, moved to /root, and then pivot-root to /root done to finally end up as /. Underlying layers are moved into /cow for your convenience.

8) At this point modifying zero-byte length files, that exist in the lowest layer, but not the middle one, in certain ways, will results in them to be corrupted, after / is remounted.

9) Corruption examples

(On both focal & eoan)

cat /etc/.pwd.lock
systemd-sysusers
cat /etc/.pwd.lock
mount -o remount /
cat /etc/.pwd.lock
overlayfs: invalid origin (etc/.pwd.lock, ftype=8000, origin ftype=4000)
cat: /etc/.pwd.lock: Input/output error

(Only on eoan)

cat /etc/machine-id
systemd-machine-id-setup
cat /etc/machine-id
mount -o remount /
cat /etc/machine-id
overlayfs: invalid origin (etc/machine-id, ftype=8000, origin ftype=4000)
cat: /etc/machine-id: Input/output error

Lots of things break once machine-id and .pwd.lock are corrupted. I.e. unable to dhcp, connect to dbus, add/remove/change users or groups, etc.

We were unable to recreate the issue outside of booting things with casper. Ie. statically on a regular host machine without pivot-root. But hopefully booting to a quite state with nothing running is sufficient to reproduce this.

Instead of booting with `bebroken init=/bin/bash` you can boot with `bebroken systemd.mask=systemd-remount-fs.service` this will complete the boot, with /etc/machine-id & .pwd.lock modified, meaning that remount of / will cause IO errors on those files.

Currently, we are shipping two hacks in casper's 25adduser script to "rm" the offending files, and create them again on the upper rw layer. They then survive remount without i/o errors. However, we'd rather not ship those hacks, and have kernel overlay fixed to work correctly with multi-lower-dir and not corrupt files upon remounting /.

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1824407

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Terry Rudd (terrykrudd)]

@dimitri are you still having this issue? There isn't much to go on in this bug submission yet.

[Launchpad Janitor (janitor)]

[Expired for linux (Ubuntu) because there has been no activity for 60 days.]

[Dimitri John Ledkov (xnox)]

Yes this is still a problem affecting Ubuntu Desktop and Server (new one) ISOs.

Errors are visible on boot, during initrd, and after installer started.

And they make the boot messages ugly.

I do not know if there is a bug in vfs (we do overlay, sometimes multilowerdir, on top of squashfs) or if our squashfs are buggy and don't have things that kernel is expecting.

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1824407

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Dimitri John Ledkov (xnox)]

And need casper in focal to migrate to support the "bebroken" keyword

[Colin Ian King (colin-king)]

Hi Dimitri,

while debugging this I found the following in setup_unionfs() in scripts/casper:

    # move the first mount; no head in busybox-initramfs
    for d in $(mount -t squashfs | cut -d\ -f 3); do
        mkdir -p "${rootmnt}/rofs"
        if [ "${UNIONFS}" = unionfs-fuse ]; then
            mount -o bind "${d}" "${rootmnt}/rofs"
        else
            mount -o move "${d}" "${rootmnt}/rofs"
        fi
        break
    done

and looking at the debug /run/initramfs/initramfs.debug log for the above stanza I see:

+ cut '-d ' -f 3
+ mount -t squashfs
+ mkdir -p /root/rofs
+ '[' overlay '=' unionfs-fuse ]
+ mount -o move /filesystem.squashfs /root/rofs
+ break

however, when I cannot reproduce this mount -o move operation by hand as I get the mount error:

mount: /root/rofs: /filesystem.squashfs is not a block device.

It appears to me that the scripts/casper mount seems to silently ignore this failure. Should the mount be a bind mount instead?

[Colin Ian King (colin-king)]

BTW, I can generate the mount move failure with the cut down script as follows (which follows the same mount patterns as the casper script)

#!/bin/bash -x
mkdir -p /cdrom
mount -t iso9660 -o ro,noatime /dev/sr0 /cdrom
sleep 1
mkdir -p /cow
mount -t tmpfs -o 'rw,noatime,mode=755' tmpfs /cow
sleep 1
mkdir -p /cow/upper
mkdir -p /cow/work
modprobe -q -b overlay
sleep 1
modprobe -q -b loop
sleep 1
dev=$(losetup -f)
mkdir -p /filesystem.squashfs
losetup $dev /cdrom/casper/filesystem.squashfs
mount -t squashfs -o ro,noatime $dev /filesystem.squashfs
sleep 1

dev=$(losetup -f)
mkdir -p /installer.squashfs
losetup $dev /cdrom/casper/installer.squashfs
mount -t squashfs -o ro,noatime $dev /installer.squashfs
sleep 1

mount -t overlay -o 'upperdir=/cow/upper,lowerdir=/installer.squashfs:/filesystem.squashfs,workdir=/cow/work' /cow /root

mkdir -p /root/rofs
mount -o move /filesystem.squashfs /root/rofs

[Colin Ian King (colin-king)]

OK, now managed to get a reproducer script to kick this bug even outside the early install context. Seems like we can force this bug by either remounting OR sync'ing and dropping caches.

Attached is the reproducer script.

Run as root, we hit the error:

cat: /root-tmp/etc/.pwd.lock: Input/output error

dmesg:
[ 42.415432] overlayfs: invalid origin (etc/.pwd.lock, ftype=8000, origin ftype=4000).

[Colin Ian King (colin-king)]

And if we change use a different file: /root-tmp/var/log/ubuntu-advantage.log we get the following error too:

[ 24.531406] SQUASHFS error: squashfs_read_data failed to read block 0xffff89c066e0b540
[ 24.531444] SQUASHFS error: Unable to read metadata cache entry [ffff89c066e0b540]

[Colin Ian King (colin-king)]

Replaced read-only squashfs with read-only ext4 partitions and can't reproduce the error.

[Colin Ian King (colin-king)]

Replaced one of the two squashfs with read-only ext4 partitions and can't reproduce the error. Seems that we need 2 stacked squashfs file systems.

[Colin Ian King (colin-king)]

4.15: ovl_get_origin_fh detects zero sized files on lower paths and treats these a special zero sized "copied up but origin unknown" magic.

[ 25.442916] ovl_check_origin: etc/.pwd.lock 2
[ 25.442918] ovl_get_origin_fh: 104 etc/.pwd.lock
[ 25.442919] ovl_get_origin_fh: 107 res=0
[ 25.442920] ovl_get_origin_fh: 117 res == 0, return NULL
[ 25.442921] ovl_get_origin: 179 fh = (null) 1
[ 25.442922] ovl_get_origin_fh: 104 etc/.pwd.lock
[ 25.442922] ovl_get_origin_fh: 107 res=0
[ 25.442923] ovl_get_origin_fh: 117 res == 0, return NULL
[ 25.442923] ovl_get_origin: 179 fh = (null) 1

5.3: the lower is / and hence is a directory hence the S_IFDIR origin return.

[ 33.320630] ovl_get_fh: 112 dentry: etc/.pwd.lock name: trusted.overlay.origin
[ 33.320632] ovl_get_fh: 115: res = 29
[ 33.320634] ovl_get_fh: 152: return fh = 000000006e71855c
[ 33.320634] ovl_check_origin: 413 fh = 000000006e71855c
[ 33.320635] ovl_check_origin_fh: 354 upperdentry = etc/.pwd.lock
[ 33.320635] ovl_decode_real_fh: 174
[ 33.320769] ovl_decode_real_fh: 211 return dentry (OK)
[ 33.320769] ovl_check_origin_fh: 360, i=0, origin = /
[ 33.320770] ovl_check_origin_fh: level=0 upper: etc/.pwd.lock 100600, lower: / 40755
[ 33.320770] ovl_check_origin_fh: 380 goto invalid
[ 33.320771] overlayfs: invalid origin (etc/.pwd.lock, ftype=8000, origin ftype=4000).
[ 33.320773] ovl_check_origin: 422 err = -5
[ 33.320774] ovl_check_origin: 429, return -5

[Colin Ian King (colin-king)]

Comparing the previous debug with 2 squashfs overlayfs lowers with the *same* data on ext4 as the 2 overlayfs lowers we have:

[ 56.257691] repro-nosquashf (1038): drop_caches: 3
[ 56.265075] ovl_get_fh: 112 dentry: etc/.pwd.lock name: trusted.overlay.origin
[ 56.265077] ovl_get_fh: 115: res = 29
[ 56.265079] ovl_get_fh: 152: return fh = 00000000b56cf4e7
[ 56.265079] ovl_check_origin: 413 fh = 00000000b56cf4e7
[ 56.265080] ovl_check_origin_fh: 354 upperdentry = etc/.pwd.lock
[ 56.265081] ovl_decode_real_fh: 174
[ 56.265081] ovl_decode_real_fh: 181 uuid not equal, return NULL
[ 56.265082] ovl_check_origin_fh: 363, i=0, origin = NULL
[ 56.265082] ovl_decode_real_fh: 174
[ 56.266162] ovl_decode_real_fh: 211 return dentry (OK)
[ 56.266163] ovl_check_origin_fh: 360, i=1, origin = /
[ 56.266164] ovl_check_origin_fh: level=1 upper: etc/.pwd.lock 100600, lower: / 100600
[ 56.266166] ovl_check_origin_fh: 395 return 0
[ 56.266166] ovl_check_origin: 422 err = 0
[ 56.266167] ovl_check_origin: 439, return 0
  (
So this works fine, note that the squashfs lower / is 40755 (S_IFDIR | S_IRWXU | S_IRGRP | S_IXGRP | S_IROTH | S_IXOTH) where as the ext3 lower / is 100600 (S_IFREG | S_IRUSR | S_IWUSR)

[Colin Ian King (colin-king)]

Adding a uuid into the superblock on squashfs seems to resolve the issue. Since squashfs does not have UUID support, my hack below generates one based on some squashfs superblock metadata that provides a good enough UUID for our purposes.

diff --git a/fs/squashfs/super.c b/fs/squashfs/super.c
index effa638d6d85..cfb34a75feb6 100644
--- a/fs/squashfs/super.c
+++ b/fs/squashfs/super.c
@@ -186,6 +186,12 @@ static int squashfs_fill_super(struct super_block *sb, void *data, int silent)
        sb->s_flags |= SB_RDONLY;
        sb->s_op = &squashfs_super_ops;

+ memcpy(&sb->s_uuid.b[0], &sblk->inodes, 4);
+ memcpy(&sb->s_uuid.b[4], &sblk->mkfs_time, 4);
+ memcpy(&sb->s_uuid.b[8], &sblk->fragments, 4);
+ memcpy(&sb->s_uuid.b[12], &sblk->compression, 2);
+ memcpy(&sb->s_uuid.b[14], &sblk->block_log, 2);
+
        err = -ENOMEM;

        msblk->block_cache = squashfs_cache_init("metadata",

[Colin Ian King (colin-king)]

The concern I have is for other file systems that also don't populate the UUID - this seems to be a general problem for overlayfs. Perhaps a UUID can be autogenerated based on the superblock rather than file system specific UUID magic if the UUID is zero.

[Dimitri John Ledkov (xnox)]

I think kernel already generates some faked up UUIDs, i.e. i think there is something for like FAT and DOS partition table, either fs uuid, or like partition table uuid, partition uuid, partition type uuid, some such.

They are "shorter" than normal, and less unique, but at least something.

Should we request squashfs userspace tooling and kernel to start having a UUID in the squashfs?

We do use squashfs a lot, all the snaps and all the installer media.

[Dimitri John Ledkov (xnox)]

There is mkfs-time of the filesystem superblock, which could be converted to a timebased UUID.

[Dimitri John Ledkov (xnox)]

blkid

/dev/nvme0n1p1: LABEL="ESP" UUID="B407-D3BC" TYPE="vfat" PARTLABEL="EFI system partition" PARTUUID="fb8a1f92-fbd6-4ce2-ade4-0a11bdbaf74b"

ah, but that UUID is actually fat serial number / volume-id

Sets the volume ID of the newly created filesystem; VOLUME-ID is a 32-bit hexadecimal number (for example, 2e24ec82). The default is a number which depends on the filesystem creation time.

It would be nice if squashfs had UUID support, and like blkid /dev/loop* printed them, and kernel used them, etc.

[Colin Ian King (colin-king)]

I was thinking of a more generalized overlayfs solution that detects if file systems don't initialize the superblock uuid and overlayfs improvises by generating the internal overlayfs uuid, something like:

diff --git a/fs/overlayfs/copy_up.c b/fs/overlayfs/copy_up.c
index 698d112bdb17..da3faaf68d69 100644
--- a/fs/overlayfs/copy_up.c
+++ b/fs/overlayfs/copy_up.c
@@ -248,6 +248,7 @@ struct ovl_fh *ovl_encode_real_fh(struct dentry *real, bool is_upper)
        void *buf;
        int buflen = MAX_HANDLE_SZ;
        uuid_t *uuid = &real->d_sb->s_uuid;
+ static const uuid_t z_uuid;

        buf = kmalloc(buflen, GFP_KERNEL);
        if (!buf)
@@ -289,7 +290,22 @@ struct ovl_fh *ovl_encode_real_fh(struct dentry *real, bool is_upper)
        if (is_upper)
                fh->flags |= OVL_FH_FLAG_PATH_UPPER;
        fh->len = fh_len;
- fh->uuid = *uuid;
+
+ if (uuid_equal(uuid, &z_uuid)) {
+ struct super_block *sb = real->d_sb;
+ u16 hash;
+
+ pr_warn("ovl_encode_real_fh: ZERO UUID, generating one from superblock\n");
+
+ memcpy(&fh->uuid.b[0], &sb->s_magic, 8);
+ memcpy(&fh->uuid.b[8], &sb->s_dev, 6);
+ hash = ((long)sb ^ (long)sb->s_fs_info) >> 12;
+ memcpy(&fh->uuid.b[14], &hash, 2);
+ } else {
+ fh->uuid = *uuid;
+ }
+

[Colin Ian King (colin-king)]

Just love the way launchpad mangles pasted code.

[Adam Conrad (adconrad)]

While there may be arguments for synthesizing UUIDs in various fs drivers, or creating them in the first place in fs-creation tools, I agree that if overlayfs has a hard dependency on UUIDs for uniquely identifying layers, it needs to fill in the gaps where the previous tools/drivers failed to provide.

So, yeah, I think Colin's on the right track here with making overlayfs itself fix its own use-case, which will cover any future filesystem/tool that otherwise fails to UUID itself.

My only review of the above is that you might want to mix a monotonic or random seed in with your superblock when generating your fake UUIDs, in case someone does something like stacking multiple empties on top of each other, or some other madness where you effectively have the same superblock more than once at the RO level, but need to identify them as different.

[Dimitri John Ledkov (xnox)]

At least it doesn't mangle email notifications! Such that in email code is
readable.

On Wed, 6 Nov 2019, 18:50 Colin Ian King, <email address hidden>
wrote:

> Just love the way launchpad mangles pasted code.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1824407
>
> Title:
> remount of multilower moved pivoted-root overlayfs root, results in
> I/O errors on some modified files
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1824407/+subscriptions
>

[Dimitri John Ledkov (xnox)]

I thought all lowerdirs must be read-only, and the first one in the stack
will win. Even if there are same ones down the stack it shouldn't matter.
But I guess needs testing.

I wouldn't want to use random, as that will eat entropy on early boot of
Ubuntu Core systems.

Do we need a separate upstream issue to create uuid in the squashfs
Superblock?

On Wed, 6 Nov 2019, 21:21 Adam Conrad, <[e-mail]> wrote:

> While there may be arguments for synthesizing UUIDs in various fs
> drivers, or creating them in the first place in fs-creation tools, I
> agree that if overlayfs has a hard dependency on UUIDs for uniquely
> identifying layers, it needs to fill in the gaps where the previous
> tools/drivers failed to provide.
>
> So, yeah, I think Colin's on the right track here with making overlayfs
> itself fix its own use-case, which will cover any future filesystem/tool
> that otherwise fails to UUID itself.
>
> My only review of the above is that you might want to mix a monotonic or
> random seed in with your superblock when generating your fake UUIDs, in
> case someone does something like stacking multiple empties on top of
> each other, or some other madness where you effectively have the same
> superblock more than once at the RO level, but need to identify them as
> different.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1824407
>
> Title:
> remount of multilower moved pivoted-root overlayfs root, results in
> I/O errors on some modified files
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1824407/+subscriptions
>

[Colin Ian King (colin-king)]

When I'm more awake tomorrow I'll send a patch upstream as a suggested fix and see if we can get a good solution on the UUIDs worked out.

[Adam Conrad (adconrad)]

I didn't mean actually random, and I think clock-based would be entirely fine, but my paranoia about stacking two things with identical superblocks might be just paranoia too. I dunno.

[Colin Ian King (colin-king)]

been iterating on a fix with upstream: https://lkml.org/lkml/2019/11/7/317

[Dimitri John Ledkov (xnox)]

pr_warn sounds awkward though

Ie. a user will not be able to do anything about it, if that's the only thing they have to mount and boot. Our installer images boot quiet, but this will make it be printed on every boot of Ubuntu installer media.

Things that a user didn't cause, and cannot control should not be at warning level.

[Colin Ian King (colin-king)]

pr_warn can be removed with a sauce patch, so no worries with that.

[Dimitri John Ledkov (xnox)]

v4 patch is at https://lkml.org/lkml/2019/11/13/799

[Colin Ian King (colin-king)]

I'm doing some testing right now on the current upstream fix, hopefully will SRU this by EOD.

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-disco' to 'verification-done-disco'. If the problem still exists, change the tag 'verification-needed-disco' to 'verification-failed-disco'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Colin Ian King (colin-king)]

Verified for disco:

Run reproducer script with old kernel: 5.0.0-37-generic, results:

cat /root-tmp/etc/.pwd.lock
cat: /root-tmp/etc/.pwd.lock: Input/output error

Run with -proposed kernel: 5.0.0-38-generic

cat /root-tmp/etc/.pwd.lock
foo

Marking as verification-done-disco

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-eoan' to 'verification-done-eoan'. If the problem still exists, change the tag 'verification-needed-eoan' to 'verification-failed-eoan'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Colin Ian King (colin-king)]

Hrm, I can't see the fix in the Ubuntu-5.3.0-24.26 kernel, so I think comment #34 a premature SRU test request. As it stands, I tested Ubuntu-5.3.0-24.26 and the issue still exists, and looking at the source the fix isn't present so that correlates with my test observations.

[Dimitri John Ledkov (xnox)]

The comment in #34 is about 5.3.0-25.27 which is in eoan-proposed.

Also ubuntu-kernel-bot must post these message when the kernel is copied into -proposed pocket, and it should state the release series and version, just like all other SRU comments do.

[Colin Ian King (colin-king)]

Tested with 5.3.0-25-generic #27-Ubuntu with the regression test and it now works fine. Marking bug as verification-done for eoan

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 5.3.0-26.28

---------------
linux (5.3.0-26.28) eoan; urgency=medium

  * eoan/linux: 5.3.0-26.28 -proposed tracker (LP: #1856807)

  * nvidia-435 is in eoan, linux-restricted-modules only builds against 430,
    ubiquity gives me the self-signed modules experience instead of using the
    Canonical-signed modules (LP: #1856407)
    - Add nvidia-435 dkms build

linux (5.3.0-25.27) eoan; urgency=medium

  * eoan/linux: 5.3.0-25.27 -proposed tracker (LP: #1854762)

  * CVE-2019-14901
    - SAUCE: mwifiex: Fix heap overflow in mmwifiex_process_tdls_action_frame()

  * CVE-2019-14896 // CVE-2019-14897
    - SAUCE: libertas: Fix two buffer overflows at parsing bss descriptor

  * CVE-2019-14895
    - SAUCE: mwifiex: fix possible heap overflow in mwifiex_process_country_ie()

  * [CML] New device id's for CMP-H (LP: #1846335)
    - mmc: sdhci-pci: Add another Id for Intel CML
    - i2c: i801: Add support for Intel Comet Lake PCH-H
    - mtd: spi-nor: intel-spi: Add support for Intel Comet Lake-H SPI serial flash
    - mfd: intel-lpss: Add Intel Comet Lake PCH-H PCI IDs

  * i915: Display flickers (monitor loses signal briefly) during "flickerfree"
    boot, while showing the BIOS logo on a black background (LP: #1836858)
    - [Config] FRAMEBUFFER_CONSOLE_DEFERRED_TAKEOVER=y

  * Please add patch fixing RK818 ID detection (LP: #1853192)
    - SAUCE: mfd: rk808: Fix RK818 ID template

  * Kernel build log filled with "/bin/bash: line 5: warning: command
    substitution: ignored null byte in input" (LP: #1853843)
    - [Debian] Fix warnings when checking for modules signatures

  * Lenovo dock MAC Address pass through doesn't work in Ubuntu (LP: #1827961)
    - r8152: Add macpassthru support for ThinkPad Thunderbolt 3 Dock Gen 2

  * Dell XPS 13 9350/9360 headphone audio hiss (LP: #1654448) // [XPS 13 9360,
    Realtek ALC3246, Black Headphone Out, Front] High noise floor (LP: #1845810)
    - ALSA: hda/realtek: Reduce the Headphone static noise on XPS 9350/9360

  * no HDMI video output since GDM greeter after linux-oem-osp1 version
    5.0.0-1026 (LP: #1852386)
    - drm/i915: Add new CNL PCH ID seen on a CML platform
    - SAUCE: drm/i915: Fix detection for a CMP-V PCH

  * [broadwell-rt286, playback] Since Linux 5.2rc2 audio playback no longer
    works on Dell Venue 11 Pro 7140 (LP: #1846539)
    - [Config] Drop snd-sof-intel-bdw build
    - SAUCE: ASoC: SOF: Intel: Broadwell: clarify mutual exclusion with legacy
      driver

  * [CML-S62] Need enable turbostat patch support for Comet lake- S 6+2
    (LP: #1847451)
    - SAUCE: tools/power turbostat: Add Cometlake support

  * External microphone can't work on some dell machines with the codec alc256
    or alc236 (LP: #1853791)
    - SAUCE: ALSA: hda/realtek - Move some alc256 pintbls to fallback table
    - SAUCE: ALSA: hda/realtek - Move some alc236 pintbls to fallback table

  * Memory leak in net/xfrm/xfrm_state.c - 8 pages per ipsec connection
    (LP: #1853197)
    - xfrm: Fix memleak on xfrm state destroy

  * CVE-2019-18660: patches for Ubuntu (LP: #1853142) // CVE-2019-18660
    - powerpc/64s: support nospectre_v2 cmdline option
    - powerp...

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 5.0.0-38.41

---------------
linux (5.0.0-38.41) disco; urgency=medium

  * disco/linux: 5.0.0-38.41 -proposed tracker (LP: #1854788)

  * [Regression] Failed to boot disco kernel built from master-next (kernel
    kernel NULL pointer dereference) (LP: #1853981)
    - SAUCE: blk-mq: Fix blk_mq_make_request for mq devices

  * CVE-2019-14901
    - SAUCE: mwifiex: Fix heap overflow in mmwifiex_process_tdls_action_frame()

  * CVE-2019-14896 // CVE-2019-14897
    - SAUCE: libertas: Fix two buffer overflows at parsing bss descriptor

  * CVE-2019-14895
    - SAUCE: mwifiex: fix possible heap overflow in mwifiex_process_country_ie()

  * [CML] New device id's for CMP-H (LP: #1846335)
    - mmc: sdhci-pci: Add another Id for Intel CML
    - i2c: i801: Add support for Intel Comet Lake PCH-H
    - mtd: spi-nor: intel-spi: Add support for Intel Comet Lake-H SPI serial flash
    - mfd: intel-lpss: Add Intel Comet Lake PCH-H PCI IDs

  * Please add patch fixing RK818 ID detection (LP: #1853192)
    - SAUCE: mfd: rk808: Fix RK818 ID template

  * [SRU][B/OEM-B/OEM-OSP1/D] Enable new Elan touchpads which are not in current
    whitelist (LP: #1853246)
    - Input: elan_i2c - export the device id whitelist
    - HID: quirks: Refactor ELAN 400 and 401 handling

  * Lenovo dock MAC Address pass through doesn't work in Ubuntu (LP: #1827961)
    - r8152: Add macpassthru support for ThinkPad Thunderbolt 3 Dock Gen 2

  * [CML-S62] Need enable turbostat patch support for Comet lake- S 6+2
    (LP: #1847451)
    - SAUCE: tools/power turbostat: Add Cometlake support

  * External microphone can't work on some dell machines with the codec alc256
    or alc236 (LP: #1853791)
    - SAUCE: ALSA: hda/realtek - Move some alc256 pintbls to fallback table
    - SAUCE: ALSA: hda/realtek - Move some alc236 pintbls to fallback table

  * Memory leak in net/xfrm/xfrm_state.c - 8 pages per ipsec connection
    (LP: #1853197)
    - xfrm: Fix memleak on xfrm state destroy

  * CVE-2019-18660: patches for Ubuntu (LP: #1853142) // CVE-2019-18660
    - powerpc/64s: support nospectre_v2 cmdline option
    - powerpc/book3s64: Fix link stack flush on context switch
    - KVM: PPC: Book3S HV: Flush link stack on guest exit to host kernel

  * Raydium Touchscreen on ThinkPad L390 does not work (LP: #1849721)
    - HID: i2c-hid: fix no irq after reset on raydium 3118

  * Make Goodix I2C touchpads work (LP: #1853842)
    - HID: i2c-hid: Remove runtime power management
    - HID: i2c-hid: Send power-on command after reset

  * Touchpad doesn't work on Dell Inspiron 7000 2-in-1 (LP: #1851901)
    - Revert "UBUNTU: SAUCE: mfd: intel-lpss: add quirk for Dell XPS 13 7390
      2-in-1"
    - lib: devres: add a helper function for ioremap_uc
    - mfd: intel-lpss: Use devm_ioremap_uc for MMIO

  * CVE-2019-19055
    - nl80211: fix memory leak in nl80211_get_ftm_responder_stats

  * [CML-S62] Need enable intel_rapl patch support for Comet lake- S 6+2
    (LP: #1847454)
    - powercap/intel_rapl: add support for CometLake Mobile
    - powercap/intel_rapl: add support for Cometlake desktop

  * [CML-S62] Need enable intel_pmc_core driver patch for Comet l...

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 5.4.0-9.12

---------------
linux (5.4.0-9.12) focal; urgency=medium

  * alsa/hda/realtek: the line-out jack doens't work on a dell AIO
    (LP: #1855999)
    - SAUCE: ALSA: hda/realtek - Line-out jack doesn't work on a Dell AIO

  * scsi: hisi_sas: Check sas_port before using it (LP: #1855952)
    - scsi: hisi_sas: Check sas_port before using it

  * CVE-2019-19078
    - ath10k: fix memory leak

  * cifs: DFS Caching feature causing problems traversing multi-tier DFS setups
    (LP: #1854887)
    - cifs: Fix retrieval of DFS referrals in cifs_mount()

  * Support DPCD aux brightness control (LP: #1856134)
    - SAUCE: drm/i915: Fix eDP DPCD aux max backlight calculations
    - SAUCE: drm/i915: Assume 100% brightness when not in DPCD control mode
    - SAUCE: drm/i915: Fix DPCD register order in intel_dp_aux_enable_backlight()
    - SAUCE: drm/i915: Auto detect DPCD backlight support by default
    - SAUCE: drm/i915: Force DPCD backlight mode on X1 Extreme 2nd Gen 4K AMOLED
      panel
    - USUNTU: SAUCE: drm/i915: Force DPCD backlight mode on Dell Precision 4K sku

  * The system cannot resume from S3 if user unplugs the TB16 during suspend
    state (LP: #1849269)
    - PCI: pciehp: Do not disable interrupt twice on suspend
    - PCI: pciehp: Prevent deadlock on disconnect

  * change kconfig of the soundwire bus driver from y to m (LP: #1855685)
    - [Config]: SOUNDWIRE=m

  * alsa/sof: change to use hda hdmi codec driver to make hdmi audio on the
    docking station work (LP: #1855666)
    - ALSA: hda/hdmi - implement mst_no_extra_pcms flag
    - ASoC: hdac_hda: add support for HDMI/DP as a HDA codec
    - ASoC: Intel: skl-hda-dsp-generic: use snd-hda-codec-hdmi
    - ASoC: Intel: skl-hda-dsp-generic: fix include guard name
    - ASoC: SOF: Intel: add support for snd-hda-codec-hdmi
    - ASoC: Intel: bxt-da7219-max98357a: common hdmi codec support
    - ASoC: Intel: glk_rt5682_max98357a: common hdmi codec support
    - ASoC: intel: sof_rt5682: common hdmi codec support
    - ASoC: Intel: bxt_rt298: common hdmi codec support
    - ASoC: SOF: enable sync_write in hdac_bus
    - [config]: SND_SOC_SOF_HDA_COMMON_HDMI_CODEC=y

  * Fix unusable USB hub on Dell TB16 after S3 (LP: #1855312)
    - SAUCE: USB: core: Make port power cycle a seperate helper function
    - SAUCE: USB: core: Attempt power cycle port when it's in eSS.Disabled state

  * Focal update: v5.4.3 upstream stable release (LP: #1856583)
    - rsi: release skb if rsi_prepare_beacon fails
    - arm64: tegra: Fix 'active-low' warning for Jetson TX1 regulator
    - arm64: tegra: Fix 'active-low' warning for Jetson Xavier regulator
    - perf scripts python: exported-sql-viewer.py: Fix use of TRUE with SQLite
    - sparc64: implement ioremap_uc
    - lp: fix sparc64 LPSETTIMEOUT ioctl
    - time: Zero the upper 32-bits in __kernel_timespec on 32-bit
    - mailbox: tegra: Fix superfluous IRQ error message
    - staging/octeon: Use stubs for MIPS && !CAVIUM_OCTEON_SOC
    - usb: gadget: u_serial: add missing port entry locking
    - serial: 8250-mtk: Use platform_get_irq_optional() for optional irq
    - tty: serial: fsl_lpuart: use the sg ...

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux-hwe - 5.3.0-26.28~18.04.1

---------------
linux-hwe (5.3.0-26.28~18.04.1) bionic; urgency=medium

  * bionic/linux-hwe: 5.3.0-26.28~18.04.1 -proposed tracker (LP: #1856849)

  [ Ubuntu: 5.3.0-26.28 ]

  * eoan/linux: 5.3.0-26.28 -proposed tracker (LP: #1856807)
  * nvidia-435 is in eoan, linux-restricted-modules only builds against 430,
    ubiquity gives me the self-signed modules experience instead of using the
    Canonical-signed modules (LP: #1856407)
    - Add nvidia-435 dkms build

linux-hwe (5.3.0-25.27~18.04.2) bionic; urgency=medium

  * bionic/linux-hwe: 5.3.0-25.27~18.04.2 -proposed tracker (LP: #1856210)

  * Packaging resync (LP: #1786013)
    - [Packaging] HWE kernel is not signed on Bionic

linux-hwe (5.3.0-25.27~18.04.1) bionic; urgency=medium

  * bionic/linux-hwe: 5.3.0-25.27~18.04.1 -proposed tracker (LP: #1855309)

  [ Ubuntu: 5.3.0-25.27 ]

  * eoan/linux: 5.3.0-25.27 -proposed tracker (LP: #1854762)
  * CVE-2019-14901
    - SAUCE: mwifiex: Fix heap overflow in mmwifiex_process_tdls_action_frame()
  * CVE-2019-14896 // CVE-2019-14897
    - SAUCE: libertas: Fix two buffer overflows at parsing bss descriptor
  * CVE-2019-14895
    - SAUCE: mwifiex: fix possible heap overflow in mwifiex_process_country_ie()
  * [CML] New device id's for CMP-H (LP: #1846335)
    - mmc: sdhci-pci: Add another Id for Intel CML
    - i2c: i801: Add support for Intel Comet Lake PCH-H
    - mtd: spi-nor: intel-spi: Add support for Intel Comet Lake-H SPI serial flash
    - mfd: intel-lpss: Add Intel Comet Lake PCH-H PCI IDs
  * i915: Display flickers (monitor loses signal briefly) during "flickerfree"
    boot, while showing the BIOS logo on a black background (LP: #1836858)
    - [Config] FRAMEBUFFER_CONSOLE_DEFERRED_TAKEOVER=y
  * Please add patch fixing RK818 ID detection (LP: #1853192)
    - SAUCE: mfd: rk808: Fix RK818 ID template
  * Kernel build log filled with "/bin/bash: line 5: warning: command
    substitution: ignored null byte in input" (LP: #1853843)
    - [Debian] Fix warnings when checking for modules signatures
  * Lenovo dock MAC Address pass through doesn't work in Ubuntu (LP: #1827961)
    - r8152: Add macpassthru support for ThinkPad Thunderbolt 3 Dock Gen 2
  * Dell XPS 13 9350/9360 headphone audio hiss (LP: #1654448) // [XPS 13 9360,
    Realtek ALC3246, Black Headphone Out, Front] High noise floor (LP: #1845810)
    - ALSA: hda/realtek: Reduce the Headphone static noise on XPS 9350/9360
  * no HDMI video output since GDM greeter after linux-oem-osp1 version
    5.0.0-1026 (LP: #1852386)
    - drm/i915: Add new CNL PCH ID seen on a CML platform
    - SAUCE: drm/i915: Fix detection for a CMP-V PCH
  * [broadwell-rt286, playback] Since Linux 5.2rc2 audio playback no longer
    works on Dell Venue 11 Pro 7140 (LP: #1846539)
    - [Config] Drop snd-sof-intel-bdw build
    - SAUCE: ASoC: SOF: Intel: Broadwell: clarify mutual exclusion with legacy
      driver
  * [CML-S62] Need enable turbostat patch support for Comet lake- S 6+2
    (LP: #1847451)
    - SAUCE: tools/power turbostat: Add Cometlake support
  * External microphone can't work on some dell machines with the codec alc256
    o...