bcache: Issues with large IO wait in bch_mca_scan() when shrinker is enabled

Bug #1898786 reported by Benjamin Allot on 2020-10-06
24
This bug affects 2 people
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Undecided
Unassigned
Bionic
Medium
Matthew Ruffell
Focal
Medium
Matthew Ruffell

Bug Description

BugLink: https://bugs.launchpad.net/bugs/1898786

[Impact]

Systems that utilise bcache can experience extremely high IO wait times when under constant IO pressure. The IO wait times seem to stay at a consistent 1 second, and never drop as long as the bcache shrinker is enabled.

If you disable the shrinker, then IO wait drops significantly to normal levels.

We did some perf analysis, and it seems we spend a huge amount of time in bch_mca_scan(), likely waiting for the mutex "&c->bucket_lock".

Looking at the recent commits in Bionic, we found the following commit merged in upstream 5.1-rc1 and backported to 4.15.0-87-generic through upstream stable:

commit 9fcc34b1a6dd4b8e5337e2b6ef45e428897eca6b
Author: Coly Li <email address hidden>
Date: Wed Nov 13 16:03:24 2019 +0800
Subject: bcache: at least try to shrink 1 node in bch_mca_scan()
Link: https://github.com/torvalds/linux/commit/9fcc34b1a6dd4b8e5337e2b6ef45e428897eca6b

It mentions in the description that:

> If sc->nr_to_scan is smaller than c->btree_pages, after the above
> calculation, variable 'nr' will be 0 and nothing will be shrunk. It is
> frequeently observed that only 1 or 2 is set to sc->nr_to_scan and make
> nr to be zero. Then bch_mca_scan() will do nothing more then acquiring
> and releasing mutex c->bucket_lock.

This seems to be what is going on here, but the above commit only addresses when nr is 0.

From what I can see, the problems we are experiencing are when nr is 1 or 2, and again, we just waste time in bch_mca_scan() waiting on c->bucket_lock, only to release c->bucket_lock since the shrinker loop never executes since there is no work to do.

[Fix]

The following commits fix the problem, and all landed in 5.6-rc1:

commit 125d98edd11464c8e0ec9eaaba7d682d0f832686
Author: Coly Li <email address hidden>
Date: Fri Jan 24 01:01:40 2020 +0800
Subject: bcache: remove member accessed from struct btree
Link: https://github.com/torvalds/linux/commit/125d98edd11464c8e0ec9eaaba7d682d0f832686

commit d5c9c470b01177e4d90cdbf178b8c7f37f5b8795
Author: Coly Li <email address hidden>
Date: Fri Jan 24 01:01:41 2020 +0800
Subject: bcache: reap c->btree_cache_freeable from the tail in bch_mca_scan()
Link: https://github.com/torvalds/linux/commit/d5c9c470b01177e4d90cdbf178b8c7f37f5b8795

commit e3de04469a49ee09c89e80bf821508df458ccee6
Author: Coly Li <email address hidden>
Date: Fri Jan 24 01:01:42 2020 +0800
Subject: bcache: reap from tail of c->btree_cache in bch_mca_scan()
Link: https://github.com/torvalds/linux/commit/e3de04469a49ee09c89e80bf821508df458ccee6

The first commit is a dependency of the other two. The first commit removes a "recently accessed" marker, used to indicate if a particular cache has been used recently, and if it has been, not consider it for cache eviction. The commit mentions that under heavy IO, all caches will end up being recently accessed, and nothing will ever be shrunk.

The second commit changes a previous design decision of skipping the first 3 caches to shrink, since it is a common case to call bch_mca_scan() with nr being 1, or 2, just like 0 was common in the very first commit I mentioned. This time, if we land on 1 or 2, the loop exits and nothing happens, and we waste time waiting on locks, just like the very first commit I mentioned. The fix is to try shrink caches from the tail of the list, and not the beginning.

The third commit fixes a minor issue where we don't want to re-arrange the linked list c->btree_cache, which is what the second commit ended up doing, and instead, just shrink the cache at the end of the linked list, and not change the order.

One minor backport / context adjustment was required in the first commit for Bionic, and the rest are all clean cherry picks to Bionic and Focal.

[Testcase]

This is kind of hard to test, since the problem shows up in production environments that are under constant IO pressure, with many different items entering and leaving the cache.

The Launchpad git server is currently suffering this issue, and has been sitting at a constant IO wait of 1 second / slightly over 1 second which was causing slow response times, which was leading to build failures when git clones ended up timing out.

We installed a test kernel, which is available in the following PPA:

https://launchpad.net/~mruffell/+archive/ubuntu/sf294907-test

Once the test kernel had been installed, IO wait times with the shrinker enabled dropped to normal levels, and the git server became responsive again. We have been monitoring the performance of the git server and watching IO wait times in grafana over the past week, and everything is looking good, and indicate that these patches solve the issue.

[Regression Potential]

If a regression were to occur, it would only affect users of bcache who have the shrinker enabled, which I believe is the default setting in bcache deployments.

In that case, users could disable the shrinker as a workaround until a fix is available. You can disable the shrinker with:
"sudo bash -c "echo 1 > /sys/fs/bcache/<uuid>/internal/btree_shrinker_disabled""

The patches remove the "recently accessed" marker, which changes the behaviour slightly when it comes to deciding what to evict from the cache. Some workloads that aren't under IO pressure may see a difference in what items are evicted from the cache, which may slightly change bcache's performance profile.

Since the patches prune the cache from the tail of the cache linked list, we will likely end up pruning more nodes than previously, since it was common to land on the first 3 caches, which were hard coded to be ignored and not pruned, which may also slightly change the bcache's performance profile.

CVE References

Benjamin Allot (ballot) wrote :

This change was made by a bot.

Changed in linux (Ubuntu):
status: New → Confirmed
Haw Loeung (hloeung) on 2020-10-06
description: updated

@Benjamin Allot is this a performance regression from the prior kernel version? If so, can you confirm the last "good" version? Perhaps we can do a bisect

Khaled El Mously (kmously) wrote :

correction: from *a prior kernel version

Benjamin Allot (ballot) wrote :

Actually we cannot be sure no. The server didn't have any metrics prior to few days ago and the issue was already there.

It's worth nothing that few servers have this bcache configuration, because the cache mode is configured as writethrough and the load is pretty significant.

So no last "good" version.

Actually, we have various IO wait issue on another platform (but running xenial-hwe kernel) and we suspected bcache already. Mentioning it to show that bcache behavior seems to be related to some disk performance since quite some time.

Benjamin Allot (ballot) wrote :

Here is a trace of the bcache event related lock in the kernel obtained with some bpfcc-tools.

klockstat-bpfcc -c bch_ -i 5 -s 3

The trace has been run in parallel with the following command line

echo "Shrinker disabled: $(date)"; sleep 60; echo "Enabling shrinker: $(date)"; echo 0 | sudo tee /sys/block/bcache0/bcache/cache/internal/btree_shrinker_disabled ; sleep 60; echo "Disabling shrinker: $(date)"; echo 1 | sudo tee /sys/block/bcache0/bcache/cache/internal/btree_shrinker_disabled; sleep 60; echo "End of test: $(date)"

The log are here : https://pastebin.canonical.com/p/jVKdbV3RrK/

Hi Benjamin,

Thanks for the detailed report.

Does this system show signs of memory pressure?

bch_mca_scan() is part of bcache's memory shrinker, and thus should be called when the system is trying to release memory from its several caches.

Also, the bucket lock usage is widespread in bcache (from a quick grep; and more used on writes, I'd imagine) thus if bch_mca_scan() is waiting a lot on it, ie, showing lock contention, it would seem like the IO load is indeed significant, as you mentioned.

Do you know the IO load profile, or could reproduce the issue with the fio tool?

If we can reproduce this, there _might_ be some heuristics to consider to use non-blocking trylock instead of blocking lock, and bail out/don't wait if the lock is taken and it doesn't seem worth it. (but that would be a development item, not a proper "bug" fix :)

cheers,
Mauricio

Benjamin Allot (ballot) on 2020-10-08
description: updated
description: updated
Benjamin Allot (ballot) wrote :

Hello Mauricio,

That was also one of the conclusion we reached yesterday.

I followed a wrong lead yesterday, after getting another more detailed output with klockstat.

I'll update the description of the bug at the end, adding the bits I found.

The system is not really having memory pressure but to confirm this, we resized the memory allocated to a VM on the server from 64GB to 44GB and looked into the shrinker behavior.
It didn't change a thing, and the IO wait started to rise again.

As for the IO load, I'm not sure, I would probably need to do a blktrace and register it (never did that, but I read that you can then passes the result to fio to reproduce a load)

Hi Benjamin,

It's interesting that the memory shrinkers are being invoked without memory pressure.
I'll try to understand it better, maybe I'm missing something.

I guess the before/after memory sizes for the test are swapped?
i.e., to alleviate potential memory pressure the VM's memory had
been _increased_ from 44G to 64G? and not the opposite.

And maybe it's not necessary to trace/reproduce the exact load with
fio; perhaps one of fio's default/generic modes ([rand]r/w/rw) w/
a variation of queue depth might hit a similar pattern on iowait.
(If that helps w/ one less step to do :)

cheers,
Mauricio

Benjamin Allot (ballot) on 2020-10-08
description: updated
Benjamin Allot (ballot) on 2020-10-08
description: updated
Benjamin Allot (ballot) wrote :

Hi Mauricio,

The tests in [5][6] and [7] have been done with a 44GB memory VM.
This VM usually has 64GB of allocated memory.
The goal was to verify that the load of the whole btree in RAM was not prevented (like in mca-reap -> down_write_trylock).

All this kernel stuff is rather new to me, so I may be following a wrong lead.

description: updated
Matthew Ruffell (mruffell) wrote :
Download full text (4.0 KiB)

Hi Benjamin,

I think Dan is onto something.

The following commit was merged in upstream 5.5-rc1, but was backported to 4.15.0-87-generic through upstream stable:

commit 9fcc34b1a6dd4b8e5337e2b6ef45e428897eca6b
Author: Coly Li <email address hidden>
Date: Wed Nov 13 16:03:24 2019 +0800
Subject: bcache: at least try to shrink 1 node in bch_mca_scan()
Link: https://github.com/torvalds/linux/commit/9fcc34b1a6dd4b8e5337e2b6ef45e428897eca6b

It mentions in the description that:

> If sc->nr_to_scan is smaller than c->btree_pages, after the above
> calculation, variable 'nr' will be 0 and nothing will be shrunk. It is
> frequeently observed that only 1 or 2 is set to sc->nr_to_scan and make
> nr to be zero. Then bch_mca_scan() will do nothing more then acquiring
> and releasing mutex c->bucket_lock.

Spending a lot of idle time in bch_mca_scan() waiting on locks seems to be a key symptom from the logs you have gathered.

I went and had a look through the commit history, and I think these three patches might solve the problem:

commit 125d98edd11464c8e0ec9eaaba7d682d0f832686
Author: Coly Li <email address hidden>
Date: Fri Jan 24 01:01:40 2020 +0800
Subject: bcache: remove member accessed from struct btree
Link: https://github.com/torvalds/linux/commit/125d98edd11464c8e0ec9eaaba7d682d0f832686

commit d5c9c470b01177e4d90cdbf178b8c7f37f5b8795
Author: Coly Li <email address hidden>
Date: Fri Jan 24 01:01:41 2020 +0800
Subject: bcache: reap c->btree_cache_freeable from the tail in bch_mca_scan()
Link: https://github.com/torvalds/linux/commit/d5c9c470b01177e4d90cdbf178b8c7f37f5b8795

commit e3de04469a49ee09c89e80bf821508df458ccee6
Author: Coly Li <email address hidden>
Date: Fri Jan 24 01:01:42 2020 +0800
Subject: bcache: reap from tail of c->btree_cache in bch_mca_scan()
Link: Link: https://github.com/torvalds/linux/commit/e3de04469a49ee09c89e80bf821508df458ccee6

The first commit is a dependency of the other two. The first commit removes a "recently accessed" marker, used to indicate if a particular cache has been used recently, and if it has been, not consider it for cache eviction. The commit mentions that under heavy IO, all caches will end up being recently accessed, and nothing will ever be shrunk.

The second commit changes a previous design decision of skipping the first 3 caches to shrink, since it is a common case to call bch_mca_scan() with nr being 1, or 2, just like 0 was common in the very first commit I mentioned. This time, if we land on 1 or 2, the loop exits and nothing happens, and we waste time waiting on locks, just like the very first commit I mentioned. The fix is to try shrink caches from the tail of the list, and not the beginning.

The third commit fixes a minor issue where we don't want to re-arrange the linked list, which is what the second commit ended up doing, and instead, just shrink the cache at the end of the linked list, and not change the order.

All three commits are clean cherry picks to 4.15.0-118-generic, and I have got a test kernel building for you in:

https://launchpad.net/~mruffell/+archive/ubuntu/sf294907-test

It should be ready in a few hours time. Make sure the build has finished before trying to install.

Please note ...

Read more...

Benjamin Allot (ballot) wrote :

Thanks Matthew, we'll try this kernel tomorrow.

I tested it on an openstack instance, the only downside is that it uninstalls the official 4.15.0-118-generic one.

Regards,

Benjamin Allot (ballot) wrote :

The testing of the new kernel looks very promising. We don't observe any of the latency/IOwait we had before even with the btree_shrinker enabled.

We'll give it a week probably, but having a backport of those patches would be fantastic for sure.

Matthew Ruffell (mruffell) wrote :

Hi Benjamin,

Great to hear that the early results look promising.

Yes, let's give it a week, so we can ensure that these patches do their job under the high loads the git server faces each day over the week.

If things still look good early next week, I will go ahead and prepare the patches for SRU to the Ubuntu kernels.

Let me know how the test kernel is going next week.

Thanks,
Matthew

summary: - Issue with bcache bch_mca_scan causing huge IO wait
+ bcache: Issues with large IO wait in bch_mca_scan() when shrinker is
+ enabled
Changed in linux (Ubuntu):
status: Confirmed → Fix Released
Changed in linux (Ubuntu Bionic):
status: New → In Progress
Changed in linux (Ubuntu Focal):
status: New → In Progress
Changed in linux (Ubuntu Bionic):
importance: Undecided → Medium
Changed in linux (Ubuntu Focal):
importance: Undecided → Medium
Changed in linux (Ubuntu Bionic):
assignee: nobody → Matthew Ruffell (mruffell)
Changed in linux (Ubuntu Focal):
assignee: nobody → Matthew Ruffell (mruffell)
description: updated
description: updated
tags: added: focal
tags: added: sts
description: updated
Ian (ian-may) on 2020-10-22
Changed in linux (Ubuntu Focal):
status: In Progress → Fix Committed
Ian (ian-may) on 2020-10-22
Changed in linux (Ubuntu Bionic):
status: In Progress → Fix Committed

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-bionic' to 'verification-done-bionic'. If the problem still exists, change the tag 'verification-needed-bionic' to 'verification-failed-bionic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-bionic
tags: added: verification-needed-focal

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

Matthew Ruffell (mruffell) wrote :

Hi Benjamin,

The kernel team have built the next kernel update, and they have placed it in -proposed for verification.

The versions are 4.15.0-125-generic for Bionic, and 5.4.0-55-generic for Focal.

Can you please schedule a maintenance window for the Launchpad git server, to install the new kernel in -proposed, and reboot into it, so we can verify that it fixes the problem.

Instructions to install (On a Bionic system):
Enable -proposed by running the following command to make a new sources.list.d entry:
1) cat << EOF | sudo tee /etc/apt/sources.list.d/ubuntu-bionic-proposed.list
# Enable Ubuntu proposed archive
deb http://archive.ubuntu.com/ubuntu/ bionic-proposed main
EOF
2) sudo apt update
3) sudo apt install linux-image-4.15.0-125-generic linux-modules-4.15.0-125-generic \
linux-modules-extra-4.15.0-125-generic linux-headers-4.15.0-125-generic linux-headers-4.15.0-125
4) sudo reboot
5) uname -rv
4.15.0-125-generic #128-Ubuntu SMP Mon Nov 9 20:51:00 UTC 2020
6) sudo rm /etc/apt/sources.list.d/ubuntu-bionic-proposed.list
7) sudo apt update

If you get a different uname, you may need to adjust your grub configuration to boot into the correct kernel. Also, since this is a production machine, make sure you remove the -proposed software source once you have installed the kernel.

Let me know how this kernel performs, and if everything seems fine after a week we will mark the Launchpad bug as verified. The timeline for release to -updates is still set for the 30th of November, give or take a few days if any CVEs turn up.

I believe this kernel should be live-patchable, although this may not be the case if the kernel is respun before release. Hopefully you will only have to schedule the maintenance window just the once.

Thanks,
Matthew

Matthew Ruffell (mruffell) wrote :

Performing verification for Focal.

Since Benjamin hasn't responded, I will try and verify the best I can.

I made a instance on AWS. I used a c5d.large instance type, and added 8gb extra EBS storage.

I installed the latest kernel from -updates to get a performance baseline. kernel is 5.4.0-54-generic.

I made a bcache disk with the following.

Note, the 8gb disk was used as the cache disk, and the 50gb disk the backing disk. Having the cache small is to try force cache evictions often, and possibly try trigger the bug.

$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
nvme2n1 259:0 0 46.6G 0 disk
nvme1n1 259:1 0 8G 0 disk
nvme0n1 259:2 0 8G 0 disk
└─nvme0n1p1 259:3 0 8G 0 part /

$ sudo apt install bcache-tools
$ sudo dd if=/dev/zero if=/dev/nvme1n1 bs=512 count=8
$ sudo dd if=/dev/zero if=/dev/nvme2n1 bs=512 count=8
$ sudo wipefs -a /dev/nvme1n1
$ sudo wipefs -a /dev/nvme2n1
$ sudo make-bcache -C /dev/nvme1n1 -B /dev/nvme2n1
UUID: 3f28ca5d-856b-42e9-bbb7-54cae12b5538
Set UUID: 756747bc-f27c-44ca-a9b9-dbd132722838
version: 0
nbuckets: 16384
block_size: 1
bucket_size: 1024
nr_in_set: 1
nr_this_dev: 0
first_bucket: 1
UUID: cc3e36fd-3694-4c50-aeac-0b79d2faab4a
Set UUID: 756747bc-f27c-44ca-a9b9-dbd132722838
version: 1
block_size: 1
data_offset: 16
$ sudo mkfs.ext4 /dev/bcache0
$ sudo mkdir /media/bcache
$ sudo mount /dev/bcache0 /media/bcache
$ echo "/dev/bcache0 /media/bcache ext4 rw 0 0" | sudo tee -a /etc/fstab

From there, I installed fio to do some benchmarks, and to try apply some IO pressure to the cache.

$ sudo apt install fio

I used the following fio jobfile:

https://paste.ubuntu.com/p/RNBmXdy3zG/

It is based on the ssd test in: https://github.com/axboe/fio/blob/master/examples/ssd-test.fio

Running the fio job gives us the following output:

https://paste.ubuntu.com/p/HrWGNDJPfv/

Now we have the baseline, I enabled -proposed and installed 5.4.0-55-generic and rebooted.

I started the fio job again, and got the following output:

# uname -rv
5.4.0-55-generic #61-Ubuntu SMP Mon Nov 9 20:49:56 UTC 2020

https://paste.ubuntu.com/p/pDVXnspmvs/

If you compare the two outputs, there really isn't much difference in latencies / read / write speeds. The bcache patches don't seem to cause any large impacts.

I managed to set up a bcache disk, and did some IO stress tests. Things seem to be okay.

Since we had positive test results on the test kernel on the Launchpad git server, and the above shows we don't appear to have any regressions, I will mark this bug as verified for Focal.

tags: added: verification-done-focal
removed: verification-needed-focal
Matthew Ruffell (mruffell) wrote :

Performing verification for Bionic.

Since Benjamin hasn't responded, I will try and verify the best I can.

I made a instance on AWS. I used a c5d.large instance type, and added 8gb extra EBS storage.

I installed the latest kernel from -updates to get a performance baseline. kernel is 4.15.0-124-generic.

I made a bcache disk with the following.

Note, the 8gb disk was used as the cache disk, and the 50gb disk the backing disk. Having the cache small is to try force cache evictions often, and possibly try trigger the bug.

$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
nvme1n1 259:0 0 46.6G 0 disk
nvme0n1 259:1 0 8G 0 disk
nvme2n1 259:2 0 8G 0 disk
└─nvme2n1p1 259:3 0 8G 0 part /

$ sudo apt install bcache-tools
$ sudo dd if=/dev/zero of=/dev/nvme0n1 bs=512 count=8
$ sudo dd if=/dev/zero of=/dev/nvme1n1 bs=512 count=8
$ sudo wipefs -a /dev/nvme0n1
$ sudo wipefs -a /dev/nvme1n1
$ sudo make-bcache -C /dev/nvme0n1 -B /dev/nvme1n1
UUID: 3f28ca5d-856b-42e9-bbb7-54cae12b5538
Set UUID: 756747bc-f27c-44ca-a9b9-dbd132722838
version: 0
nbuckets: 16384
block_size: 1
bucket_size: 1024
nr_in_set: 1
nr_this_dev: 0
first_bucket: 1
UUID: cc3e36fd-3694-4c50-aeac-0b79d2faab4a
Set UUID: 756747bc-f27c-44ca-a9b9-dbd132722838
version: 1
block_size: 1
data_offset: 16
$ sudo mkfs.ext4 /dev/bcache0
$ sudo mkdir /media/bcache
$ sudo mount /dev/bcache0 /media/bcache
$ echo "/dev/bcache0 /media/bcache ext4 rw 0 0" | sudo tee -a /etc/fstab

From there, I installed fio to do some benchmarks, and to try apply some IO pressure to the cache.

$ sudo apt install fio

I used the following fio jobfile:

https://paste.ubuntu.com/p/RNBmXdy3zG/

It is based on the ssd test in: https://github.com/axboe/fio/blob/master/examples/ssd-test.fio

Running the fio job gives us the following output:

https://paste.ubuntu.com/p/ghkQcyT2sv/

Now we have the baseline, I enabled -proposed and installed 4.15.0-125-generic and rebooted.

I started the fio job again, and got the following output:

# uname -rv
4.15.0-125-generic #128-Ubuntu SMP Mon Nov 9 20:51:00 UTC 2020

https://paste.ubuntu.com/p/DSTnKvXMGZ/

If you compare the two outputs, there really isn't much difference in latencies / read / write speeds. The bcache patches don't seem to cause any large impacts.

I managed to set up a bcache disk, and did some IO stress tests. Things seem to be okay.

Since we had positive test results on the test kernel on the Launchpad git server, and the above shows we don't appear to have any regressions, I will mark this bug as verified for Bionic.

tags: added: verification-done-bionic
removed: verification-needed-bionic
Benjamin Allot (ballot) wrote :

Hello Matthew, sorry for the lack of response.

I'll check with Launchpad people if we can justify a reboot of the server soon and will keep you posted !

Regards,

Matthew Ruffell (mruffell) wrote :

Hi Benjamin,

No worries about being busy.

Now, the kernel is scheduled to be released early next week, around the 30th of November. I think at this stage it is best to wait it out and install the kernel once it reaches -updates.

That way you will have a fixed kernel that is supported by livepatch, and you don't have to justify a reboot twice.

I did some regression testing in my comments above, and everything looks okay. These patches also worked great in your test kernel. We have done the best can to verify the kernel in the time given, so don't worry about testing at this stage.

I'll let you know once the kernel has reached -updates, likely Monday or Tuesday next week.

Thanks,
Matthew

Benjamin Allot (ballot) wrote :

Ack !

I'll check with Launchpad team then, I think they would probably prefer to wait for the -updates indeed.

Thanks again for your work dans Dan's.

Cheers,

Launchpad Janitor (janitor) wrote :
Download full text (78.9 KiB)

This bug was fixed in the package linux - 5.4.0-56.62

---------------
linux (5.4.0-56.62) focal; urgency=medium

  * focal/linux: 5.4.0-56.62 -proposed tracker (LP: #1905300)

  * CVE-2020-4788
    - selftests/powerpc: rfi_flush: disable entry flush if present
    - powerpc/64s: flush L1D on kernel entry
    - powerpc/64s: flush L1D after user accesses
    - selftests/powerpc: entry flush test

linux (5.4.0-55.61) focal; urgency=medium

  * focal/linux: 5.4.0-55.61 -proposed tracker (LP: #1903175)

  * Update kernel packaging to support forward porting kernels (LP: #1902957)
    - [Debian] Update for leader included in BACKPORT_SUFFIX

  * Avoid double newline when running insertchanges (LP: #1903293)
    - [Packaging] insertchanges: avoid double newline

  * EFI: Fails when BootCurrent entry does not exist (LP: #1899993)
    - efivarfs: Replace invalid slashes with exclamation marks in dentries.

  * CVE-2020-14351
    - perf/core: Fix race in the perf_mmap_close() function

  * raid10: Block discard is very slow, causing severe delays for mkfs and
    fstrim operations (LP: #1896578)
    - md: add md_submit_discard_bio() for submitting discard bio
    - md/raid10: extend r10bio devs to raid disks
    - md/raid10: pull codes that wait for blocked dev into one function
    - md/raid10: improve raid10 discard request
    - md/raid10: improve discard request for far layout
    - dm raid: fix discard limits for raid1 and raid10
    - dm raid: remove unnecessary discard limits for raid10

  * Bionic: btrfs: kernel BUG at /build/linux-
    eTBZpZ/linux-4.15.0/fs/btrfs/ctree.c:3233! (LP: #1902254)
    - btrfs: drop unnecessary offset_in_page in extent buffer helpers
    - btrfs: extent_io: do extra check for extent buffer read write functions
    - btrfs: extent-tree: kill BUG_ON() in __btrfs_free_extent()
    - btrfs: extent-tree: kill the BUG_ON() in insert_inline_extent_backref()
    - btrfs: ctree: check key order before merging tree blocks

  * Ethernet no link lights after reboot (Intel i225-v 2.5G) (LP: #1902578)
    - igc: Add PHY power management control

  * Undetected Data corruption in MPI workloads that use VSX for reductions on
    POWER9 DD2.1 systems (LP: #1902694)
    - powerpc: Fix undetected data corruption with P9N DD2.1 VSX CI load emulation
    - selftests/powerpc: Make alignment handler test P9N DD2.1 vector CI load
      workaround

  * [20.04 FEAT] Support/enhancement of NVMe IPL (LP: #1902179)
    - s390: nvme ipl
    - s390: nvme reipl
    - s390/ipl: support NVMe IPL kernel parameters

  * uvcvideo: add mapping for HEVC payloads (LP: #1895803)
    - media: uvcvideo: Add mapping for HEVC payloads

  * Focal update: v5.4.73 upstream stable release (LP: #1902115)
    - ibmveth: Switch order of ibmveth_helper calls.
    - ibmveth: Identify ingress large send packets.
    - ipv4: Restore flowi4_oif update before call to xfrm_lookup_route
    - mlx4: handle non-napi callers to napi_poll
    - net: fec: Fix phy_device lookup for phy_reset_after_clk_enable()
    - net: fec: Fix PHY init after phy_reset_after_clk_enable()
    - net: fix pos incrementment in ipv6_route_seq_next
    - net/smc: fix valid DMBE buffer sizes
    - net...

Changed in linux (Ubuntu Focal):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :
Download full text (33.1 KiB)

This bug was fixed in the package linux - 4.15.0-126.129

---------------
linux (4.15.0-126.129) bionic; urgency=medium

  * bionic/linux: 4.15.0-126.129 -proposed tracker (LP: #1905305)

  * CVE-2020-4788
    - SAUCE: powerpc/64s: Define MASKABLE_RELON_EXCEPTION_PSERIES_OOL
    - SAUCE: powerpc/64s: move some exception handlers out of line
    - powerpc/64s: flush L1D on kernel entry
    - SAUCE: powerpc: Add a framework for user access tracking
    - powerpc: Implement user_access_begin and friends
    - powerpc: Fix __clear_user() with KUAP enabled
    - powerpc/uaccess: Evaluate macro arguments once, before user access is
      allowed
    - powerpc/64s: flush L1D after user accesses

linux (4.15.0-125.128) bionic; urgency=medium

  * bionic/linux: 4.15.0-125.128 -proposed tracker (LP: #1903137)

  * Update kernel packaging to support forward porting kernels (LP: #1902957)
    - [Debian] Update for leader included in BACKPORT_SUFFIX

  * Avoid double newline when running insertchanges (LP: #1903293)
    - [Packaging] insertchanges: avoid double newline

  * EFI: Fails when BootCurrent entry does not exist (LP: #1899993)
    - efivarfs: Replace invalid slashes with exclamation marks in dentries.

  * CVE-2020-14351
    - perf/core: Fix race in the perf_mmap_close() function

  * raid10: Block discard is very slow, causing severe delays for mkfs and
    fstrim operations (LP: #1896578)
    - md: add md_submit_discard_bio() for submitting discard bio
    - md/raid10: extend r10bio devs to raid disks
    - md/raid10: pull codes that wait for blocked dev into one function
    - md/raid10: improve raid10 discard request
    - md/raid10: improve discard request for far layout

  * Bionic: btrfs: kernel BUG at /build/linux-
    eTBZpZ/linux-4.15.0/fs/btrfs/ctree.c:3233! (LP: #1902254)
    - btrfs: use offset_in_page instead of open-coding it
    - btrfs: use BUG() instead of BUG_ON(1)
    - btrfs: drop unnecessary offset_in_page in extent buffer helpers
    - btrfs: extent_io: do extra check for extent buffer read write functions
    - btrfs: extent-tree: kill BUG_ON() in __btrfs_free_extent()
    - btrfs: extent-tree: kill the BUG_ON() in insert_inline_extent_backref()
    - btrfs: ctree: check key order before merging tree blocks

  * Bionic update: upstream stable patchset 2020-11-04 (LP: #1902943)
    - USB: gadget: f_ncm: Fix NDP16 datagram validation
    - gpio: tc35894: fix up tc35894 interrupt configuration
    - vsock/virtio: use RCU to avoid use-after-free on the_virtio_vsock
    - vsock/virtio: stop workers during the .remove()
    - vsock/virtio: add transport parameter to the
      virtio_transport_reset_no_sock()
    - net: virtio_vsock: Enhance connection semantics
    - Input: i8042 - add nopnp quirk for Acer Aspire 5 A515
    - ftrace: Move RCU is watching check after recursion check
    - drm/amdgpu: restore proper ref count in amdgpu_display_crtc_set_config
    - drivers/net/wan/hdlc_fr: Add needed_headroom for PVC devices
    - drm/sun4i: mixer: Extend regmap max_register
    - net: dec: de2104x: Increase receive ring size for Tulip
    - rndis_host: increase sleep time in the query-response loop
    - nvme-core: get/put ctrl ...

Changed in linux (Ubuntu Bionic):
status: Fix Committed → Fix Released
Matthew Ruffell (mruffell) wrote :

Hi Benjamin,

I have good news. The SRU has completed, and the new kernels have now been released to -updates. Their versions are:

Bionic:
4.15.0-126-generic

Focal:
5.4.0-56-generic

You can go ahead and schedule that maintenance window now, to install the latest kernel from -updates. These kernels also have full livepatch support, which is good news for you.

Let me know how the 4.15.0-126-generic kernel goes on the Launchpad git server, since it should perform just as well as the test kernel you are currently running.

Thanks,
Matthew

Matthew Ruffell (mruffell) wrote :

Hi Benjamin,

The respun kernel has now landed in -updates, and is version 4.15.0-128-generic.

Please re-schedule the maintenance window for the Launchpad git server, and re-attempt moving to the fixed kernel.

Thanks,
Matthew

Haw Loeung (hloeung) wrote :

@mruffell, that's for Bionic, but for Focal is that still 5.4.0-56-generic? Or is there a new respun kernel to include these patches?

Matthew Ruffell (mruffell) wrote :

Hi @hloeung, these patches are available in 4.15.0-128-generic, and 5.4.0-58-generic.

They are both re-spins of 4.15.0-126-generic and 5.4.0-56-generic, respectively.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers