Console extremely slow with 3.13 and newer kernels for certified servers with Matrox G200er2 or similar

[Mike Jester]

When I first came across this issue I thought it was the same as bug #1316035 (https://bugs.launchpad.net/ubuntu/+source/xorg-server/+bug/1316035), but this bug seems to be about X and it is reporting xorg-server as the package with the issue. I've seen a report on the certification program already on this, but it was lacking follow-up and suggested a bug report, question 255332 (https://answers.launchpad.net/ubuntu-certification/+question/255332). I've already asked about this on the forums, and received no replies.

I originally encountered this issue due to having a bunch of Dell servers that I was testing with kernel 3.13 on 12.04. The text console is so slow that it is unusable when using vi or doing an operation similar to dpkg -l |grep xorg. Comparing the exact same system with kernel 3.2 or 3.11 loaded and running lsmod, vesafb is listed and performance is excellent. It is not listed on the 3.13 system. The same issue exists on 14.04 with the 3.13 or 3.16 kernel.

Attempting to do modprobe vesa or modprobe vesafb returns modprobe: FATAL: Module vesa not found and modprobe: FATAL: Module vesafb not found respectively.

Attempting to use uvesafb on this hardware results in a black screen. Attempting to use vga16fb results in good performance, but I am locked to 640x480. I tried to use fbset to force higher resolutions, but it either results in a distorted display or tells me there is not enough memory.

I need a 1280x1024 console with performance equal to 12.04 with kernels 3.2-3.11. The server I'm using for testing is a Dell R320.

This appears to be regression as I can duplicate it by simply doing a fresh install of 12.04.1 and not having the issue, while doing a fresh install of 12.04.5, 14.04.1, or 14.04.2 results in a very slow console.
---
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 Mar 20 13:17 seq
 crw-rw---- 1 root audio 116, 33 Mar 20 13:17 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.14.1-0ubuntu3.7
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: [Errno 2] No such file or directory
DistroRelease: Ubuntu 14.04
HibernationDevice: RESUME=UUID=3255aa87-ba68-4b74-bdb0-1112aba7d46e
InstallationDate: Installed on 2015-03-20 (0 days ago)
InstallationMedia: Ubuntu-Server 14.04.1 LTS "Trusty Tahr" - Release amd64 (20140722.3)
IwConfig: Error: [Errno 2] No such file or directory
MachineType: Dell Inc. PowerEdge R320
Package: linux (not installed)
PciMultimedia:

ProcEnviron:
 TERM=linux
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB: 0 VESA VGA
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.13.0-46-generic root=UUID=fda4445f-efe2-4e93-9adb-ecaf6386f8cd ro consoleblank=0
ProcVersionSignature: Ubuntu 3.13.0-46.79-generic 3.13.11-ckt15
RelatedPackageVersions:
 linux-restricted-modules-3.13.0-46-generic N/A
 linux-backports-modules-3.13.0-46-generic N/A
 linux-firmware 1.127.11
RfKill: Error: [Errno 2] No such file or directory
Tags: trusty
Uname: Linux 3.13.0-46-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: root
_MarkForUpload: True
dmi.bios.date: 09/23/2013
dmi.bios.vendor: Dell Inc.
dmi.bios.version: 2.0.21
dmi.board.name: 0R5KP9
dmi.board.vendor: Dell Inc.
dmi.board.version: A09
dmi.chassis.type: 23
dmi.chassis.vendor: Dell Inc.
dmi.modalias: dmi:bvnDellInc.:bvr2.0.21:bd09/23/2013:svnDellInc.:pnPowerEdgeR320:pvr:rvnDellInc.:rn0R5KP9:rvrA09:cvnDellInc.:ct23:cvr:
dmi.product.name: PowerEdge R320
dmi.sys.vendor: Dell Inc.

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1434581

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Mike Jester (mikejester)]

apport information

[Mike Jester (mikejester)]

apport information

[Mike Jester (mikejester)]

apport information

[Mike Jester (mikejester)]

apport information

[Mike Jester (mikejester)]

apport information

[Mike Jester (mikejester)]

apport information

[Mike Jester (mikejester)]

apport information

[Mike Jester (mikejester)]

apport information

[Mike Jester (mikejester)]

apport information

[Mike Jester (mikejester)]

apport information

[Joseph Salisbury (jsalisbury)]

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.0 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

If you are unable to test the mainline kernel, for example it will not boot, please add the tag: 'kernel-unable-to-test-upstream'.
Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.0-rc4-vivid/

[Mike Jester (mikejester)]

I can do that on Monday when I'm back in the office no problem. This is a test machine so I can do just about anything with it. Thanks!

[Mike Jester (mikejester)]

I updated the BIOS to 2.3.3, and I used kernel 4.0-rc5 that was posted this morning. No change. Running dpkg -l is still a slideshow.

I removed the bios-outdated tag, let me know if I need to do another collect.

[Mike Jester (mikejester)]

In an attempt to narrow it down, I downloaded some more kernels from mainline to test. I reloaded the machine with 12.04 and tried 3.11.0-26 (via linux-generic-lts-saucy meta) and 3.12.0 from the mainline repository. 3.12.0 is slow like 3.13 and 3.11.0-26 performs fine as noted in the original report.

The only commit I could find involving vesafb between those was in 3.12-rc1, but I could be missing something.

https://lkml.org/lkml/2013/8/2/787

[Joseph Salisbury (jsalisbury)]

Can you test the following two Upstream kernels:

v3.11 final: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.11-saucy/
v3.12-rc1: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.12-rc1-saucy/

We can bisect between these two versions if 3.11 final is good and v3.12-rc1 had the bug.

[Mike Jester (mikejester)]

Chris, the output is:

2.3.3
07/10/2014

[Mike Jester (mikejester)]

I tried more kernels this morning, and unfortunately my results weren't so good.

All of the v3.12-rc# kernels, v3.11.0 and v3.11.10.15 give me a black screen. The system does boot and function, but I am blind. I'm attaching the dmesg from v3.12.0, v3.12-rc1 and v3.11.0.

The last versions of 3.8 (lts-raring) and 3.11 (lts-saucy) in the official repositories perform well, with no issues.

I'm going to try v3.10-saucy, v3.9-saucy and v3.8-raring as well.

[Mike Jester (mikejester)]

I tested more mainline kernels: 3.2.0 through 3.10.0. The corresponding Ubuntu kernels (ex: lts-quantal 3.5.0-54 vs mainline 3.5.0) work properly. The ones from mainline all black screen.

If I go to / and do find . -name vesafb.ko only the Ubuntu kernels from 3.11 and earlier have the file.

Was this driver added by the Ubuntu kernel team but left out of 3.13 and 3.16?

[Joseph Salisbury (jsalisbury)]

Vesafb was a module in the 3.11 and older kernels. It is now built into the kernel, which is why that file does not exist.

There some vesa specific SAUCE patches in Ubuntu that are not upstream, so that may be why the upstream kernels don't work.

We could bisect between the Ubuntu kernels instead of the mainline kernel. We would first need to identify the last Ubuntu kernel that did not have the bug and the first that did. Can you test the following two kernels:

3.11.0-12.19: https://launchpad.net/ubuntu/+source/linux/3.11.0-12.19/+build/5088396
3.13.0-1.16: https://launchpad.net/ubuntu/+source/linux/3.13.0-1.16/+build/5421709

The 3.11.0-12.19 is that last 3.11 based kernel Trusty used before moving to the 3.12 kernel. The 3.13.0-1.16 kernel is the first 3.13 based kernel that Trusty used.

All of the Trust kernels can be found at:
https://launchpad.net/ubuntu/trusty/+source/linux

[Mike Jester (mikejester)]

3.13.0-1 that you linked does not have the bug.

3.13.0-6.23 does not have it: https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/ppa/+build/5537054

3.13.0-7.24 appears to be a failed build, but maybe one of the commits is related: https://launchpad.net/ubuntu/+source/linux/3.13.0-7.24/+build/5549751

3.13.0-7.25 has the bug: https://launchpad.net/ubuntu/+source/linux/3.13.0-7.25/+build/5551372

3.13.0-7.26 has the bug: https://launchpad.net/ubuntu/+source/linux/3.13.0-7.26/+build/5558945

[Mike Jester (mikejester)]

I built 3.13.0-48.80 with CONFIG_X86_SYSFB=y since it was set to n in 3.13.0-7.24. This resolves the issue. I can upload the built kernel if necessary.

[Stefan Bader (smb)]

I would have to do more testing, but it does not look like the simple framebuffer driver ever got fixed to release the memory resource and so, as soon as we turn on that, we will have the old issue of VMs (using cirrus default gfx) coming up with no graphical console at all.
That said, the simple framebuffer did not exist before 3.13. Reading quickly through the report I think you said you used VESA framebuffer before. That is now built into the kernel (so trying to modprobe does fail). Though maybe not. It would help to find out what framebuffer driver actually was used. Probably that info is already in one of the attached dmesg files. I need to have a look if I have more time. Things to play around would be to look at /etc/default/grub and play around with grub not enabling gfx (term == console) or pre-setting the desired resolution.

[Mike Jester (mikejester)]

That's correct, the VESA framebuffer (vesafb) is what I saw on lsmod on older kernels. I can do any other testing you'd like with different kernels or versions of Ubuntu as the system I'm doing this on is purely test. Pre-setting the desired resolution results in a slow console and setting GRUB_TERMINAL=console results in a good speed console, but the resolution is too low.

What you stated about checking the dmesg and see what framebuffer driver is actually being used made me wonder if I had missed something, so I did a bit of investigation.

In 3.13 with CONFIG_X86_SYSFB=on I get simple-framebuffer in my dmesg. With it off, I get vesafb.

Is there a way for me to force simple-framebuffer?

Is it possible that either the system isn't using vesafb in older kernels even though it lists it?

Is it possible there was regression in vesafb when it was merged into the kernel?

[Stefan Bader (smb)]

SYSFB/simpleframebuffer did not exist before 3.12 (iirc). When having CONFIG_X86_SYSFB=N there is no way to force it since it does not get built (as before). And given the other issues I do not think we should.

The difference with vesafb is not necessarily in code (and as that has been around for quite a while its rather unlikely the code changed there). The change just made the same code being part of the kernel instead of a loadable module.

What I think has changed is the way that grub might be setting up a framebuffer device and passing that into the kernel. Which is the reason I suggested playing around with that. Beside of generally preferring 80x24 in a server environment, I think I generally prefer to keep grub not changing anything there (which is what the GRUB_TERM=console should do + running update-grup to make that active). Then switching to the higher resolution is postponed until the kernel is taking over.

There should be messages in dmesg about any switching over and afaik /proc/fb would contain the currently used one (if switching was successful, might lie if that fails).

[Mike Jester (mikejester)]

Understood. I agree that not completely breaking virtual machines in some hypervisors is important. While I'm required to use bare-metal installs on these systems, I depend on VMs as much as the next person. There has to be an alternate solution.

Are you suggesting there's a kernel change needed that shouldn't load vesafb until later? Or are you saying I should set GRUB_TERMINAL=console and do something once it's booted? I can't modprobe vesafb now after all. Sorry if I'm a bit slow on the uptake here -- this is my first rodeo. I am primarily a C++ developer but I have not spent any time in the kernel before.

Unfortunately I can't change the config in any way that would cause regression in production. That includes lowering the resolution or just using 80x24. They're pretty strict on both regression and security here. Also, there is no SSH to these boxes so any interaction I have with them is sitting physically in front of them.

I got some more dmesg outputs and checked /proc/fb on each.

12.04 3.2 and 12.04 3.13 SYSFB=N displayed the same: 0 VESA VGA
12.04 3.13 SYSFB=Y was: 0 simple

[Stefan Bader (smb)]

No kernel change. Basically changes to /etc/default/grub are used when running update-grub to create the grub configuration. Leaving GRUB_TERM=console commented out causes grub to initialize some higher resolution graphics framebuffer mode. And (not quite sure) it might be that about some point in time this changed to be a higher resolution.

But looking at the two dmesg outputs you posted (possibly you have both kernels installed on the same box, so grub setup is the same in both cases just the newer kernel ends up being slower) shows both vesafb get initialized nearly identical. So right now its hard to say why things go so much slower in the newer kernels. The simple framebuffer imo should not be so much different in how it works. Cannot say why that works faster. simplefb might just be one of them. There is a whole bunch blacklisted in /etc/modprobe.d/blacklist-framebuffer.conf because they tend to cause issues.

One other detail I noticed in the dmesg output was that the newer kernel managed to initialize local ipmi drivers. Not sure how much this is actually needed. As long as its not required to have access to ipmi directly (not lan) one can disable ipmi_si and dev_ipmi can be disabled in /etc/default/openipmi. Just mentioning that since console usually can be accessed via ipmi and maybe there is some kind of interference.

[Mike Jester (mikejester)]

Yep, one machine, three kernels.

I didn't have /etc/default/openipmi since I don't have the openipmi package installed. I tried installing it and flipping those off, also I tried blacklisting ipmi_si and ipmi_devintf. Unfortunately, there was no change.

I can install and post logs for just about anything on this machine that's needed to help resolve the issue.

I also tried the final 4.0 kernel, and it behaves the same as the earlier 4.0-rc5 kernel I tried, so I updated the tags.

[Stefan Bader (smb)]

Ok, I try whether I can see a similar regression on a host I got at home when I get there. If that does not work or if you want to try out beforehand, http://kernel.ubuntu.com/~kernel-ppa/mainline/ has a number of builds between so you might try 3.12 (with simplefb blacklisted) to get a closer range of when this changed to the worse.

[Mike Jester (mikejester)]

Thanks, that'll help if your host has the G200 based IPMI card.

Mainline kernels before 3.13 don't work at all, and the versions of 3.13 built with SYSFB=y are fine while the ones with SYSFB=n are not. Comments 15 and 19.

I downloaded the source for 3.11.0-12, 3.12.0-1 and 3.13.0-1 from (https://launchpad.net/ubuntu/trusty/+source/linux) and built 3.12 and 3.13 with SYSFB=n. I'll be able to try them when I get back in the office on Monday.

[Mike Jester (mikejester)]

3.11.0-12 does not have the bug, while 3.12.0-1 with SYSFB=n does. It looks like 3.12 is when simplefb was implemented (as you noted in a previous post) and when vesafb was integrated.

Based on Joseph Salisbury's comment #24 above noting that the older Ubuntu kernels had SAUCE patches to make vesafb work on platforms like this, is it safe to conclude that the version of vesafb in the Ubuntu-specific kernels 3.11.0-12 and earlier is not the same as the version in upstream?

[Stefan Bader (smb)]

I think he was guessing that there might be additional patches, but if you tried the mainline builds for 3.11.x we can rule that out. The mainline builds are without any special driver patches for that reason. It looks like some kind of regression in vesafb (either due to changing that or something that happens because it gets initialized much earlier now). The simplefb happens to be one that does not show that issue but due to the other issues would be a problem for us if activated. So I would prefer to find out how to fix vesafb or whether another fb driver may work as well.
Unfortunately I did not yet get to try my box here due to suffering a bit from the jet lag. I am on it now, just a bit slow.

[Mike Jester (mikejester)]

3.11.x and earlier versions from mainline give me no display -- just a black screen on boot. The system does work, and I posted a dmesg from that boot earlier (dmesg110.txt is the name of the file.) So there's definitely some sort of SAUCE patch or patches that make vesafb work with those kernels, or at least with this particular hardware.

I tried a bunch of different fb drivers in the original post, but I did not try enabling matroxfb when re-building the kernel.

No worries, take your time.

[Stefan Bader (smb)]

So at least I can confirm that a boot with 3.13 and vesafb set to 1280x1024 by grub (the default otherwise was 640x480) scrolls by quite slowly. I need to get a pre-3.12 there to get to see the before state.

Looking at the code in 3.13, I see only one change to vesafb that came related to introducting simplefb:

commit 2e5155ecbb729d3a2e7d1cea7c18493516fec55a
Author: David Herrmann <email address hidden>
Date: Fri Aug 2 14:05:25 2013 +0200

    fbdev: vesafb: bind to platform-framebuffer device

I could not immediately find any SAUCE patches but I got rid of copies of previous releases as they went out of support and our archive is currently undergoing some maintenance. The change above is not directly addressing optimizations but changed the setup in a way which maybe affects passing in options (like use of mtrr).

[Stefan Bader (smb)]

Hm, interesting. When booting with 3.11 from Saucy, scrolling seems quicker (my interactivity local is buggered by a keyboard monitor switch which some day I should replace), at least I could not really read what scrolled by while I could with 3.13. And looking at /proc/mtrr I found a write-combining range added which was not done with 3.11:

reg03: base=0x0fc000000 ( 4032MB), size= 8MB, count=1: write-combining

Cannot say why right now but that could well explain the slowdown.

[Mike Jester (mikejester)]

Both of your finds are interesting considering the vesafb SAUCE patches seem to be making a change to mtrr.

https://lists.ubuntu.com/archives/kernel-team/2011-May/015600.html

https://lists.ubuntu.com/archives/kernel-team/2013-March/027111.html

[Mike Jester (mikejester)]

I added the change from the SAUCE patch in May of 2011 to 3.13.0-48 and rebuilt. It fixes the problem.

-static int mtrr __read_mostly; /* disable mtrr */
+static int mtrr __read_mostly = 3; /* disable mtrr */

I've attached the kernel, 3-13.0-48.80+testfb.

[Stefan Bader (smb)]

Ah, that now makes sense. And since I could reach and search the older git trees today I found that we had only one SAUCE patch to vesafb in Saucy/13.10. Which was a combination of previous patches but titled as modularization patch. So having vesafb as a module was special and since Trusty/14.04 we did not need it to be a module any longer. So the patch was dropped without noticing the mtrr part.

As a quick band-aid for production systems you could add "video=vesafb:mtrr:3" to standard kernels, which should result in the same. Though I think we should reintroduce the write-combining as default.

[Stefan Bader (smb)]

SRU Justification:

Impact: After dropping a sauce patch to make vesafb a module we accidentally dropped a change to make mtrr:3 (write-combining remap range) the default. This results in a performance degradation for graphics cards that have no drm framebuffer driver (especially on higher resolutions).

Fix: Change default for mtrr setting from none to 3 (write-combining).

Testcase: With any hardware that ends up having vesafb (VESA VGA) as framebuffer device performance is slower and the mapped address will not show up in /proc/mtrr. After changing the default /proc/mtrr should show a write-combining entry and performance should be improved.

[Mike Jester (mikejester)]

That kernel flag works. Thanks for pushing this along.

[Luis Henriques (henrix)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-trusty' to 'verification-done-trusty'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Luis Henriques (henrix)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-utopic' to 'verification-done-utopic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Luis Henriques (henrix)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-vivid' to 'verification-done-vivid'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Mike Jester (mikejester)]

I tried all three kernels on a Dell R320 with no kernel command line options and they all resolve the issue.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 3.19.0-17.17

---------------
linux (3.19.0-17.17) vivid; urgency=low

  [ Brad Figg ]

  * Release Tracking Bug
    - LP: #1452000

  [ Damien Lespiau ]

  * SAUCE: i915_bpo: drm/i915/skl: Fix stepping check for a couple of W/As
    - LP: #1449469
  * SAUCE: i915_bpo: drm/i915/skl: Implement WaDisableVFUnitClockGating
    - LP: #1449469
  * SAUCE: i915_bpo: drm/i915/skl: Add the INIT power domain to the MISC
    I/O power well
    - LP: #1449469
  * SAUCE: i915_bpo: drm/i915/skl: Fix the CTRL typo in the DPLL_CRTL1
    defines
    - LP: #1449469
  * SAUCE: i915_bpo: drm/i915/skl: Make the Misc I/O power well part of the
    PLLS domain
    - LP: #1449469
  * SAUCE: i915_bpo: drm/i915/skl: Deinit/init the display at
    suspend/resume
    - LP: #1449469
  * SAUCE: i915_bpo: drm/i915/skl: Change CDCLK behind PCU's back
    - LP: #1449469
  * SAUCE: i915_bpo: drm/i915/skl: gen6+ platforms support runtime PM
    - LP: #1449469

  [ Imre Deak ]

  * SAUCE: i915_bpo: drm/i915/gen9: fix PIPE_CONTROL flush for
    VS_INVALIDATE
    - LP: #1449469

  [ Leann Ogasawara ]

  * [Config] Set CONFIG_XEN_MAX_DOMAIN_MEMORY defaults

  [ Matt Roper ]

  * SAUCE: i915_bpo: drm/i915: Switch to full atomic helpers for plane
    updates/disable, take two
    - LP: #1449469

  [ Sonika Jindal ]

  * SAUCE: i915_bpo: drm/i915/skl: Allow universal planes to position
    - LP: #1449469
  * SAUCE: i915_bpo: drm/i915/skl: Support for 90/270 rotation
    - LP: #1449469
  * SAUCE: i915_bpo: drm/i915/skl: Add back HDMI translation table
    - LP: #1449469

  [ Stefan Bader ]

  * SAUCE: vesafb: Set mtrr:3 (write-combining) as default
    - LP: #1434581

  [ Timo Aaltonen ]

  * SAUCE: Call i915_bpo specific functions from the hda driver
    - LP: #1449464
  * SAUCE: i915_bpo: Use get_display_clock_speed
    - LP: #1449469
  * SAUCE: i915_bpo: Add a few register definitions
    - LP: #1449469

  [ Upstream Kernel Changes ]

  * Revert "sparc/PCI: Clip bridge windows to fit in upstream windows"
    - LP: #1446316
  * Revert "PM / hibernate: avoid unsafe pages in e820 reserved regions"
    - LP: #1446316
  * Revert "libceph: use memalloc flags for net IO"
    - LP: #1446316
  * Revert "net: Reset secmark when scrubbing packet"
    - LP: #1451996
  * ASoC: da732x: Fix control-less DAPM routes
    - LP: #1446316
  * ASoC: ak4671: Fix control-less DAPM routes
    - LP: #1446316
  * ASoC: sn95031: Fix control-less DAPM routes
    - LP: #1446316
  * ASoC: sgtl5000: remove useless register write clearing CHRGPUMP_POWERUP
    - LP: #1446316
  * ASoC: pcm1681: Fix wrong value references for boolean kctl
    - LP: #1446316
  * ASoC: cs4271: Fix wrong value references for boolean kctl
    - LP: #1446316
  * ASoC: es8238: Fix wrong value references for boolean kctl
    - LP: #1446316
  * ASoC: wm8960: Fix wrong value references for boolean kctl
    - LP: #1446316
  * ASoC: tas5086: Fix wrong value references for boolean kctl
    - LP: #1446316
  * ASoC: wm8731: Fix wrong value references for boolean kctl
    - LP: #1446316
  * ASoC: wm2000: Fix wrong value references for boolean kctl
    - LP: #1446316
  * ASoC: wm8903: Fix wrong value referen...

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 3.13.0-53.88

---------------
linux (3.13.0-53.88) trusty; urgency=low

  [ Brad Figg ]

  * Release Tracking Bug
    - LP: #1454785

  [ Upstream Kernel Changes ]

  * mmc: card: Don't access RPMB partitions for normal read/write
    - LP: #1454013

linux (3.13.0-53.87) trusty; urgency=low

  [ Luis Henriques ]

  * Release Tracking Bug
    - LP: #1452736

  [ dann frazier ]

  * [Config] CONFIG_{EFI_PARAMS_FROM_FDT,GENERIC_EARLY_IOREMAP,LIBFDT}=y
    - LP: #1441876
  * Move get_dram_base to arm private file
    - LP: #1441876
  * arm64: Implement efi_enabled()
    - LP: #1441876
  * [Config] CONFIG_RTC_DRV_EFI=y on arm64
    - LP: #1441291

  [ Kamal Mostafa ]

  * Fix "mei: me: release hw from reset only during the reset flow"
    - LP: #1450813

  [ Stefan Bader ]

  * SAUCE: vesafb: Set mtrr:3 (write-combining) as default
    - LP: #1434581

  [ Upstream Kernel Changes ]

  * Revert "net: cx82310_eth: use common match macro"
    - LP: #1451900
  * netfilter: nf_conntrack: reserve two bytes for nf_ct_ext->len
    - LP: #1442080
    - CVE-2014-9715
  * add generic fixmap.h
    - LP: #1441876
  * mm: create generic early_ioremap() support
    - LP: #1441876
  * arm64: initialize pgprot info earlier in boot
    - LP: #1441876
  * arm64: add early_ioremap support
    - LP: #1441876
  * arm64: fixmap: fix missing sub-page offset for earlyprintk
    - LP: #1441876
  * efi: create memory map iteration helper
    - LP: #1441876
  * efi: Add get_dram_base() helper function
    - LP: #1441876
  * lib: add fdt_empty_tree.c
    - LP: #1441876
  * doc: efi-stub.txt updates for ARM
    - LP: #1441876
  * efi: add helper function to get UEFI params from FDT
    - LP: #1441876
  * arm64: Add function to create identity mappings
    - LP: #1441876
  * efi: Add shared FDT related functions for ARM/ARM64
    - LP: #1441876
  * arm64: add EFI runtime services
    - LP: #1441876
  * doc: arm: add UEFI support documentation
    - LP: #1441876
  * arm64: efi: add EFI stub
    - LP: #1441876
  * doc: arm64: add description of EFI stub support
    - LP: #1441876
  * efi/arm64: ignore dtb= when UEFI SecureBoot is enabled
    - LP: #1441876
  * arm64: efi: only attempt efi map setup if booting via EFI
    - LP: #1441876
  * PCI: Don't clear ASPM bits when the FADT declares it's unsupported
    - LP: #1441335
  * regmap: Skip read-only registers in regcache_sync()
    - LP: #1448830
  * rtc: ia64: allow other architectures to use EFI RTC
    - LP: #1441291
  * rtc: Disable EFI rtc for x86
    - LP: #1441291
  * mei: me: fix hw ready reset flow
    - LP: #1450813
  * Input: serio - add firmware_id sysfs attribute
    - LP: #1414930
  * Input: i8042 - add firmware_id support
    - LP: #1414930
  * Input: Add INPUT_PROP_TOPBUTTONPAD device property
    - LP: #1414930
  * Input: synaptics - report INPUT_PROP_TOPBUTTONPAD property
    - LP: #1414930
  * Input: synaptics - add a matches_pnp_id helper function
    - LP: #1414930
  * Input: synaptics - change min/max quirk table to pnp-id matching
    - LP: #1414930
  * Input: psmouse - add psmouse_matches_pnp_id helper function
    - LP: #1414930
  * Input: synaptics - split synapt...

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 3.16.0-38.52

---------------
linux (3.16.0-38.52) utopic; urgency=low

  [ Luis Henriques ]

  * Release Tracking Bug
    - LP: #1452623

  [ Stefan Bader ]

  * SAUCE: vesafb: Set mtrr:3 (write-combining) as default
    - LP: #1434581

  [ Upstream Kernel Changes ]

  * regmap: Skip read-only registers in regcache_sync()
    - LP: #1448830
  * fuse: notify: don't move pages
    - LP: #1449548
  * fuse: set stolen page uptodate
    - LP: #1449548
  * dm thin: fix to consistently zero-fill reads to unprovisioned blocks
    - LP: #1449548
  * dm: hold suspend_lock while suspending device during device deletion
    - LP: #1449548
  * dm snapshot: suspend origin when doing exception handover
    - LP: #1449548
  * dm snapshot: suspend merging snapshot when doing exception handover
    - LP: #1449548
  * dm io: deal with wandering queue limits when handling REQ_DISCARD and
    REQ_WRITE_SAME
    - LP: #1449548
  * crypto: arm/aes update NEON AES module to latest OpenSSL version
    - LP: #1449548
  * mac80211: drop unencrypted frames in mesh fwding
    - LP: #1449548
  * mac80211: disable u-APSD queues by default
    - LP: #1449548
  * ASoC: ak4671: Fix control-less DAPM routes
    - LP: #1449548
  * ASoC: da732x: Fix control-less DAPM routes
    - LP: #1449548
  * ASoC: sn95031: Fix control-less DAPM routes
    - LP: #1449548
  * virtio_console: init work unconditionally
    - LP: #1449548
  * virtio_console: avoid config access from irq
    - LP: #1449548
  * clocksource: efm32: Fix a NULL pointer dereference
    - LP: #1449548
  * clockevents: sun5i: Fix setup_irq init sequence
    - LP: #1449548
  * x86/vdso: Fix the build on GCC5
    - LP: #1449548
  * ASoC: sgtl5000: remove useless register write clearing CHRGPUMP_POWERUP
    - LP: #1449548
  * regmap: regcache-rbtree: Fix present bitmap resize
    - LP: #1449548
  * regulator: Only enable disabled regulators on resume
    - LP: #1449548
  * regulator: core: Fix enable GPIO reference counting
    - LP: #1449548
  * Input: psmouse - add psmouse_matches_pnp_id helper function
    - LP: #1449548
  * Input: synaptics - split synaptics_resolution(), query first
    - LP: #1449548
  * Input: synaptics - log queried and quirked dimension values
    - LP: #1449548
  * Input: synaptics - query min dimensions for fw v8.1
    - LP: #1449548
  * Input: synaptics - remove obsolete min/max quirk for X240
    - LP: #1449548
  * Input: synaptics - support min/max board id in min_max_pnpid_table
    - LP: #1449548
  * Input: synaptics - skip quirks when post-2013 dimensions
    - LP: #1449548
  * Input: synaptics - fix middle button on Lenovo 2015 products
    - LP: #1449548
  * Input: synaptics - handle spurious release of trackstick buttons
    - LP: #1449548
  * Input: synaptics - do not retrieve the board id on old firmwares
    - LP: #1449548
  * vt6655: RFbSetPower fix missing rate RATE_12M
    - LP: #1449548
  * x86/asm/entry/32: Fix user_mode() misuses
    - LP: #1449548
  * ASoC: adav80x: Fix wrong value references for boolean kctl
    - LP: #1449548
  * ASoC: ak4641: Fix wrong value references for boolean kctl
    - LP: #1449548
  * ASoC: cs4271: Fix wrong...

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 3.19.0-18.18

---------------
linux (3.19.0-18.18) vivid; urgency=low

  [ Luis Henriques ]

  * Release Tracking Bug
    - LP: #1456732

  [ Upstream Kernel Changes ]

  * Revert "drm/i915: remove intel_pipe_set_base() (v4)"
    - LP: #1453593

linux (3.19.0-17.17) vivid; urgency=low

  [ Brad Figg ]

  * Release Tracking Bug
    - LP: #1452000

  [ Damien Lespiau ]

  * SAUCE: i915_bpo: drm/i915/skl: Fix stepping check for a couple of W/As
    - LP: #1449469
  * SAUCE: i915_bpo: drm/i915/skl: Implement WaDisableVFUnitClockGating
    - LP: #1449469
  * SAUCE: i915_bpo: drm/i915/skl: Add the INIT power domain to the MISC
    I/O power well
    - LP: #1449469
  * SAUCE: i915_bpo: drm/i915/skl: Fix the CTRL typo in the DPLL_CRTL1
    defines
    - LP: #1449469
  * SAUCE: i915_bpo: drm/i915/skl: Make the Misc I/O power well part of the
    PLLS domain
    - LP: #1449469
  * SAUCE: i915_bpo: drm/i915/skl: Deinit/init the display at
    suspend/resume
    - LP: #1449469
  * SAUCE: i915_bpo: drm/i915/skl: Change CDCLK behind PCU's back
    - LP: #1449469
  * SAUCE: i915_bpo: drm/i915/skl: gen6+ platforms support runtime PM
    - LP: #1449469

  [ Imre Deak ]

  * SAUCE: i915_bpo: drm/i915/gen9: fix PIPE_CONTROL flush for
    VS_INVALIDATE
    - LP: #1449469

  [ Leann Ogasawara ]

  * [Config] Set CONFIG_XEN_MAX_DOMAIN_MEMORY defaults

  [ Matt Roper ]

  * SAUCE: i915_bpo: drm/i915: Switch to full atomic helpers for plane
    updates/disable, take two
    - LP: #1449469

  [ Sonika Jindal ]

  * SAUCE: i915_bpo: drm/i915/skl: Allow universal planes to position
    - LP: #1449469
  * SAUCE: i915_bpo: drm/i915/skl: Support for 90/270 rotation
    - LP: #1449469
  * SAUCE: i915_bpo: drm/i915/skl: Add back HDMI translation table
    - LP: #1449469

  [ Stefan Bader ]

  * SAUCE: vesafb: Set mtrr:3 (write-combining) as default
    - LP: #1434581

  [ Timo Aaltonen ]

  * SAUCE: Call i915_bpo specific functions from the hda driver
    - LP: #1449464
  * SAUCE: i915_bpo: Use get_display_clock_speed
    - LP: #1449469
  * SAUCE: i915_bpo: Add a few register definitions
    - LP: #1449469

  [ Upstream Kernel Changes ]

  * Revert "sparc/PCI: Clip bridge windows to fit in upstream windows"
    - LP: #1446316
  * Revert "PM / hibernate: avoid unsafe pages in e820 reserved regions"
    - LP: #1446316
  * Revert "libceph: use memalloc flags for net IO"
    - LP: #1446316
  * Revert "net: Reset secmark when scrubbing packet"
    - LP: #1451996
  * ASoC: da732x: Fix control-less DAPM routes
    - LP: #1446316
  * ASoC: ak4671: Fix control-less DAPM routes
    - LP: #1446316
  * ASoC: sn95031: Fix control-less DAPM routes
    - LP: #1446316
  * ASoC: sgtl5000: remove useless register write clearing CHRGPUMP_POWERUP
    - LP: #1446316
  * ASoC: pcm1681: Fix wrong value references for boolean kctl
    - LP: #1446316
  * ASoC: cs4271: Fix wrong value references for boolean kctl
    - LP: #1446316
  * ASoC: es8238: Fix wrong value references for boolean kctl
    - LP: #1446316
  * ASoC: wm8960: Fix wrong value references for boolean kctl
    - LP: #1446316
  * ASoC: tas5086: Fix wrong value references for boolean kctl
    - ...