System freeze after update to 3.16.0-69-generic [NULL pointer reference in radeon_fence_ref]

[Philip Muškovac]

So far I've seen this happen on 2 systems today. On one only X froze, the other one panicked.
Booting into 3.16.0-67 is the workaround for now

ProblemType: KernelOops
Annotation: Your system might become unstable now and might need to be restarted.
Date: Wed Apr 6 09:57:21 2016
Failure: oops
OopsText:
 BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
 IP: [<ffffffffc03b23ed>] radeon_fence_ref+0xd/0x50 [radeon]
 PGD 0
 Oops: 0002 [#1] SMP
 Modules linked in: xt_nat xt_tcpudp veth xt_conntrack ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter ip_tables x_tables nf_nat nf_conntrack bridge stp llc aufs arc4 md4 pci_stub vboxpci(OE) vboxnetadp(OE) vboxnetflt(OE) vboxdrv(OE) rfcomm bnep bluetooth 6lowpan_iphc nls_utf8 cifs binfmt_misc fscache snd_hda_codec_via snd_hda_codec_generic radeon snd_hda_codec_hdmi snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep joydev snd_pcm snd_seq_midi ttm snd_seq_midi_event drm_kms_helper drm snd_rawmidi snd_seq i2c_algo_bit snd_seq_device kvm snd_timer snd ppdev serio_raw soundcore shpchp lp edac_core edac_mce_amd i2c_nforce2 mac_hid parport_pc asus_atk0110 k10temp parport hid_cherry hid_generic usbhid pata_acpi hid psmouse sata_nv forcedeth pata_amd
 CPU: 1 PID: 1543 Comm: Xorg Tainted: G OE 3.16.0-69-generic #89~14.04.1-Ubuntu
 Hardware name: System manufacturer System Product Name/M4N68T-M-V2, BIOS 0501 08/03/2010
 task: ffff8800b9b45bb0 ti: ffff8800b9e28000 task.ti: ffff8800b9e28000
 RIP: 0010:[<ffffffffc03b23ed>] [<ffffffffc03b23ed>] radeon_fence_ref+0xd/0x50 [radeon]
 RSP: 0018:ffff8800b9e2bad0 EFLAGS: 00010286
 RAX: 0000000000000000 RBX: ffff880035cf55f8 RCX: ffff880035cf4000
 RDX: 0000000000000001 RSI: 0000000000000000 RDI: 0000000000000000
 RBP: ffff8800b9e2bbd0 R08: ffff880035cf4d08 R09: ffff880035cf4000
 R10: 0000000000000000 R11: ffff8800b9e2bde8 R12: ffff880035cf5550
 R13: ffff8800b9e2bba8 R14: ffff8800b9e2bb78 R15: 0000000000000020
 FS: 00007f6e2ab398c0(0000) GS:ffff8801bfc40000(0000) knlGS:0000000000000000
 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
 CR2: 0000000000000008 CR3: 00000000b8192000 CR4: 00000000000007e0
 Stack:
  ffffffffc0414ad4 ffff8800b9b45bb0 ffff8800b9b45bb0 ffff880035cf4000
  ffff8800b9e2bca8 0000edc0b9e2bb90 0000000000000000 0020000000000100
  000000000004a8e0 ffff8800b9e2bb78 0000000000000000 0000000000000000
 Call Trace:
  [<ffffffffc0414ad4>] ? radeon_sa_bo_new+0x2b4/0x4f0 [radeon]
  [<ffffffff8118fc68>] ? insert_pfn.isra.61+0x48/0xa0
  [<ffffffffc03c7f43>] radeon_ib_get+0x33/0xe0 [radeon]
  [<ffffffffc03ca67b>] radeon_cs_ioctl+0x14b/0x770 [radeon]
  [<ffffffffc02be9ec>] drm_ioctl+0x1ec/0x660 [drm]
  [<ffffffff8105bfbc>] ? __do_page_fault+0x20c/0x560
  [<ffffffffc039604b>] radeon_drm_ioctl+0x4b/0x80 [radeon]
  [<ffffffff811e8d80>] do_vfs_ioctl+0x2e0/0x4c0
  [<ffffffff811e8fe1>] SyS_ioctl+0x81/0xa0
  [<ffffffff8177258d>] system_call_fastpath+0x1a/0x1f
 Code: dc 48 8b 3b 89 c1 44 89 e2 48 c7 c6 70 66 4a c0 31 c0 e8 d7 75 10 c1 eb c2 0f 1f 44 00 00 66 66 66 66 90 48 89 f8 ba 01 00 00 00 <f0> 0f c1 57 08 83 c2 01 83 fa 01 7e 01 c3 80 3d 5e 8b 11 00 00
 RIP [<ffffffffc03b23ed>] radeon_fence_ref+0xd/0x50 [radeon]
  RSP <ffff8800b9e2bad0>
 CR2: 0000000000000008
 ---[ end trace 4dfa7ee83f5a67e2 ]---

Package: linux-image-3.16.0-69-generic 3.16.0-69.89~14.04.1
SourcePackage: linux
Tags: kernel-oops
Uname: Linux 3.16.0-69-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.19
Architecture: amd64
Dependencies:
 adduser 3.113+nmu3ubuntu3
 apt-utils 1.0.1ubuntu2.11
 base-passwd 3.5.33
 busybox-initramfs 1:1.21.0-1ubuntu1
 coreutils 8.21-1ubuntu5.4
 cpio 2.11+dfsg-1ubuntu1.2
 dbus 1.6.18-0ubuntu4.3
 debconf 1.5.51ubuntu2
 debconf-i18n 1.5.51ubuntu2
 debianutils 4.4
 dmsetup 2:1.02.77-6ubuntu2
 dpkg 1.17.5ubuntu5.5
 e2fslibs 1.42.9-3ubuntu1.3
 e2fsprogs 1.42.9-3ubuntu1.3
 findutils 4.4.2-7
 gcc-4.8-base 4.8.4-2ubuntu1~14.04.1
 gcc-4.9-base 4.9.3-0ubuntu4
 gettext-base 0.18.3.1-1ubuntu3
 grub-common 2.02~beta2-9ubuntu1.7
 grub-gfxpayload-lists 0.6
 grub-pc 2.02~beta2-9ubuntu1.7
 grub-pc-bin 2.02~beta2-9ubuntu1.7
 grub2-common 2.02~beta2-9ubuntu1.7
 ifupdown 0.7.47.2ubuntu4.4
 initramfs-tools 0.103ubuntu4.3
 initramfs-tools-bin 0.103ubuntu4.3
 initscripts 2.88dsf-41ubuntu6.3
 insserv 1.14.0-5ubuntu2
 iproute2 3.12.0-2ubuntu1
 isc-dhcp-client 4.2.4-7ubuntu12.4
 isc-dhcp-common 4.2.4-7ubuntu12.4
 klibc-utils 2.0.3-0ubuntu1
 kmod 15-0ubuntu6
 libacl1 2.2.52-1
 libapparmor1 2.8.95~2430-0ubuntu5.3
 libapt-inst1.5 1.0.1ubuntu2.11
 libapt-pkg4.12 1.0.1ubuntu2.11
 libasprintf0c2 0.18.3.1-1ubuntu3
 libattr1 1:2.4.47-1ubuntu1
 libaudit-common 1:2.3.2-2ubuntu1
 libaudit1 1:2.3.2-2ubuntu1
 libblkid1 2.20.1-5.1ubuntu20.7
 libbz2-1.0 1.0.6-5
 libc6 2.19-0ubuntu6.7
 libcap2 1:2.24-0ubuntu2
 libcgmanager0 0.24-0ubuntu7.5
 libcomerr2 1.42.9-3ubuntu1.3
 libdb5.3 5.3.28-3ubuntu3
 libdbus-1-3 1.6.18-0ubuntu4.3
 libdebconfclient0 0.187ubuntu1
 libdevmapper1.02.1 2:1.02.77-6ubuntu2
 libdrm2 2.4.64-1~ubuntu14.04.1
 libexpat1 2.1.0-4ubuntu1.1
 libfreetype6 2.5.2-1ubuntu2.5
 libfuse2 2.9.2-4ubuntu4.14.04.1
 libgcc1 1:4.9.3-0ubuntu4
 libgpm2 1.20.4-6.1
 libjson-c2 0.11-4+deb.sury.org~trusty+2 [origin: LP-PPA-ondrej-php5-5.6]
 libjson0 0.11-4+deb.sury.org~trusty+2 [origin: LP-PPA-ondrej-php5-5.6]
 libklibc 2.0.3-0ubuntu1
 libkmod2 15-0ubuntu6
 liblocale-gettext-perl 1.05-7build3
 liblzma5 5.1.1alpha+20120614-2ubuntu2
 libmount1 2.20.1-5.1ubuntu20.7
 libncurses5 5.9+20140118-1ubuntu1
 libncursesw5 5.9+20140118-1ubuntu1
 libnih-dbus1 1.0.3-4ubuntu25
 libnih1 1.0.3-4ubuntu25
 libpam-modules 1.1.8-1ubuntu2.2
 libpam-modules-bin 1.1.8-1ubuntu2.2
 libpam-runtime 1.1.8-1ubuntu2.2
 libpam-systemd 204-5ubuntu20.18
 libpam0g 1.1.8-1ubuntu2.2
 libpcre3 1:8.31-2ubuntu2.2
 libplymouth2 0.8.8-0ubuntu17.1
 libpng12-0 1.2.50-1ubuntu2.14.04.2
 libprocps3 1:3.3.9-1ubuntu2.2
 libselinux1 2.2.2-1ubuntu0.1
 libsemanage-common 2.2-1
 libsemanage1 2.2-1
 libsepol1 2.2-1ubuntu0.1
 libslang2 2.2.4-15ubuntu1
 libss2 1.42.9-3ubuntu1.3
 libstdc++6 4.8.4-2ubuntu1~14.04.1
 libsystemd-daemon0 204-5ubuntu20.18
 libsystemd-login0 204-5ubuntu20.18
 libtext-charwidth-perl 0.04-7build3
 libtext-iconv-perl 1.7-5build2
 libtext-wrapi18n-perl 0.06-7
 libtinfo5 5.9+20140118-1ubuntu1
 libudev1 204-5ubuntu20.18
 libustr-1.0-1 1.0.4-3ubuntu2
 libuuid1 2.20.1-5.1ubuntu20.7
 libxtables10 1.4.21-1ubuntu1
 lsb-base 4.1+Debian11ubuntu6
 makedev 2.3.1-93ubuntu1
 module-init-tools 15-0ubuntu6
 mount 2.20.1-5.1ubuntu20.7
 mountall 2.53
 multiarch-support 2.19-0ubuntu6.7
 netbase 5.2
 os-prober 1.63ubuntu1.1
 passwd 1:4.1.5.1-1ubuntu9.2
 perl-base 5.18.2-2ubuntu1.1
 plymouth 0.8.8-0ubuntu17.1
 plymouth-theme-ubuntu-text 0.8.8-0ubuntu17.1
 procps 1:3.3.9-1ubuntu2.2
 psmisc 22.20-1ubuntu2
 sensible-utils 0.0.9
 systemd-services 204-5ubuntu20.18
 sysv-rc 2.88dsf-41ubuntu6.3
 sysvinit-utils 2.88dsf-41ubuntu6.3
 tar 1.27.1-1
 tzdata 2016c-0ubuntu0.14.04
 ucf 3.0027+nmu1
 udev 204-5ubuntu20.18
 upstart 1.12.1-0ubuntu4.2
 util-linux 2.20.1-5.1ubuntu20.7
 uuid-runtime 2.20.1-5.1ubuntu20.7
 zlib1g 1:1.2.8.dfsg-1ubuntu1
DistroRelease: Ubuntu 14.04
InstallationDate: Installed on 2015-10-19 (169 days ago)
InstallationMedia: Ubuntu 14.04.2 LTS "Trusty Tahr" - Release amd64 (20150218.1)
PackageArchitecture: amd64
ProcVersionSignature: Ubuntu 3.16.0-69.89~14.04.1-generic 3.16.7-ckt25
Title: BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
UpgradeStatus: No upgrade log present (probably fresh install)
_MarkForUpload: True

[Philip Muškovac (yofel)]

Probably relevant: https://lkml.org/lkml/2016/3/9/590

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Joseph Salisbury (jsalisbury)]

It looks like the offending commit is already reverted in lts-utopic -next:

a907233 Revert "drm/radeon: hold reference to fences in radeon_sa_bo_new"
72a5cff drm/radeon: hold reference to fences in radeon_sa_bo_new

I built a test kernel of -next, which will be the next release. The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1566727/

Can you test this kernel and see if it resolves this bug?

[Kamal Mostafa (kamalmostafa)]

To address this bug, we're going to release 3.16.0-70.90~14.04.1 comprised of just this fix. That kernel will become available in the -proposed archive in the next day or two (an announcement will be posted here).

In the meantime, a preview build of that kernel is available here (this replaces jsalisbury's test kernel from comment #4):

http://kernel.ubuntu.com/~kamal/lp1566726.0/

Affected users, please confirm that preview kernel resolves the problem (likely only the linux-image-3.16.0-70...deb needs to be installed).

[Matt (darigitin)]

Just to pile on, ran into the same issue in my CS computer lab. causing multiple systems to lock up. Thank you for the quick fix.

[Philip Muškovac (yofel)]

Kamal's build resolves the problem on one of our machines, I'll test the other one once the package is in proposed.

[Kamal Mostafa (kamalmostafa)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-trusty' to 'verification-done-trusty'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Kamal Mostafa (kamalmostafa)]

Thanks for the confirmation Philip. The official fixed kernel is now ready in -proposed; We'd like confirmation (from any affected user) for this kernel as well.

Quick link to the -proposed amd64 .deb (easier than the full "EnableProposed" method):

https://launchpad.net/ubuntu/+archive/primary/+files/linux-image-3.16.0-70-generic_3.16.0-70.90~14.04.1_amd64.deb

[Tim Wescott (ww3ib0-tim)]

just loaded the above 3.16.0-70 kernel -- I didn't stay long enough to see if it solves the problem, because it killed my trackball (and old USB Logitech). 3.16.0-67 works, though (it's what I'm using at the moment).

[Kamal Mostafa (kamalmostafa)]

Tim Wescott - Its seems most likely that your trackball problem is a separate issue introduced between -67 and -69. Please verify that by checking your trackball with -69 installed. Assuming that the trackball problem happens there too, file a new bug report. (If on the other hand, you find that -70 really does introduce the trackball problem, certainly let us know here!). The "-69" kernel (3.16.0-69.89~14.04.1) is the current main -updates kernel version for linux-lts-utopic, so it should be available to install via your standard apt sources.

[Kamal Mostafa (kamalmostafa)]

Tim Wescott - Another possibility for your trackball problem... It might just be that your trackball driver is actually in the "linux-image-extras" package, so manually installing the -70 "linux-image" .deb using the link I provided in comment #9 is insufficient. Try installing both of these:

https://launchpad.net/ubuntu/+archive/primary/+files/linux-image-3.16.0-70-generic_3.16.0-70.90~14.04.1_amd64.deb

https://launchpad.net/ubuntu/+archive/primary/+files/linux-image-extra-3.16.0-70-generic_3.16.0-70.90~14.04.1_amd64.deb

[Philip Muškovac (yofel)]

And the build from -proposed fixes the other machine as well, thanks.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux-lts-utopic - 3.16.0-70.90~14.04.1

---------------
linux-lts-utopic (3.16.0-70.90~14.04.1) trusty; urgency=low

  [ Kamal Mostafa ]

  * Release Tracking Bug
    - LP: #1566973

  [ Upstream Kernel Changes ]

  * Revert "drm/radeon: hold reference to fences in radeon_sa_bo_new"
    - LP: #1566726

 -- Kamal Mostafa <email address hidden> Wed, 06 Apr 2016 09:40:04 -0700

[Adam Conrad (adconrad)]

The verification of the Stable Release Update for linux-lts-utopic has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Tim Wescott (ww3ib0-tim)]

Kamal Mostafa:

You were correct. Installing the "extras" made my track ball work. The computer has now held up for several hours, which is considerably longer than it survived under the -69 version. So -- I think the -70 fixes it, at least for me.

[Passuf (passuf)]

I can confirm that the -70 update fixes the problem on my machine. Thanks for everything!

[Grégory (gregory-grosso)]

No problem since -70 update. Thanks for all.