system fails to boot with kernel 3.10.0 when using radeon.audio=1

Bug #1195687 reported by Marco Trevisan (Treviño)
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Linux
Fix Released
Critical
linux (Ubuntu)
Fix Released
Undecided
Unassigned
upstart (Ubuntu)
Invalid
Undecided
Unassigned

Bug Description

Since the upgrade to the kernel 3.10.0 I can't boot, due to an init (it seems) crash.
If I try to boot in recovery mode and then I continue to a normal session, all works well.
Going back to kernel 3.9.0 all works again.

UPDATE: It seems that the problem is caused by the radeon.audio=1 string, adding that to kernel parameters, makes my system not to boot (when HDMI is plugged in).

ProblemType: CrashDistroRelease: Ubuntu 13.10
Package: upstart 1.8-0ubuntu7
ProcVersionSignature: Ubuntu 3.9.0-7.15-generic 3.9.7
Uname: Linux 3.9.0-7-generic x86_64
ApportVersion: 2.10.2-0ubuntu3
Architecture: amd64
Date: Mon Jun 24 18:29:12 2013
ExecutablePath: /sbin/init
InstallationDate: Installed on 2010-07-10 (1083 days ago)InstallationMedia: Ubuntu 10.04 LTS "Lucid Lynx" - Release amd64 (20100429)
MarkForUpload: True
ProcCmdline: init --user
ProcEnviron:
 LANGUAGE=it_IT
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=it_IT.UTF-8
 SHELL=/bin/bash
ProcKernelCmdline: BOOT_IMAGE=/boot/vmlinuz-3.9.0-7-generic root=UUID=4811a166-63ca-4702-8e63-0c357cc2e2f7 ro crashkernel=384M-2G:64M,2G-:128M quiet splash radeon.audio=1SourcePackage: upstart
UpgradeStatus: Upgraded to saucy on 2012-10-10 (261 days ago)
UpstartBugCategory: Unknown
UpstartRunningSessionVersion: init (upstart 1.8-ubuntu-full-serialization)
UpstartRunningSystemVersion: init (upstart 1.8-ubuntu-full-serialization)
modified.conffile..etc.init.d.gdm: [modified]
modified.conffile..etc.init.d.networking: [modified]
mtime.conffile..etc.init.d.gdm: 2013-06-17T15:24:42
mtime.conffile..etc.init.d.networking: 2013-05-18T07:37:44
---
ApportVersion: 2.10.2-0ubuntu3Architecture: amd64DistroRelease: Ubuntu 13.10
InstallationDate: Installed on 2010-07-10 (1087 days ago)InstallationMedia: Ubuntu 10.04 LTS "Lucid Lynx" - Release amd64 (20100429)
MarkForUpload: True
Package: upstart 1.8-0ubuntu7
PackageArchitecture: amd64
ProcKernelCmdline: BOOT_IMAGE=/boot/vmlinuz-3.10.0-1-generic root=UUID=4811a166-63ca-4702-8e63-0c357cc2e2f7 ro recovery nomodeset
ProcVersionSignature: Ubuntu 3.10.0-1.8-generic 3.10.0-rc7
Tags: saucy
Uname: Linux 3.10.0-1-generic x86_64
UpgradeStatus: Upgraded to saucy on 2012-10-10 (264 days ago)
UpstartBugCategory: System
UserGroups: adm admin audio cdrom debian-tor dialout dip fuse lp lpadmin netdev plugdev sambashare sbuild vboxusers video
modified.conffile..etc.at.deny: [inaccessible: [Errno 13] Permesso negato: u'/etc/at.deny']
modified.conffile..etc.init.d.gdm: [modified]
modified.conffile..etc.init.d.networking: [modified]
mtime.conffile..etc.init.d.gdm: 2013-06-17T15:24:42
mtime.conffile..etc.init.d.networking: 2013-05-18T07:37:44
upstart.logrotate.log: gzip: stdin: file size changed while zipping
upstart.upstart-file-bridge.log:
 Job got added /com/ubuntu/Upstart/jobs/unicast_2dlocal_2davahi
 Job got added /com/ubuntu/Upstart/jobs/update_2dnotifier_2dcrash
 Job got added /com/ubuntu/Upstart/jobs/update_2dnotifier_2drelease
 upstart-file-bridge: Impossibile scrivere il file pid: Permesso negato
---
ApportVersion: 2.10.2-0ubuntu3
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC1: marco 3320 F.... pulseaudio
 /dev/snd/controlC0: marco 3320 F.... pulseaudioDistroRelease: Ubuntu 13.10
HibernationDevice: RESUME=UUID=534ddf57-b0f4-4367-ae56-d2878b2f614f
InstallationDate: Installed on 2010-07-10 (1087 days ago)InstallationMedia: Ubuntu 10.04 LTS "Lucid Lynx" - Release amd64 (20100429)
MachineType: Acer Aspire 4820TG
MarkForUpload: True
Package: linux-image-3.10.0-1-generic 3.10.0-1.8
PackageArchitecture: amd64
ProcFB: 0 radeondrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.9.0-7-generic root=UUID=4811a166-63ca-4702-8e63-0c357cc2e2f7 ro crashkernel=384M-2G:64M,2G-:128M quiet splash radeon.audio=1
ProcVersionSignature: Ubuntu 3.9.0-7.15-generic 3.9.7
RelatedPackageVersions:
 linux-restricted-modules-3.9.0-7-generic N/A
 linux-backports-modules-3.9.0-7-generic N/A
 linux-firmware 1.110
Tags: saucy
Uname: Linux 3.9.0-7-generic x86_64
UpgradeStatus: Upgraded to saucy on 2012-10-10 (264 days ago)
UserGroups: adm admin audio cdrom debian-tor dialout dip fuse lp lpadmin netdev plugdev sambashare sbuild vboxusers video
dmi.bios.date: 03/16/2011
dmi.bios.vendor: INSYDE
dmi.bios.version: V1.25
dmi.board.asset.tag: Base Board Asset Tag
dmi.board.name: JM41_CP
dmi.board.vendor: Acer
dmi.board.version: Base Board Version
dmi.chassis.type: 10
dmi.chassis.vendor: Chassis Manufacturer
dmi.chassis.version: Chassis Version
dmi.modalias: dmi:bvnINSYDE:bvrV1.25:bd03/16/2011:svnAcer:pnAspire4820TG:pvrV1.25:rvnAcer:rnJM41_CP:rvrBaseBoardVersion:cvnChassisManufacturer:ct10:cvrChassisVersion:
dmi.product.name: Aspire 4820TG
dmi.product.version: V1.25
dmi.sys.vendor: Acer

Revision history for this message
Marco Trevisan (Treviño) (3v1n0) wrote :
description: updated
Revision history for this message
Marco Trevisan (Treviño) (3v1n0) wrote : BootLog.txt

apport information

description: updated
tags: added: apport-collected
description: updated
Revision history for this message
Marco Trevisan (Treviño) (3v1n0) wrote : Dependencies.txt

apport information

Revision history for this message
Marco Trevisan (Treviño) (3v1n0) wrote : HookError_upstart.txt

apport information

Revision history for this message
Marco Trevisan (Treviño) (3v1n0) wrote : ProcEnviron.txt

apport information

Revision history for this message
James Hunt (jamesodhunt) wrote : Re: Init crashes with kernel 3.10.0

Does your system fail to boot or is it that you are simply unable to login?

If it is failing to boot, please remove 'quiet' and 'splash' from your kernel command-line and either attach a log of the console output, or just attach a picture showing crash details.

Revision history for this message
Marco Trevisan (Treviño) (3v1n0) wrote :

Unfortunately also after removing them, I'm getting a log and then only a blank screen (making impossible to read or take a pic of anything). Any better way to get a log?

Changed in upstart (Ubuntu):
status: New → Invalid
summary: - Init crashes with kernel 3.10.0
+ Init crashes with kernel 3.10.0 when using radeon.audio=1
Revision history for this message
Marco Trevisan (Treviño) (3v1n0) wrote : AlsaInfo.txt

apport information

description: updated
description: updated
Revision history for this message
Marco Trevisan (Treviño) (3v1n0) wrote : BootDmesg.txt

apport information

Revision history for this message
Marco Trevisan (Treviño) (3v1n0) wrote : CRDA.txt

apport information

Revision history for this message
Marco Trevisan (Treviño) (3v1n0) wrote : CurrentDmesg.txt

apport information

Revision history for this message
Marco Trevisan (Treviño) (3v1n0) wrote : Dependencies.txt

apport information

Revision history for this message
Marco Trevisan (Treviño) (3v1n0) wrote : IwConfig.txt

apport information

Revision history for this message
Marco Trevisan (Treviño) (3v1n0) wrote : Lspci.txt

apport information

Revision history for this message
Marco Trevisan (Treviño) (3v1n0) wrote : Lsusb.txt

apport information

Revision history for this message
Marco Trevisan (Treviño) (3v1n0) wrote : ProcCpuinfo.txt

apport information

Revision history for this message
Marco Trevisan (Treviño) (3v1n0) wrote : ProcEnviron.txt

apport information

Revision history for this message
Marco Trevisan (Treviño) (3v1n0) wrote : ProcInterrupts.txt

apport information

Revision history for this message
Marco Trevisan (Treviño) (3v1n0) wrote : ProcModules.txt

apport information

Revision history for this message
Marco Trevisan (Treviño) (3v1n0) wrote : PulseList.txt

apport information

Revision history for this message
Marco Trevisan (Treviño) (3v1n0) wrote : RfKill.txt

apport information

Revision history for this message
Marco Trevisan (Treviño) (3v1n0) wrote : UdevDb.txt

apport information

Revision history for this message
Marco Trevisan (Treviño) (3v1n0) wrote : UdevLog.txt

apport information

Revision history for this message
Marco Trevisan (Treviño) (3v1n0) wrote : WifiSyslog.txt

apport information

Revision history for this message
James Hunt (jamesodhunt) wrote : Re: Init crashes with kernel 3.10.0 when using radeon.audio=1

14:41 <Trevinho> jodh: unfortunately I'm getting a black screen...
14:41 <Trevinho> jodh: also removing splash and quiet I'm getting a fast log,
                 then the screen gets blank
14:43 <Trevinho> (and nothing in /var/crash this time)

14:55 <Trevinho> jodh: I'm using the custom cmd line for linux radeon.audio=1
14:55 <Trevinho> jodh: removing that it boots with normal boot profile..
14:55 <Trevinho> jodh: that's weird btw... as it seems to work in recovery

15:02 <Trevinho> jodh: just checked... not using hdmi audio in recovery mode
                 so it seems mostly a kernel issue.. mh

summary: - Init crashes with kernel 3.10.0 when using radeon.audio=1
+ system fails to boot with kernel 3.10.0 when using radeon.audio=1
Revision history for this message
Brad Figg (brad-figg) wrote : Status changed to Confirmed

This change was made by a bot.

Changed in linux (Ubuntu):
status: New → Confirmed
Revision history for this message
Marco Trevisan (Treviño) (3v1n0) wrote :
Download full text (4.6 KiB)

Looking at the kern.log after a boot failure, I see that it contains:

Jul 8 23:23:03 tricky kernel: [ 30.386559] BUG: unable to handle kernel NULL pointer dereference at (null)
Jul 8 23:23:03 tricky kernel: [ 30.386626] IP: [<ffffffffa024be90>] evergreen_hdmi_enable+0x20/0x90 [radeon]
Jul 8 23:23:03 tricky kernel: [ 30.386715] PGD 0
Jul 8 23:23:03 tricky kernel: [ 30.386731] Oops: 0000 [#1] SMP
Jul 8 23:23:03 tricky kernel: [ 30.387209] CPU: 2 PID: 2184 Comm: Xorg Not tainted 3.10.0-996-generic #201307020454
Jul 8 23:23:03 tricky kernel: [ 30.387260] Hardware name: Acer Aspire 4820TG/JM41_CP, BIOS V1.25 03/16/2011
Jul 8 23:23:03 tricky kernel: [ 30.387306] task: ffff88022c33ddc0 ti: ffff88022f3d2000 task.ti: ffff88022f3d2000
Jul 8 23:23:03 tricky kernel: [ 30.387354] RIP: 0010:[<ffffffffa024be90>] [<ffffffffa024be90>] evergreen_hdmi_enable+0x20/0x90 [radeon]
Jul 8 23:23:03 tricky kernel: [ 30.387447] RSP: 0018:ffff88022f3d3b68 EFLAGS: 00010246
Jul 8 23:23:03 tricky kernel: [ 30.387482] RAX: ffff88022b02de00 RBX: ffff88022b02ea00 RCX: 0000000000000007
Jul 8 23:23:03 tricky kernel: [ 30.387529] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff88022b02ea00
Jul 8 23:23:03 tricky kernel: [ 30.387575] RBP: ffff88022f3d3b78 R08: 0000000000000023 R09: 0000000000000022
Jul 8 23:23:03 tricky kernel: [ 30.387621] R10: ffff88022b724880 R11: 0000000000000000 R12: ffff88022b1f0000
Jul 8 23:23:03 tricky kernel: [ 30.387667] R13: ffff88022bb13000 R14: ffff88022bb13000 R15: ffff8802307e6000
Jul 8 23:23:03 tricky kernel: [ 30.387716] FS: 00007f34c3505980(0000) GS:ffff88023bc80000(0000) knlGS:0000000000000000
Jul 8 23:23:03 tricky kernel: [ 30.387768] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 8 23:23:03 tricky kernel: [ 30.387807] CR2: 0000000000000000 CR3: 000000022ed44000 CR4: 00000000000007e0
Jul 8 23:23:03 tricky kernel: [ 30.387854] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jul 8 23:23:03 tricky kernel: [ 30.387901] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jul 8 23:23:03 tricky kernel: [ 30.387947] Stack:
Jul 8 23:23:03 tricky kernel: [ 30.387963] ffff88022b1f0000 ffff88022bb13000 ffff88022f3d3ba8 ffffffffa0252f3f
Jul 8 23:23:03 tricky kernel: [ 30.388016] ffff88022b02ea00 ffff88022bb13478 ffff88022bb13000 00000000ffffffde
Jul 8 23:23:03 tricky kernel: [ 30.388070] ffff88022f3d3bd8 ffffffffa010ab3a ffff88022e4b7cf0 ffff8802307e6000
Jul 8 23:23:03 tricky kernel: [ 30.388122] Call Trace:
Jul 8 23:23:03 tricky kernel: [ 30.388173] [<ffffffffa0252f3f>] radeon_atom_encoder_disable+0x15f/0x170 [radeon]
Jul 8 23:23:03 tricky kernel: [ 30.388231] [<ffffffffa010ab3a>] drm_helper_disable_unused_functions+0x14a/0x190 [drm_kms_helper]
Jul 8 23:23:03 tricky kernel: [ 30.388295] [<ffffffffa010c5fe>] drm_crtc_helper_set_config+0x95e/0xb50 [drm_kms_helper]
Jul 8 23:23:03 tricky kernel: [ 30.388357] [<ffffffffa00931f8>] ? ttm_bo_vm_fault+0x288/0x3f0 [ttm]
Jul 8 23:23:03 tricky kernel: [ 30.388420] [<ffffffffa01effec>] ? radeon_ttm_fault+0x5c/0x70 [radeon]
Jul 8 23:23:03 tricky kernel: [ 30.388482]...

Read more...

Revision history for this message
In , Marco Trevisan (Treviño) (3v1n0) wrote :
Download full text (4.8 KiB)

If I enable the HDMI audio using the Mobility Radeon HD 5650 card with kernel 3.10.0 (or newer), I get a kernel oops that prevents me to boot.

This is a regression, since everything is fine with kernel 3.9.0.

Also, when I enable the switchable graphics in the bios, the notebook boots but as soon as I use vgaswitcheroo to switch to the discrete card, I still get a black screen and the system can't be used (when restoring the integrate card, it goes back to work).

This is the kernel failure I've found:
Jul 8 23:23:03 tricky kernel: [ 30.386559] BUG: unable to handle kernel NULL pointer dereference at (null)
Jul 8 23:23:03 tricky kernel: [ 30.386626] IP: [<ffffffffa024be90>] evergreen_hdmi_enable+0x20/0x90 [radeon]
Jul 8 23:23:03 tricky kernel: [ 30.386715] PGD 0
Jul 8 23:23:03 tricky kernel: [ 30.386731] Oops: 0000 [#1] SMP
Jul 8 23:23:03 tricky kernel: [ 30.387209] CPU: 2 PID: 2184 Comm: Xorg Not tainted 3.10.0-996-generic #201307020454
Jul 8 23:23:03 tricky kernel: [ 30.387260] Hardware name: Acer Aspire 4820TG/JM41_CP, BIOS V1.25 03/16/2011
Jul 8 23:23:03 tricky kernel: [ 30.387306] task: ffff88022c33ddc0 ti: ffff88022f3d2000 task.ti: ffff88022f3d2000
Jul 8 23:23:03 tricky kernel: [ 30.387354] RIP: 0010:[<ffffffffa024be90>] [<ffffffffa024be90>] evergreen_hdmi_enable+0x20/0x90 [radeon]
Jul 8 23:23:03 tricky kernel: [ 30.387447] RSP: 0018:ffff88022f3d3b68 EFLAGS: 00010246
Jul 8 23:23:03 tricky kernel: [ 30.387482] RAX: ffff88022b02de00 RBX: ffff88022b02ea00 RCX: 0000000000000007
Jul 8 23:23:03 tricky kernel: [ 30.387529] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff88022b02ea00
Jul 8 23:23:03 tricky kernel: [ 30.387575] RBP: ffff88022f3d3b78 R08: 0000000000000023 R09: 0000000000000022
Jul 8 23:23:03 tricky kernel: [ 30.387621] R10: ffff88022b724880 R11: 0000000000000000 R12: ffff88022b1f0000
Jul 8 23:23:03 tricky kernel: [ 30.387667] R13: ffff88022bb13000 R14: ffff88022bb13000 R15: ffff8802307e6000
Jul 8 23:23:03 tricky kernel: [ 30.387716] FS: 00007f34c3505980(0000) GS:ffff88023bc80000(0000) knlGS:0000000000000000
Jul 8 23:23:03 tricky kernel: [ 30.387768] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 8 23:23:03 tricky kernel: [ 30.387807] CR2: 0000000000000000 CR3: 000000022ed44000 CR4: 00000000000007e0
Jul 8 23:23:03 tricky kernel: [ 30.387854] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jul 8 23:23:03 tricky kernel: [ 30.387901] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jul 8 23:23:03 tricky kernel: [ 30.387947] Stack:
Jul 8 23:23:03 tricky kernel: [ 30.387963] ffff88022b1f0000 ffff88022bb13000 ffff88022f3d3ba8 ffffffffa0252f3f
Jul 8 23:23:03 tricky kernel: [ 30.388016] ffff88022b02ea00 ffff88022bb13478 ffff88022bb13000 00000000ffffffde
Jul 8 23:23:03 tricky kernel: [ 30.388070] ffff88022f3d3bd8 ffffffffa010ab3a ffff88022e4b7cf0 ffff8802307e6000
Jul 8 23:23:03 tricky kernel: [ 30.388122] Call Trace:
Jul 8 23:23:03 tricky kernel: [ 30.388173] [<ffffffffa0252f3f>] radeon_atom_encoder_disable+0x15f/0x170 [radeon]
Jul 8 23:23:03 tricky kernel: [ 30.388231] [<ffffffffa010ab3a>] drm_helper_disable_unused_functions+0x14a/0x190 [drm_kms_helper]
Jul 8 23:23:03 tricky kernel: [ 30.38829...

Read more...

Revision history for this message
In , agd5f (agd5f) wrote :

Please attach your full dmesg output.

Revision history for this message
In , agd5f (agd5f) wrote :

Created attachment 82200
possible fix

Does this patch fix the issue?

Revision history for this message
In , Marco Trevisan (Treviño) (3v1n0) wrote :

Created attachment 82202
dmesg

Full dmesg of the failure

Changed in linux:
importance: Unknown → Critical
status: Unknown → Confirmed
Revision history for this message
In , Marco Trevisan (Treviño) (3v1n0) wrote :

So, I've tried the patch on a 3.10 kernel that was affected and... Great, it Works! :)
FYI, I've also added a couple of error logs in the code to see where exactly the failure was and this was the result:

[ 2.073137] [drm] fb mappable at 0xC035F000
[ 2.073139] [drm] vram apper at 0xC0000000
[ 2.073140] [drm] size 8294400
[ 2.073141] [drm] fb depth is 24
[ 2.073142] [drm] pitch is 7680
[ 2.073252] fbcon: radeondrmfb (fb0) is primary device
[ 2.073323] [drm:evergreen_hdmi_enable] *ERROR* Invalid DIG (dig: ffff880229d0fc00, dig->afmt: (null)) /home/marco/Dev/debs/linux-3.10.0/drivers/gpu/drm/radeon/evergreen_hdmi.c:298
[ 2.573589] [drm:evergreen_hdmi_enable] *ERROR* Invalid DIG (dig: ffff880229d0fc00, dig->afmt: (null)) /home/marco/Dev/debs/linux-3.10.0/drivers/gpu/drm/radeon/evergreen_hdmi.c:298
[ 2.607449] Console: switching to colour frame buffer device 170x48
[ 2.610160] radeon 0000:01:00.0: fb0: radeondrmfb frame buffer device
[ 2.610162] radeon 0000:01:00.0: registered panic notifier
[ 2.610237] [drm] Initialized radeon 2.33.0 20080528 for 0000:01:00.0 on minor 0

Full dmesg at http://pastebin.ubuntu.com/5873667/
I don't know if this may be only the side effect of another issue (wrong initialization order?).

One more thing I discovered in the past days, is that the kernel crash was not happening in case that the HDMI was plugged after that ligthtdm was running (attaching it anytime before was leading to a freeze).

Revision history for this message
In , Marco Trevisan (Treviño) (3v1n0) wrote :

Comment on attachment 82200
possible fix

Review of attachment 82200:
-----------------------------------------------------------------

Why not including some DRM_ERROR logs as well?

Revision history for this message
Marco Trevisan (Treviño) (3v1n0) wrote :

So, the proposed patch on https://bugs.freedesktop.org/show_bug.cgi?id=66714 is enough to fix this boot failure, please merge this in ubuntu kernel :)

tags: added: patch
description: updated
Revision history for this message
In , Deathsimple (deathsimple) wrote :

(In reply to comment #5)
> Comment on attachment 82200 [details] [review]
> possible fix
>
> Review of attachment 82200 [details] [review]:
> -----------------------------------------------------------------
>
> Why not including some DRM_ERROR logs as well?

Because that isn't a bug.

You just have an LVDS pannel connected to a DIG encoder and so this encoder doesn't have an audio block.

Alex patch is already quite right.

Changed in linux:
status: Confirmed → Fix Released
Revision history for this message
Paolo Pisati (p-pisati) wrote :
Revision history for this message
Marco Trevisan (Treviño) (3v1n0) wrote :

Thanks Paolo booting with the image you linked and http://people.canonical.com/~ppisati/linux-image-extra-3.10.0-3-generic_3.10.0-3.13~lp1195687_amd64.deb all went fine as expected... Awesome! ;-)

Revision history for this message
In , Marco Trevisan (Treviño) (3v1n0) wrote :

Not sure if this has been already done, but I guess that this fix should be backported to linux 3.10

Revision history for this message
Marco Trevisan (Treviño) (3v1n0) wrote :

This has been fixed in ubuntu package linux-image-3.10.0-6 (commit http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-saucy.git;a=commit;h=a77b0b46), thanks! ;-)

Changed in linux (Ubuntu):
status: Confirmed → Fix Released
Revision history for this message
In , agd5f (agd5f) wrote :

The patch will show up in the stable kernels eventually.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.