Xorg assert failure: Xorg: /usr/include/xorg/privates.h:122: dixGetPrivateAddr: Assertion `key->initialized' failed.

Bug #1745345 reported by Joao Cerveira
64
This bug affects 12 people
Affects Status Importance Assigned to Milestone
xorg-server (Ubuntu)
Confirmed
High
Unassigned

Bug Description

Xorg assert failure: Xorg: /usr/include/xorg/privates.h:122: dixGetPrivateAddr: Assertion `key->initialized' failed.

This actually seems to happen most frequently in VirtualBox.

WORKAROUND:

Select "Ubuntu on Wayland" from the login screen before logging in.

---

It happened after upgrade to bionic, in the first login, after logon screen.
Instead of opening the desktop area, gave the error. I had to do hard reset to be able to use X.
After hard reset showed the same error but after a while it entered desktop.

ProblemType: Crash
DistroRelease: Ubuntu 18.04
Package: xserver-xorg-core 2:1.19.5-0ubuntu2
ProcVersionSignature: Ubuntu 4.13.0-31.34-generic 4.13.13
Uname: Linux 4.13.0-31-generic x86_64
ApportVersion: 2.20.8-0ubuntu6
Architecture: amd64
AssertionMessage: Xorg: /usr/include/xorg/privates.h:122: dixGetPrivateAddr: Assertion `key->initialized' failed.
CompositorRunning: None
CurrentDesktop: GNOME-Greeter:GNOME
Date: Thu Jan 25 10:09:41 2018
DistUpgraded: 2018-01-25 02:34:06,581 ERROR got error from PostInstallScript ./xorg_fix_proprietary.py ([Errno 2] Ficheiro ou directoria inexistente: './xorg_fix_proprietary.py': './xorg_fix_proprietary.py')
DistroCodename: bionic
DistroVariant: ubuntu
ExecutablePath: /usr/lib/xorg/Xorg
ExtraDebuggingInterest: Yes, if not too technical
GraphicsCard:
 Intel Corporation HD Graphics 5500 [8086:1616] (rev 09) (prog-if 00 [VGA controller])
   Subsystem: Toshiba America Info Systems HD Graphics 5500 [1179:f920]
   Subsystem: Toshiba America Info Systems Radeon R7 M260 [1179:f923]
InstallationDate: Installed on 2018-01-07 (17 days ago)
InstallationMedia: Ubuntu 17.10 "Artful Aardvark" - Release amd64 (20171018)
MachineType: TOSHIBA SATELLITE L50-B
ProcCmdline: /usr/lib/xorg/Xorg vt1 -displayfd 3 -auth /run/user/121/gdm/Xauthority -background none -noreset -keeptty -verbose 3
ProcEnviron:
 LANGUAGE=pt:pt_BR:en
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=pt_PT.UTF-8
 SHELL=/bin/false
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.13.0-31-generic root=UUID=7985a6e4-e304-4f48-97b5-9111dc373a81 ro quiet splash vt.handoff=7
Signal: 6
SourcePackage: xorg-server
StacktraceTop:
 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
 __GI_abort () at abort.c:90
 __assert_fail_base (fmt=<optimized out>, assertion=assertion@entry=0x7f549469ed25 "key->initialized", file=file@entry=0x7f549469ed08 "/usr/include/xorg/privates.h", line=line@entry=122, function=function@entry=0x7f549469fd00 "dixGetPrivateAddr") at assert.c:92
 __GI___assert_fail (assertion=0x7f549469ed25 "key->initialized", file=0x7f549469ed08 "/usr/include/xorg/privates.h", line=122, function=0x7f549469fd00 "dixGetPrivateAddr") at assert.c:101
 ?? () from /usr/lib/xorg/modules/drivers/amdgpu_drv.so
Title: Xorg assert failure: Xorg: /usr/include/xorg/privates.h:122: dixGetPrivateAddr: Assertion `key->initialized' failed.
UpgradeStatus: Upgraded to bionic on 2018-01-25 (0 days ago)
UserGroups:

dmi.bios.date: 01/21/2015
dmi.bios.vendor: INSYDE Corp.
dmi.bios.version: 2.10
dmi.board.asset.tag: Type2 - Board Asset Tag
dmi.board.name: Type2 - Board Product Name1
dmi.board.vendor: Type2 - Board Vendor Name1
dmi.board.version: Type2 - Board Version
dmi.chassis.asset.tag: No Asset Tag
dmi.chassis.type: 10
dmi.chassis.vendor: OEM Chassis ManuFacturer
dmi.chassis.version: OEM Chassis Version
dmi.modalias: dmi:bvnINSYDECorp.:bvr2.10:bd01/21/2015:svnTOSHIBA:pnSATELLITEL50-B:pvrPSKTNE-01U011EP:rvnType2-BoardVendorName1:rnType2-BoardProductName1:rvrType2-BoardVersion:cvnOEMChassisManuFacturer:ct10:cvrOEMChassisVersion:
dmi.product.family: INVALID
dmi.product.name: SATELLITE L50-B
dmi.product.version: PSKTNE-01U011EP
dmi.sys.vendor: TOSHIBA
version.compiz: compiz N/A
version.libdrm2: libdrm2 2.4.89-1
version.libgl1-mesa-dri: libgl1-mesa-dri 17.2.4-0ubuntu2
version.libgl1-mesa-glx: libgl1-mesa-glx 17.2.4-0ubuntu2
version.xserver-xorg-core: xserver-xorg-core 2:1.19.5-0ubuntu2
version.xserver-xorg-input-evdev: xserver-xorg-input-evdev N/A
version.xserver-xorg-video-ati: xserver-xorg-video-ati 1:7.10.0-1
version.xserver-xorg-video-intel: xserver-xorg-video-intel N/A
version.xserver-xorg-video-nouveau: xserver-xorg-video-nouveau 1:1.0.15-2

Revision history for this message
Joao Cerveira (jcerveira) wrote :
Revision history for this message
Apport retracing service (apport) wrote :

StacktraceTop:
 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
 __GI_abort () at abort.c:90
 __assert_fail_base (fmt=<optimized out>, assertion=assertion@entry=0x7f549469ed25 "key->initialized", file=file@entry=0x7f549469ed08 "/usr/include/xorg/privates.h", line=line@entry=122, function=function@entry=0x7f549469fd00 <__PRETTY_FUNCTION__.10748> "dixGetPrivateAddr") at assert.c:92
 __GI___assert_fail (assertion=assertion@entry=0x7f549469ed25 "key->initialized", file=file@entry=0x7f549469ed08 "/usr/include/xorg/privates.h", line=line@entry=122, function=function@entry=0x7f549469fd00 <__PRETTY_FUNCTION__.10748> "dixGetPrivateAddr") at assert.c:101
 dixGetPrivateAddr (key=<optimized out>, key=<optimized out>, privates=0x562c7e5c38e0) at /usr/include/xorg/privates.h:122

Revision history for this message
Apport retracing service (apport) wrote : Stacktrace.txt
Revision history for this message
Apport retracing service (apport) wrote : StacktraceSource.txt
Revision history for this message
Apport retracing service (apport) wrote : ThreadStacktrace.txt
Changed in xorg-server (Ubuntu):
importance: Undecided → Medium
tags: removed: need-amd64-retrace
information type: Private → Public
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in xorg-server (Ubuntu):
status: New → Confirmed
Revision history for this message
Daniel van Vugt (vanvugt) wrote :

This is actually the top Xorg crash in 20.04 but a glitch on our the tracking site "errors.ubuntu.com" makes it unclickable and unusable :(

https://errors.ubuntu.com/?release=Ubuntu%2020.04&package=xorg-server&period=year

tags: added: focal
Revision history for this message
Ubuntu QA Website (ubuntuqa) wrote :

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
http://iso.qa.ubuntu.com/qatracker/reports/bugs/1745345

tags: added: iso-testing
Revision history for this message
Norbert (nrbrtx) wrote :

I need to recall that on QEMU/KVM (while using virt-manager the crashes occur with Cirrus, QXL, VGA video drivers, but does not occur with VMVGA video driver) and VirtualBox (tested all versions with default `vboxvideo` driver) the crash is caused by `gstreamer-vaapi` package (see https://packages.ubuntu.com/focal/gstreamer1.0-vaapi). It is a dependency of `ubuntu-restricted-addons` (see https://packages.ubuntu.com/focal/ubuntu-restricted-addons).

So ubuntu-mate-welcome and for example Cheese with Yelp may be fixed by removing this package with

```
sudo apt-get purge gstreamer1.0-vaapi
sudo apt-get autoremove --purge # to remove libgstreamer-plugins-bad1.0-0 (is not a problem)
```

So please carefully analyze the crash dumps for errors in `gstreamer1.0-vaapi` and fix this package.

I'm not sure about the bug inside `xorg-server` codebase.

tags: added: eoan
removed: single-occurrence third-party-packages
Revision history for this message
Norbert (nrbrtx) wrote :

For original problem with Ubuntu MATE Welcome one can use crash file from this link - https://bugs.launchpad.net/ubuntu/+source/xorg-server/+bug/1796437/+attachment/5319138/+files/_usr_lib_xorg_Xorg.0.crash .

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in gstreamer-vaapi (Ubuntu):
status: New → Confirmed
Revision history for this message
Norbert (nrbrtx) wrote : apport information

ProblemType: Bug
ApportVersion: 2.20.11-0ubuntu15
Architecture: amd64
BootLog: Error: [Errno 13] Permission denied: '/var/log/boot.log'
CompizPlugins: No value set for `/apps/compiz-1/general/screen0/options/active_plugins'
CompositorRunning: None
CurrentDesktop: MATE
DistUpgraded: Fresh install
DistroCodename: focal
DistroRelease: Ubuntu 20.04
DistroVariant: ubuntu
GraphicsCard: InnoTek Systemberatung GmbH VirtualBox Graphics Adapter [80ee:beef] (prog-if 00 [VGA controller])
InstallationDate: Installed on 2020-01-19 (1 days ago)
InstallationMedia: Ubuntu-MATE 20.04 LTS "Focal Fossa" - Alpha amd64 (20200119)
Lsusb:
 Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
 Bus 002 Device 002: ID 80ee:0021 VirtualBox USB Tablet
 Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Lsusb-t:
 /: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=ohci-pci/12p, 12M
     |__ Port 1: Dev 2, If 0, Class=Human Interface Device, Driver=usbhid, 12M
 /: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/12p, 480M
Package: xorg-server (not installed)
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.4.0-9-generic root=UUID=a7e811b1-6cbc-4d90-b1c9-8698611249e9 ro quiet splash
ProcVersionSignature: Ubuntu 5.4.0-9.12-generic 5.4.3
Tags: focal ubuntu reproducible
Uname: Linux 5.4.0-9-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm cdrom dip lpadmin lxd plugdev sambashare sudo
_MarkForUpload: True
version.compiz: compiz N/A
version.libdrm2: libdrm2 2.4.100-4
version.libgl1-mesa-dri: libgl1-mesa-dri 19.2.4-1ubuntu1
version.libgl1-mesa-glx: libgl1-mesa-glx N/A
version.xserver-xorg-core: xserver-xorg-core 2:1.20.6-1ubuntu1
version.xserver-xorg-input-evdev: xserver-xorg-input-evdev N/A
version.xserver-xorg-video-ati: xserver-xorg-video-ati 1:19.1.0-1
version.xserver-xorg-video-intel: xserver-xorg-video-intel 2:2.99.917+git20190815-1
version.xserver-xorg-video-nouveau: xserver-xorg-video-nouveau 1:1.0.16-1

tags: added: apport-collected reproducible
Revision history for this message
Norbert (nrbrtx) wrote : CurrentDmesg.txt

apport information

Revision history for this message
Norbert (nrbrtx) wrote : DpkgLog.txt

apport information

Revision history for this message
Norbert (nrbrtx) wrote : LightdmDisplayLog.txt

apport information

Revision history for this message
Norbert (nrbrtx) wrote : LightdmLog.txt

apport information

Revision history for this message
Norbert (nrbrtx) wrote : Lspci.txt

apport information

Revision history for this message
Norbert (nrbrtx) wrote : Lsusb-v.txt

apport information

Revision history for this message
Norbert (nrbrtx) wrote : ProcCpuinfo.txt

apport information

Revision history for this message
Norbert (nrbrtx) wrote : ProcCpuinfoMinimal.txt

apport information

Revision history for this message
Norbert (nrbrtx) wrote : ProcEnviron.txt

apport information

Revision history for this message
Norbert (nrbrtx) wrote : ProcInterrupts.txt

apport information

Revision history for this message
Norbert (nrbrtx) wrote : ProcModules.txt

apport information

Revision history for this message
Norbert (nrbrtx) wrote : UdevDb.txt

apport information

Revision history for this message
Norbert (nrbrtx) wrote : XorgLog.txt

apport information

Revision history for this message
Norbert (nrbrtx) wrote : XorgLogOld.txt

apport information

Revision history for this message
Norbert (nrbrtx) wrote : Xrandr.txt

apport information

Revision history for this message
Norbert (nrbrtx) wrote : xdpyinfo.txt

apport information

Revision history for this message
Norbert (nrbrtx) wrote :
Revision history for this message
Norbert (nrbrtx) wrote :

Crash on VirtualBox with `gstreamer1.0-vaapi` installed also is caused by:
`gnome-books`, `gnome-contacts`, `gnome-control-center`, `gnome-documents`, `gnome-help`, `gnome-music`, `gnome-photo`, `gnome-sound-recorder`, `gnome-subtitles`.

Revision history for this message
Daniel van Vugt (vanvugt) wrote :

Please don't run apport-collect on bugs you don't own. Please also avoid attaching crash files to bugs as we can't process them that way. You should run this command to create a new bug for each crash file:

   ubuntu-bug YOURFILE.crash

no longer affects: gstreamer-vaapi (Ubuntu)
Revision history for this message
Norbert (nrbrtx) wrote :

@vanvugt, could you please elaborate why do you think that `gstreamer-vaapi` is not related to this bug (or really bug 1796437).

I see that installation of the single `gstreamer1.0-vaapi` package make system crash. While its uninstallation removes the crash. Just tested 20200203 MATE ISO.

Do you/anybody-from-Canonical really work on this issue?

Changed in gstreamer-vaapi (Ubuntu):
status: New → Confirmed
Revision history for this message
linex83 (linex83) wrote :

Removing the package "gstreamer1.0-vaapi" solved the issue for me, too.

Revision history for this message
Daniel van Vugt (vanvugt) wrote :

It is not appropriate to have a gstreamer-vaapi task here because there's nothing you can do in that package that is an acceptable fix for Xorg crashing. It's not a fix, it is a workaround.

If you remove "gstreamer1.0-vaapi", then that might be a good workaround for some people. But no possible change to that package is an acceptable fix to stop any other package from crashing Xorg in similar ways in future.

Changed in gstreamer-vaapi (Ubuntu):
status: Confirmed → Invalid
Revision history for this message
intherye (intherye) wrote :

This also happened on a 20.04 Xubuntu 64-bit VM in Qemu/KVM: Xorg crashing at login, next login works.

Revision history for this message
Daniel van Vugt (vanvugt) wrote :

Also tracking (sounds closely related) in:
https://gitlab.freedesktop.org/xorg/xserver/issues/781

Changed in xorg-server:
status: Unknown → New
Revision history for this message
Bill (franksmcb) (franksmcb) wrote :

This does occur with 20200216 ISO with Ubuntu MATE using Virt-manager/QEMU, however this issue does not occur using VirtualBox.

Revision history for this message
Norbert (nrbrtx) wrote :
Download full text (3.1 KiB)

@Bill (franksmcb) (franksmcb)

This is a bit broader. VirtualBox is still affected. I use 5.1 because of 25 VMs inside it.

As I already wrote - see https://bugs.launchpad.net/ubuntu/+source/xorg-server/+bug/1796437/comments/16 :

> Norbert (nrbrtx) wrote on 2020-01-16
>
>Still reproducible on 20200115 and 20200116 builds.
>
>---
>
>Reproducible method 1:
>
>1. boot installation media
>2. select [o] Normal installation, [x] Download updates, [x] Install third-party software
>3. Wait installation to finish
>4. Boot the installed system, wait for Snappy unpacking/initialization/seeding (check from tty1 with `snap list`)
>5. Login to desktop session
>6. Get crash of whole Xorg after login when of Ubuntu MATE Welcome window is starting to appear.
>
>Reproducible method 2:
>
>1. boot installation media
>2. select [o] Normal installation, [ ] Download updates, [ ] Install third-party software
>3. Wait installation to finish
>4. Boot the installed system, wait for Snappy unpacking/initialization/seeding (check from tty1 with `snap list`)
>5. Install Ubuntu restricted extras with `sudo apt-get install ubuntu-restricted-extras` and reboot
>6. Login to desktop session
>7. Get crash of whole Xorg after login when of Ubuntu MATE Welcome window is starting to appear.
>
>Guest settings: choosing video acceleration disabled or 3d video acceleration does not change >behavior. Increasing VRAM from default 16 Mb to 128 Mb (max for my machine) does not help either.
>VirtualBox versions tested:
>
>* 5.1.38-dfsg-0ubuntu1.16.04.3 (from multiverse) - crash,
>* 5.0.40 (from Oracle) - crash,
>* 5.1.38 (from Oracle) - crash,
>* 5.2.36 (from Oracle) - crash,
>* 6.0.16 (from Oracle) - crash,
>* 6.1.2 (from Oracle) - crash.
>
>---
>
>Removing `ubuntu-mate-welcome` from autostart with `sudo rm /etc/xdg/autostart/ubuntu-mate-welcome-autostart.desktop` fixes the login issue. But direct launch of it with `ubuntu-mate-welcome` ends with another crash.

and https://bugs.launchpad.net/ubuntu/+source/xorg-server/+bug/1796437/comments/17 :

> Norbert (nrbrtx) wrote on 2020-01-18
>
>With virt-manager the crashes occur with Cirrus, QXL, VGA video drivers.
>But does not occur with VMVGA video driver.
>Possibly kernel (drm) issue.

and https://bugs.launchpad.net/ubuntu/+source/xorg-server/+bug/1796437/comments/19 :

> Norbert (nrbrtx) wrote on 2020-01-18
>
>I found the real reason for the crashes. The problematic package is the `gstreamer1.0-vaapi` (see https://packages.ubuntu.com/focal/gstreamer1.0-vaapi).
>It is a dependency of `ubuntu-restricted-addons` (see https://packages.ubuntu.com/focal/ubuntu-restricted-addons).
>
>So ubuntu-mate-welcome and for example Cheese may be fixed by removing this package with
>
>```
>sudo apt-get purge gstreamer1.0-vaapi
>sudo apt-get autoremove --purge # to remove libgstreamer-plugins-bad1.0-0 (is not a problem)
>```
>
>So please carefully analyze the crash dumps for errors in `gstreamer1.0-vaapi` and fix this package.

End of quotes.

I would recommend to test any modern VirtualBox (according to https://www.virtualbox.org/wiki/Download_Old_Builds this means >= 5.2) with non-VMVGA video driver to reproduce the issue. And report here the results....

Read more...

Revision history for this message
Bill (franksmcb) (franksmcb) wrote :

This continues to be an issue with Virt-Manager/QEMU with the 20200306 ISO of Ubuntu MATE and the 20200305 ISO of Ubuntu Budgie.

With VirtualBox 6.1.4 the following results occur:
1. Using VMSVGA there is no crash
2. Using VBoxSVGA the crash occurs.

Revision history for this message
Norbert (nrbrtx) wrote :

Still happens with Ubuntu MATE 20.04 LTS (20200313) using VirtualBox 5.1.

Revision history for this message
Bill (franksmcb) (franksmcb) wrote :

Ubuntu MATE 20.04 2020315

QEMU: crash
VirtualBox VBoxVGA crash
VirtualBox VMSVGA: no crash

In this case VirtualBox 6.1.4 (version shipping in 20.04)

Revision history for this message
Gold Star (goldstar611) wrote :

I would like to also mention that Ubuntu 20.04 Daily (not MATE) crashes on Virtualbox and KVM and that, as mentioned previously, running `sudo apt-get purge gstreamer1.0-vaapi` fixes the crash to login issue. Another workaround is to use Ubuntu on Wayland but in Virtualbox no mouse pointer is visible.

Revision history for this message
Norbert (nrbrtx) wrote :

Seeing this on Ubuntu MATE 20.04 LTS beta 20200402:

> fresh install using `[x] Install third-party software for graphics and Wi-Fi hardware and additional media formats` on VirtualBox 5.1 leads to crashes on VirtualBox 5.1.

Changed in gstreamer-vaapi (Ubuntu):
status: Invalid → Confirmed
Changed in gstreamer-vaapi (Ubuntu):
status: Confirmed → Invalid
Revision history for this message
Daniel van Vugt (vanvugt) wrote :
no longer affects: gstreamer-vaapi (Ubuntu)
Changed in xorg-server (Ubuntu):
importance: Medium → High
tags: added: champagne rls-ff-incoming
description: updated
Revision history for this message
Daniel van Vugt (vanvugt) wrote :

Looks like we might need to split this into multiple bugs somehow, because the same assertion has kept failing over the years and multiple fixes have already been applied:

Fixed in 1.20.7:
https://gitlab.freedesktop.org/xorg/xserver/-/commit/cfc5e5040c934

Fixed in 1.20.1:
https://gitlab.freedesktop.org/xorg/xserver/-/commit/de40a55235614

Revision history for this message
Daniel van Vugt (vanvugt) wrote :
Revision history for this message
Norbert (nrbrtx) wrote :

For me the main question is - how installation of `gstreamer1.0-vaapi` leads to crash?
And why purging of `gstreamer1.0-vaapi` helps to avoid the crash.

@Daniel van Vugt (vanvugt) did you tried to reproduce this crash?

Revision history for this message
Mikko Rantalainen (mira) wrote :

It seems to me that fix https://gitlab.freedesktop.org/xorg/xserver/-/commit/cfc5e5040c934 is the correct one. Is it possible to reproduce the crash with that one patch applied even without all the other proposed fixes?

Revision history for this message
Daniel van Vugt (vanvugt) wrote :

Norbert,

VAAPI may trigger this crash because it involves calling low-level driver functions within Xorg. That does not make it a problem with any VAAPI packages, but is still a bug in Xorg. Or multiple bugs in Xorg.

Revision history for this message
Sebastien Bacher (seb128) wrote :

It's not clear what issue are still remaining on focal, might we worth having a new report for that serie to not confuse with past problems that got resolved. Tagging as rls-ff-notfixing meanwhile because it's not clear how much that's still a problem on focal

tags: added: rls-ff-notfixing
removed: rls-ff-incoming
Revision history for this message
Daniel van Vugt (vanvugt) wrote :

What we should actually do is to distinguish the bugs by call stack, as that will be a little different for each flavour of this crash.

Unfortunately several of the most recent reports (focal) have no call stack at all. So separating them is not an option yet.

Revision history for this message
Daniel van Vugt (vanvugt) wrote :

The (new??) problem with focal has moved to bug 1861609. So just about everyone should probably use bug 1861609 now.

tags: removed: champagne
Revision history for this message
Daniel van Vugt (vanvugt) wrote :

Mikko,

Re comment #49, we already have that fix (see comment #46).

Norbert (nrbrtx)
Changed in xorg-server (Ubuntu):
status: Confirmed → Fix Released
Changed in xorg-server (Ubuntu):
status: Fix Released → Confirmed
Norbert (nrbrtx)
tags: removed: eoan
Changed in xorg-server:
status: New → Fix Released
no longer affects: xorg-server
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.