Make duplicate signature more specific for DBusException and OSError

Bug #989819 reported by Sebastien Bacher on 2012-04-27
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Daisy
High
Unassigned
apport (Ubuntu)
High
Martin Pitt
Trusty
High
Martin Pitt
Xenial
High
Martin Pitt

Bug Description

Looking through some db bug examples and saw some incorrect bug associations, one example

"Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/LanguageSelector/gtk/GtkLanguageSelector.py", line 62, in wrapper
    res = f(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/LanguageSelector/gtk/GtkLanguageSelector.py", line 74, in wrapper
    res = f(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/LanguageSelector/gtk/GtkLanguageSelector.py", line 1090, in on_combobox_locale_chooser_changed
    self.writeUserFormats()
  File "/usr/lib/python2.7/dist-packages/LanguageSelector/gtk/GtkLanguageSelector.py", line 775, in writeUserFormats
    self.writeUserFormatsSetting(userFormats=code)
  File "/usr/lib/python2.7/dist-packages/LanguageSelector/LanguageSelector.py", line 71, in writeUserFormatsSetting
    iface.SetFormatsLocale(macr['SYSLOCALE'])
  File "/usr/lib/python2.7/dist-packages/dbus/proxies.py", line 70, in __call__
    return self._proxy_method(*args, **keywords)
  File "/usr/lib/python2.7/dist-packages/dbus/proxies.py", line 145, in __call__
    **keywords)
  File "/usr/lib/python2.7/dist-packages/dbus/connection.py", line 651, in call_blocking
    message, timeout)
DBusException: org.freedesktop.Accounts.Error.PermissionDenied: Not authorized"

has been associated to https://bugs.launchpad.net/ubuntu/+source/accountsservice/+bug/930785

"Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/LanguageSelector/gtk/GtkLanguageSelector.py", line 62, in wrapper
    res = f(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/LanguageSelector/gtk/GtkLanguageSelector.py", line 74, in wrapper
    res = f(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/LanguageSelector/gtk/GtkLanguageSelector.py", line 1081, in on_combobox_locale_chooser_changed
    self.writeUserFormats()
  File "/usr/lib/python2.7/dist-packages/LanguageSelector/gtk/GtkLanguageSelector.py", line 775, in writeUserFormats
    self.writeUserFormatsSetting(userFormats=code)
  File "/usr/lib/python2.7/dist-packages/LanguageSelector/LanguageSelector.py", line 71, in writeUserFormatsSetting
    iface.SetFormatsLocale(macr['SYSLOCALE'])
  File "/usr/lib/python2.7/dist-packages/dbus/proxies.py", line 70, in __call__
    return self._proxy_method(*args, **keywords)
  File "/usr/lib/python2.7/dist-packages/dbus/proxies.py", line 145, in __call__
    **keywords)
  File "/usr/lib/python2.7/dist-packages/dbus/connection.py", line 651, in call_blocking
    message, timeout)
DBusException: org.freedesktop.Accounts.Error.Failed: 'bg_BG.UTF-8' is not a valid locale name"

The "functions signature" is the same, the exceptions are different though, it should probably consider that as part of the signature

SRU TEST CASE:
--------------
 * Modify /usr/bin/pybuild to append these two lines right after "def main(cfg):"

    raise OSError(99, 'some OS error')

 * Now running "pybuild" will provoke an OSError, and you should get a /var/crash/_usr_share_dh-python_pybuild.1000.crash

 * With current apport, the crash signature looks like this, i. e. it does not include the errno (99):

  $ python3 -c 'import apport; r = apport.Report(); r.load(open("/var/crash/_usr_share_dh-python_pybuild.1000.crash", "rb")); print(r.crash_signature())'
/usr/share/dh-python/pybuild:OSError:/usr/share/dh-python/pybuild@449:main

 * With this apport update (you need to rm the .crash and re-generate it), it will look like this instead:

  /usr/share/dh-python/pybuild:OSError(99):/usr/share/dh-python/pybuild@449:main

 i. e. the errno 99 will be used to disambiguate/duplicate the crash.

This works similarly for DBusErrors, but this is much harder to test manually. The backported patch includes automatic test cases which cover the DBusError cases, though.

Related branches

Evan (ev) on 2012-05-16
Changed in whoopsie-daisy (Ubuntu):
assignee: nobody → Evan Dandrea (ev)
importance: Undecided → High
milestone: none → ubuntu-12.10
Evan (ev) on 2012-05-18
Changed in daisy:
assignee: nobody → Evan Dandrea (ev)
importance: Undecided → High
no longer affects: whoopsie-daisy (Ubuntu)
Brian Murray (brian-murray) wrote :

I believe the crash signature used by daisy is actually just the crash signature apport provides it. So the fix for this bug would need to happen in apport and its crash_signature function from report.py.

Brian Murray (brian-murray) wrote :

In bug 772083 we can see that the retracer used the incomplete signature to mark it as a duplicate of bug 781360.

Changed in apport (Ubuntu):
status: New → Triaged
importance: Undecided → High
Brian Murray (brian-murray) wrote :

We can see how the crash signature is created by apport for python tracebacks here: http://bazaar.launchpad.net/~ubuntu-core-dev/ubuntu/quantal/apport/ubuntu/view/head:/apport/report.py#L1206.

The problematic bit follows:

return self['ExecutablePath'] + ':' + trace[-1].split(':')[0] + sig

Here were just using the first part of the last line of the trace to create the signature. Considering just the trace[-1].split(':')[0] portion we have the same partial signature for bugs 772083 781360 - 'SystemError'. However, if we expand this out a bit we have:

772083: SystemError: E:Encountered a section with no Package: header, E:Problem with MergeList /var/lib/apt/lists/archive.canonical.com_ubuntu_dists_natty_partner_binary-i386_Packages, E:The package lists or status file could not be parsed or opened.
781630: SystemError: E:Write error - write (28: No space left on device), E:IO Error saving source cache, E:The package lists or status file could not be parsed or opened.

which is quite different. For the partial signature we could use all of trace[-1] or a portion there of. I have code that will download all the tracebacks for a package and compare their old partial signatures to a new one.

Evan (ev) on 2012-11-28
Changed in daisy:
status: New → Confirmed
Evan (ev) on 2013-05-30
Changed in daisy:
assignee: Evan Dandrea (ev) → nobody
Martin Pitt (pitti) wrote :

Due to the nature of Python exceptions this indeed can only ever be a heuristic. Some Python errors are already very specific, and we *don't* want their message be part of the signature. For example, we want to treat PermissionError('/home/fred/foo') and PermissionError('/home/joe/bar') as one and the same bug. Likewise, in the example in the description we would want "org.freedesktop.Accounts.Error.Failed: 'bg_BG.UTF-8' is not a valid locale name" to be treated exactly the same as any other locale name. Exception messages also often have localized or otherwise user/system specific context in them.

For DBusError the situation is still relatively easy -- this is a well-known one from dbus-python which actually wraps a "real" exception on the server. For the duplicate signature we certainly should unwrap that inner exception, so that we end up with something like "DBusException:org.freedesktop.Accounts.Error.PermissionDenied".

The other case of pyapt's SystemErrors are harder to dissect as there is no structure in the exceptions and their messages. So the only real options that we have are to keep the status quo (don't use the message at all) or take it wholesale. But I think the latter wouldn't be helpful as it would consider too many things as "not the same bug". One could go further and just take the last word, but I think that would be completely arbitrary and still not help (translations/not capturing the important part).

So I propose to do one step at a time and unwrap DBusExceptions for now.

Martin Pitt (pitti) wrote :

We should also dissect OSErrors and append errno. Python 3 already does that by having specific subclasses such as PermissionError for EACCES, but this is still useful for Python 2 or if we run into an errno which doesn't have a subclass.

Changed in apport (Ubuntu):
assignee: nobody → Martin Pitt (pitti)
status: Triaged → In Progress
summary: - the signatures match code should probably consider the exception for
- python errors
+ Make duplicate signature more specific for DBusException and OSError
Martin Pitt (pitti) wrote :

Implemented in apport trunk r3066.

Changed in apport (Ubuntu):
status: In Progress → Fix Committed
Martin Pitt (pitti) wrote :

This should also be SRUed.

Changed in apport (Ubuntu Trusty):
assignee: nobody → Martin Pitt (pitti)
status: New → Triaged
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package apport - 2.20.1-0ubuntu1

---------------
apport (2.20.1-0ubuntu1) xenial; urgency=medium

  * New upstream release. Changes since our previous snapshot:
    - crash-digger: Untag bugs which cannot be retraced instead of stopping
      crash-digger. This led to too many pointless manual restarts on broken bug
      reports.
    * Disambiguate overly generic Python exceptions in duplicate signature
      computation: dbus-glib's DBusException wraps a "real" server-side
      exception, so add the class of that to disambiguate different crashes;
      for OSError that is not a known subclass like FileNotFoundError, add the
      errno. (LP: #989819)

 -- Martin Pitt <email address hidden> Thu, 31 Mar 2016 16:16:37 +0200

Changed in apport (Ubuntu Xenial):
status: Fix Committed → Fix Released
Martin Pitt (pitti) wrote :

Trusty backport of http://bazaar.launchpad.net/~apport-hackers/apport/trunk/revision/3066 uploaded to the SRU review queue.

description: updated
Changed in apport (Ubuntu Trusty):
status: Triaged → In Progress

Hello Sebastien, or anyone else affected,

Accepted apport into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apport/2.14.1-0ubuntu3.20 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in apport (Ubuntu Trusty):
status: In Progress → Fix Committed
tags: added: verification-needed
Brian Murray (brian-murray) wrote :

bdmurray@clean-trusty-amd64:~$ apt-cache policy apport
apport:
  Installed: 2.14.1-0ubuntu3.20
  Candidate: 2.14.1-0ubuntu3.20
  Version table:
 *** 2.14.1-0ubuntu3.20 0
        500 http://ubuntu.osuosl.org/ubuntu/ trusty-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     2.14.1-0ubuntu3.19 0
        500 http://ubuntu.osuosl.org/ubuntu/ trusty-updates/main amd64 Packages
     2.14.1-0ubuntu3.18 0
        500 http://ubuntu.osuosl.org/ubuntu/ trusty-security/main amd64 Packages
     2.14.1-0ubuntu3 0
        500 http://ubuntu.osuosl.org/ubuntu/ trusty/main amd64 Packages
bdmurray@clean-trusty-amd64:~$ python3 -c 'import apport; r = apport.Report(); r.load(open("/var/crash/_usr_share_dh-python_pybuild.1000.crash", "rb")); print(r.crash_signature())'
/usr/share/dh-python/pybuild:OSError(99):/usr/share/dh-python/pybuild@449:main

tags: added: verification-done
removed: verification-needed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package apport - 2.14.1-0ubuntu3.20

---------------
apport (2.14.1-0ubuntu3.20) trusty-proposed; urgency=medium

  * Disambiguate overly generic Python exceptions in duplicate signature
    computation: dbus-glib's DBusException wraps a "real" server-side
    exception, so add the class of that to disambiguate different crashes;
    for OSError that is not a known subclass like FileNotFoundError, add the
    errno. (LP: #989819)

 -- Martin Pitt <email address hidden> Fri, 01 Apr 2016 16:27:39 +0200

Changed in apport (Ubuntu Trusty):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for apport has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Changed in apport (Ubuntu Trusty):
importance: Undecided → High
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers