not all packages from PPAs have '[origin: ' in Package section

[Brian Murray]

Test Case
---------
1) Install galicaster package from wiki.teltek.es (or any other random deb file will do)
2) Run the executable
3) Find the pid of the program and kill it with signal 11
4) Inspect the .crash file (after it is finished being written) and observe the "Package: " key doesn't not contain "[origin: unknown]"

With the version of apport from -proposed it will contain information about it being from an unknown origin.

Original Description
--------------------
I was querying for columns in the counters column family starting with 'Ubuntu ' as I was looking for the counts of bugs reported about packages. In the column family I discovered 'Ubuntu 12.10:indicator-stickynotes' and was having trouble finding errors for that package because it is from a PPA. Given that the package is from a PPA it shouldn't be appearing in the counters column family at all though.

From daisy/submit.py:

    package = data.get('Package', '')
    src_package = data.get('SourcePackage', '')
    problem_type = data.get('ProblemType', '')
    third_party = False
    if '[origin:' in package:
        third_party = True

Then before running update_release_pkg_counter we check for 'if not third_party'. Looking at least some of the oops for indicator-stickynotes all of them had '[origin:' in the Package key.

[Brian Murray (brian-murray)]

Ah, here is one without '[origin:' in the Package line:

http://errors.ubuntu.com/oops/2efdce4c-726c-11e2-b91a-e4115b0f8a4a
http://errors.ubuntu.com/oops/0938e0fa-6efe-11e2-ad63-2c768aafd08c

[Brian Murray (brian-murray)]

So it may have something to do with apport or whoopsie.

[Brian Murray (brian-murray)]

Looking at the crashes in the bucket at http://errors.ubuntu.com/%2Fopt%2Fextras.ubuntu.com%2Findicator-stickynotes%2Findicator-stickynotes.py%3ATypeError%3Aupdate_color%3Afunction I found the following apport versions:

2.6.1-0ubuntu3: {'ppa': 2, 'not ppa': 0}
2.6.1-0ubuntu6: {'ppa': 23, 'not ppa': 18}
2.6.1-0ubuntu7: {'ppa': 1, 'not ppa': 0}
2.6.1-0ubuntu9: {'ppa': 42, 'not ppa': 2}
2.6.1-0ubuntu10: {'ppa': 0, 'not ppa': 1}

There is no clear pattern here other than ubuntu6 being wrong more often than not. Additionally, looking at the apport changelog there are no changes to the code that sets the origin.

[Brian Murray (brian-murray)]

My previous query was incomplete:

2.6.1-0ubuntu3: {'ppa': 11, 'not ppa': 2}
2.6.1-0ubuntu6: {'ppa': 64, 'not ppa': 42}
2.6.1-0ubuntu7: {'ppa': 2, 'not ppa': 1}
2.6.1-0ubuntu9: {'ppa': 110, 'not ppa': 16}
2.6.1-0ubuntu10: {'ppa': 5, 'not ppa': 2}

but regardless the results are not much different.

[Evan (ev)]

Apport checks to see if a package is from a third-party origin so that it can not send it to Launchpad. It does this through setting the UnreportableReason field when is_distro_package returns False, then checking for it in check_unreportable. This latter check happens after a .upload file is created for the report (and whoopsie doesn't send the UnreportableReason field - which is probably a bug, it would be nice to increment counters for these to see what error states people are hitting in apport), and so this functionality wont affect whoopsie sending the report to daisy.

That leaves us with the Package field, and determining the conditions under which it adds '[origin:'. One possibility from looking at the code is that people are manually downloading the deb and installing it. Apt wouldn't have a record for it, and wouldn't add an origin field. Apport in turn wouldn't set the '[origin:' in the report file (apport/report.py:251) and daisy in turn would interpret that as a native package (because a native package doesn't have '[origin:' set either).

There's at least one place referring to it by the deb rather than the PPA:
http://www.webupd8.org/2012/11/pin-notes-on-your-desktop-with.html

So I think what we really want here is the is_distro_package check, not the '[origin:' check a level down. Unfortunately the only place we effectively have that in the report file right now is the UnreportableReason field, which can contain the translated string:
_('This is not an official %s package. Please remove any third party package and try again.')

Perhaps the solution is to add a new field here? I had created an InvalidMachineReason field that was to be a machine-parseable version of the UnreportableReason field in my merge proposal for the grouped reports dialog:

https://code.launchpad.net/~ev/apport/grouped_reports/+merge/144591

Perhaps something along those lines? Best to confer with Martin.

As an aside, do we care if the libraries used come from a third-party? Right now in daisy/submit.py we're checking if just the package comes from a third-party before updating the release:package row in the Counters CF, but we're checking if either the package or any of its dependencies are from a third-party before updating the BucketMetadata CF (where FirstSeen, LastSeen, and Source live).

Thanks for investigating this, Brian.

[Brian Murray (brian-murray)]

Perhaps we could have apport set '[origin: unknown]' for debs that people have downloaded and installed.

[Martin Pitt (pitti)]

Right, usually Apport doesn't report problems for non-distro packages (and I still think we shouldn't send them to daisy as well, but let's agree to disagree here). To make daisy's job easier with this, we can certainly add [origin: unknown].

[Martin Pitt (pitti)]

Fixed in trunk r2607.

[Launchpad Janitor (janitor)]

This bug was fixed in the package apport - 2.9.2-0ubuntu1

---------------
apport (2.9.2-0ubuntu1) raring; urgency=low

  * New upstream release:
    - report.py, add_package_info(): Add "[origin: unknown]" tag to
      Package/Dependencies fields for non-distro package whose origin cannot
      be determined. (LP: #1148116)
    - Adjust kernel_crashdump to the format kdump-tools produces. Thanks Louis
      Bouchard.
    - Write core dumps on SIGQUIT if ulimit allows. Thanks Graeme Hewson.
      (LP: #1153662)
  * debian/apport.upstart: Also check for kdump-tools-style kernel crash
    reports for calling kernel_crashdump. Thanks Louis Bouchard.
 -- Martin Pitt <email address hidden> Tue, 19 Mar 2013 11:48:27 +0100

[Brian Murray (brian-murray)]

I noticed that the Error Tracker retracers experience many retrace failures from packages that are not official Ubuntu packages, for example we asked for core files, and failed to retrace, for all of the following crashes.

OOPS Package Release
c4e3cf3e-5db4-11e5-a4ce-fa163e4aaad4 phoneflashtool 12.04
a29e7956-5d96-11e5-aa63-fa163e4ccdf2 rstudio 12.04
dd29fcd4-5ec8-11e5-940c-fa163e22e467 imule 12.04
ffa4c758-5d92-11e5-afe2-fa163e78b027 ultimate-player 12.04
0a01a590-5da2-11e5-aaef-fa163e339c81 sogoupinyin 12.04
ed60591e-5ddc-11e5-974e-fa163e707a72 galicaster 12.04 (modified conf file?)

We shouldn't be asking for core files for these crashes because these packages aren't from Ubuntu, but we are because Precise does not contain the fix for this bug.

[Brian Murray (brian-murray)]

Hello Brian, or anyone else affected,

Accepted apport into precise-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apport/2.0.1-0ubuntu17.11 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

bdmurray@clean-precise-amd64:~$ apt-cache policy apport
apport:
  Installed: 2.0.1-0ubuntu17.11
  Candidate: 2.0.1-0ubuntu17.11
  Version table:
 *** 2.0.1-0ubuntu17.11 0
        500 http://ubuntu.osuosl.org/ubuntu/ precise-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     2.0.1-0ubuntu17.10 0
        500 http://ubuntu.osuosl.org/ubuntu/ precise-updates/main amd64 Packages
        500 http://ubuntu.osuosl.org/ubuntu/ precise-security/main amd64 Packages
     2.0.1-0ubuntu5 0
        500 http://ubuntu.osuosl.org/ubuntu/ precise/main amd64 Packages
bdmurray@clean-precise-amd64:~$ grep Package /var/crash/_usr_share_galicaster_run_galicaster.py.1000.crash
Package: galicaster 1.4.1 [origin: unknown]
PackageArchitecture: all
SourcePackage: galicaster

This looks good to me.

[Launchpad Janitor (janitor)]

This bug was fixed in the package apport - 2.0.1-0ubuntu17.11

---------------
apport (2.0.1-0ubuntu17.11) precise-proposed; urgency=medium

  * report.py, add_package_info(): Add "[origin: unknown]" tag to
    Package/Dependencies fields for a package whose origin cannot
    be determined. (LP: #1148116)

 -- Brian Murray <email address hidden> Thu, 24 Sep 2015 12:48:19 -0700

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for apport has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.