do not require apt lists to be pre-installed

[Pat McGowan]

On the phone images the space required for the lists in /var/lib/apt/lists is excessive (around 70MB) for the device targets we have. The partition it currently resides in did not include this requirement when their size was specified.

Removing those prevents one from using apport-cli to report bugs about packages that are from the -updates pocket. This is due to the fact that apport will not allow one to report bugs about packages which it cannot determine are from the official archive or a native origin PPA.

Possible solutions include generating the list dynamically as part of the writable partition or including a smaller simpler manifest.

This report was inspired by bug #1471903

[Oliver Grawert (ogra)]

another option would be to simply make apport use the manifest files on cdimage or launchpad or provide another way of handling this data remotely ... after all whoopsie and apport rely on being able to send the reports to a server so a usable network connection exists at some point.
this would also help for future use of apport on snappy personal or snappy based phone images where we have neither apt nor dpkg available at all.

[Martin Pitt (pitti)]

Hey Pat! Can you please elaborate what the actual effect of this is? AFAIK apport needs the apt lists only to determine/validate the origin of a package (which we can/should work around on a system image indeed), but *not* to determine the Package:/Dependencies: fields. I. e. mapping files to packages and determining package versions should only require the dpkg database in /var/lib/dpkg/, *not* apt.

[Martin Pitt (pitti)]

Ah, bug 1471903 has some more info, I updated the description. So it indeed is about the origin detection.

[Oliver Grawert (ogra)]

note that we don't have /var/lib/dpkg on snappy either, it would be good to find a solution we can use on snappy personal and snappy phone in the future...

[Martin Pitt (pitti)]

Committed to wily branch: http://bazaar.launchpad.net/~ubuntu-core-dev/ubuntu/wily/apport/ubuntu/revision/2449

[Martin Pitt (pitti)]

This fix will now get along with a completely empty /var/lib/apt/lists/. Note that the directory /var/lib/apt/lists/partial/ itself still needs to be present (otherwise apt/python-apt will try to re-create it), but it does not need to contain any files any more. This should address this immediate bug and is unintrusive. I uploaded it to wily now and also uploaded a backport to the overlay PPA (2.17.2-0ubuntu1.3touch1). I tested this on mako on the rc-proposed/ubuntu channel, image #228.

This does not yet address snappy as completely dropping the dpkg database will lose any association of a file to a package, and thus also disable package hooks and the like. It's conceivable to write a completely new packaging backend for snappy which doesn't use dpkg/apt any more. But this shouldn't just be a drive-by bug report and is a lot of work:

  * Snappy team should discuss what kind of bug reports and crashes they expect for the snappy OS itself. I suppose this should not apply to third-party snaps but only the Ubuntu-provided minimal OS (or rather big desktop in case of personal). E. g. we supposedly don't need any kind of "Package:" error, we might not want the normal Ubuntu package hooks. Also, do you want the usual Package:/Dependencies: information, or something simpler (just the image version, etc.)?

  * Snappy team should discuss what kind of UI they want: Should crash reports be always sent non-interactively, phone style? Or should there be some UI to address privacy issues?

 * Is apport even still the right tool for this, or overly complex for Snappy and you actually just care for crashes? (There are alternatives like http://www.freedesktop.org/software/systemd/man/coredumpctl.html)

 * If the above discussion decides that the snappy team still wants the features of apport, we need a new apport packaging backend which can implement the API in http://bazaar.launchpad.net/~apport-hackers/apport/trunk/view/head:/apport/packaging.py without referring to dpkg/apt. I. e. this would mostly mean to provide an online service to map a file to a package, a file and a system image version to a package version, map a package of a particular version/system image to its dependencies, etc.

I. e. the snappy side of this requires some discussions and a proper blueprint. This shouldn't block this immediate phone issue which is apparently rather urgent, so I went ahead with the above simple and focussed fix.

[Launchpad Janitor (janitor)]

This bug was fixed in the package apport - 2.18-0ubuntu9

---------------
apport (2.18-0ubuntu9) wily; urgency=medium

  * Revert changes to data/package_hook to include the package version. This
    just hides the problem that somewhere during whoopsie add_package_info()
    is not called. (See LP #1485787)
  * packaging-apt-dpkg.py, is_distro_package(): If there is no origin and
    /etc/system-image/channel.ini exists, assume the package is from a
    read-only system image and accept it as distro package. With this we don't
    need /var/lib/apt/lists/ indexes any more just to confirm the origin.
    (LP: #1489410)
  * Merge fixes from trunk:
    - whoopsie-upload-all: Intercept OSError too (e. g. "No space left on
      device"). (LP: #1476258)
    - apport-retrace: Only consider the file name of a source file, not its
      path; the latter often contains parts like "../" or directories which are
      specific to a build machine. This fixes most broken StacktraceSource
      results. (LP: #1462491)

 -- Martin Pitt <email address hidden> Mon, 31 Aug 2015 11:35:54 +0200