-updates, -security missing from apt lists

[Brian Murray]

As indicated by Robie in bug 1467830 vivid-updates is missing from /var/lib/apt/lists on the phablet images for Ubuntu 15.04. This prevents one from using apport-cli to report bugs about packages that are from the -updates pocket. This is due to the fact that apport will not allow one to report bugs about packages which it cannot determine are from the official archive or a native origin PPA.

[Brian Murray (brian-murray)]

phablet@ubuntu-phablet:~$ ls /var/lib/apt/lists/
lock
partial
ports.ubuntu.com_ubuntu-ports_dists_vivid_main_binary-armhf_Packages
ports.ubuntu.com_ubuntu-ports_dists_vivid_main_i18n_Translation-en
ports.ubuntu.com_ubuntu-ports_dists_vivid_Release
ports.ubuntu.com_ubuntu-ports_dists_vivid_Release.gpg
ports.ubuntu.com_ubuntu-ports_dists_vivid_restricted_binary-armhf_Packages
ports.ubuntu.com_ubuntu-ports_dists_vivid_restricted_i18n_Translation-en
ports.ubuntu.com_ubuntu-ports_dists_vivid_universe_binary-armhf_Packages
ports.ubuntu.com_ubuntu-ports_dists_vivid_universe_i18n_Translation-en
ppa.launchpad.net_ci-train-ppa-service_stable-phone-overlay_ubuntu_dists_vivid_main_binary-armhf_Packages
ppa.launchpad.net_ci-train-ppa-service_stable-phone-overlay_ubuntu_dists_vivid_main_i18n_Translation-en
ppa.launchpad.net_ci-train-ppa-service_stable-phone-overlay_ubuntu_dists_vivid_Release
ppa.launchpad.net_ci-train-ppa-service_stable-phone-overlay_ubuntu_dists_vivid_Release.gpg

It looks like -security is also missing.

[Brian Murray (brian-murray)]

This also has the effect of identifying packages in Dependencies (in an apport report) as being from an unknown origin, even though they are from the Ubuntu archive. Here's a portion of Dependencies from a crash report from Ubuntu 15.04:

libcryptsetup4 2:1.6.1-1ubuntu7
libcups2 2.0.2-1ubuntu3.1 [origin: unknown]
libcurl3 7.38.0-3ubuntu2.2 [origin: unknown]
libcurl3-gnutls 7.38.0-3ubuntu2.2 [origin: unknown]
libdatrie1 0.2.8-1
libdb5.3 5.3.28-9

Example crash:

https://errors.ubuntu.com/oops/9b051a18-23cd-11e5-a99b-fa163e75317b

[Łukasz Zemczak (sil2100)]

Makes sense, will be working on this shortly. We are also missing an entry for the overlay-ppa in apt lists for our rc-proposed images.

[Brian Murray (brian-murray)]

Could you elaborate on your last comment regarding the missing entry in apt lists? Where specifically will you be adding the entry?

[Łukasz Zemczak (sil2100)]

@Brian: I need to add the overlay entries to the sources.list. Since those are visible in apt/lists, but not there. In the same time I'm also trying to get -security and -updates added to apt/lists correctly.

I'm still learning my way around all this, but I see that in fact it's live-build that's removing those entries. lb_chroot_archives clears out updates, security and proposed from that for ubuntu builds by default. I guess I can somehow work around that, so no worries.

[Brian Murray (brian-murray)]

On Wed, Jul 29, 2015 at 04:59:36PM -0000, Łukasz Zemczak wrote:
> @Brian: I need to add the overlay entries to the sources.list. Since
> those are visible in apt/lists, but not there. In the same time I'm also
> trying to get -security and -updates added to apt/lists correctly.
>
> I'm still learning my way around all this, but I see that in fact it's
> live-build that's removing those entries. lb_chroot_archives clears out
> updates, security and proposed from that for ubuntu builds by default. I
> guess I can somehow work around that, so no worries.

That seems like a bug to me, as we would run into similar situations
(not being able to use ubuntu-bug with a package from any of those
pockets) from point release media like 14.04.3.

--
Brian Murray

[Łukasz Zemczak (sil2100)]

I think so to, but I'll try to find the rationale for that in history. If there's no real rational reason for this happening, I'll propose changing that to only removing *-proposed_*.

[Łukasz Zemczak (sil2100)]

I prepared a patch to live-build that basically modified the patch to not remove -security_ and -updates_ from the apt lists as a proposition. I think this is a sane idea as, in theory, both security and updates pockets should generally be pretty empty when we build images and never actually have too many packages. While this gives us a lot of positives: both for ubuntu-touch and for the installer/live images themselves.

[Brian Murray (brian-murray)]

@Łukasz - Is somebody reviewing your debdiff?

[Łukasz Zemczak (sil2100)]

Adam recommended actually removing the patch altogether, which I agree would make sense as well. If, of course, no one disapproves by saying we'll get too fat by including lists for updates, security and proposed at once - which I suppose shouldn't be a problem.

[Łukasz Zemczak (sil2100)]

The debdiff.

[Oliver Grawert (ogra)]

how doe it help to keep outdated readonly lists around apart from wasting a lot of space ? this patch must stay, it is pretty useless to have lists shipped in the readonly images on the phone that were only generated at build time of the image and never get updated. how about we make the dir writable and have apport-cli update the lists when a user the first report or some such (you have to run it manually via a developer mode console or the terminal anyway)

[Brian Murray (brian-murray)]

It helps because we will never ask for core files from crashes from packages for which the package origin is unknown. If the apt list is outdated then the package field in the report will look like this:

Package: coreutils 8.23-3ubuntu1 [origin: unknown]

The code in daisy that asks for core files includes the following:

def retraceable_package(package):
    if not "[origin: " in package:
        return True
    elif "[origin: Ubuntu RTM]" in package:
        return True
    elif "[origin: LP-PPA-ci-train-ppa-service" in package:
        return True
    else:
        return False

This helps prevent us for asking for core files (that would fail to retrace) for every random application that people install. However, at the same time if people are running a version of apport that had an SRU in vivid and that makes into to the phone and it crashes on armhf, then we will never receive core files for that crash. I think that's a pretty good reason to have the lists updated.

[Oliver Grawert (ogra)]

this is inside an armhf chroot on wily with -updates, -security and -proposed enabled (note that is not including the overlay PPA indeed, this would add another few MB):

root@localhost:/# du -hcs /var/lib/apt/lists/
81M /var/lib/apt/lists/
81M total
root@localhost:/# tar czf sources.lists.tgz /var/lib/apt/lists/
root@localhost:/# ls -lh sources.lists.tgz
-rw-r--r-- 1 root root 20M Aug 15 10:37 sources.lists.tgz

our tarball size is limited to the size of the cache partition, all three tarballs (rootfs, device, customization) need to fit into it at the same time for the update to work, someone needs to check if this is still the case, afaik we do not have much wiggle room and 20MB are quite a lot.

on the readonly rootfs partition we have between 200 and 300MB free space for allowing to make the device writable and install debug, autopilot or test packages from a silo, cutting off 89MB+ is quite a big task here and we can not re-partition the devices.

[Brian Murray (brian-murray)]

Did you mean a vivid chroot with -updates, -security and -proposed? Regardless, should -proposed be included as those packages wouldn't be installed on the devices. Additionally, did your sources.list file also include deb-src lines? Removing that would reduce the size of the files in /var/lib/apt/lists. Finally, don't we want to look at the difference between the existing size of /var/lib/apt/lists/ and one which contains -updates and -security? Doing that in a vivid chroot, although on amd64, I see the following:

(vivid-amd64)root@impulse:/home/bdmurray# ls -lh lists-no-updates-security.tar.gz
-rw-r--r-- 1 root root 17M Aug 17 13:37 lists-no-updates-security.tar.gz
(vivid-amd64)root@impulse:/home/bdmurray# ls -lh lists-with-updates-security.tar.gz
-rw-r--r-- 1 root root 18M Aug 17 13:36 lists-with-updates-security.tar.gz

That is only a difference of 1M unless I'm missing something.

[Oliver Grawert (ogra)]

it was a wily chroot (I didnt have a vivid one at hand) but that should only vary minimally.

note that we force deb-src for all entries (since up to now we could and it eases development a lot, i guess if you really want to ship the pre-generated lists, we'd have to reverse this decision (and check if there are any developer tools that make use of these entries))

while we dont enable -proposed, we do enable the overlay PPA (including source as above), which should bring roughly a similar amount of package entries in as -proposed does (a growing number somewhere between 100 and 200).

i think with the current sources.list setup that is shipped my numbers are not far off.

could we not consider some other solution inside apport ?
like making it use a simple manifest file instead of apt to determine if a package in the readonly image is valid. after all we dont officially support making the system writable and using apt so the manifest generated at build time will have accurate data for our use cases and is only a few kilobyte big.

[Łukasz Zemczak (sil2100)]

I think the solution of removing the patch is still ok since as Brian mentioned, -updates and -security enabled only increases the size slightly. The overlay PPA in apt lists is not related to this bug as we already have those in our read-only images. There was no patch that would remove those, so this change would not affect this. Sure, we can think of a better way later to remove the overlay apt lists or make those not required, but I suppose in the meantime removing the patch that's deleting -updates and -security is vaild.

So I'm still +1 on removing the patch.

[Oliver Grawert (ogra)]

note that we are not shipping *any* package lists today, it is 0 vs whatever tens of MB we add by removing the patch, there is no "slightly bigger"

also this patch isnt enough i think, IIRC there are other places in livecd-rootfs where the package lists are forcefully removed to make sure we dont waste space on them, please also make the product team aware before making this change, since this limits the allowed space the custom tarball can occupy.
also check if there are any developer tools that might use the deb-src entries for pulling build deps or whatnot (and add another patch to remove all teh deb-src entries). using apport-cli will be done by perhaps 1% of the users, since you need to use a terminal or developer mode to even execute it, do we really want to make everyone suffer by this instead of working out a proper solution that doesnt waste our extremely limited free space for a feature used by a fraction of users ?

[Robie Basak (racb)]

On Tue, Aug 18, 2015 at 09:05:32AM -0000, Oliver Grawert wrote:
> ...since you need to use a terminal or developer
> mode to even execute it...

You just need to enable ssh, which is trivial. You do need a terminal to
do that, but no developer mode. I cannot run "sudo apt-get update"
manually because I don't wish to remount read-write. I'm running my
phone in production and want to file bugs for problems as such.

> ...instead of working out a proper solution that doesnt waste our
> extremely limited free space for a feature used by a fraction of users

The fraction of users who are prepared to file bugs are useful because
they cause improvements for all users. Please don't cause a fix for this
to be delayed just because it could be done better but leaving it broken
in the meantime. If the fix is workable, we should fix it now.

[Łukasz Zemczak (sil2100)]

@Oliver
Yes we are shipping package lists today. Check the ubuntu-touch generated rootfs - /var/lib/apt/lists has vivid packagelists and the stable-phone-overlay packages lists. So we do ship those and we even ship the overlay ones. There size will not change much if we include -security and -updates into that. Where did you check that we ship none right now? This was the only place where we remove package lists from the rootfs - and it was *only* for -security and -updates.

[Oliver Grawert (ogra)]

@lukasz
wow, that is seriously wrong, there was code that completely wiped them on purpose, someone must have removed the livecd-rootfs patches the OEM team added in the beginning.
this clearly cuts down the numbers that we used to compute the partition sizes. back when we did that there were no such files, i see 66M on my phone used currently, this needs to go.
we can make /var/lib/apt/lists writable instead and have a wrapper for apport-cli that updates the packages lists on the first run.

@robie
out of 10 million users, how many do you expect to use apport-cli ? 1000 ? 5000?
should the rest really suffer by having size limitations just for these few while we can have better solutions like creating these files on the fly, having them pre-generated on a server where we can download them or simply fixing apport to not need them and use a manifest file ? the whole thing is moot as soon as we switch to snappy where we don't even have apt.

lets pretty please fix this properly in apport and not by wasting more space on disk that we havent initially accounted for.

[Robie Basak (racb)]

On Tue, Aug 18, 2015 at 09:59:24AM -0000, Oliver Grawert wrote:
> @robie
> out of 10 million users, how many do you expect to use apport-cli ? 1000 ? 5000?
> should the rest really suffer by having size limitations just for these few while we can have better solutions like creating these files on the fly, having them pre-generated on a server where we can download them or simply fixing apport to not need them and use a manifest file ? the whole thing is moot as soon as we switch to snappy where we don't even have apt.

My point is that it isn't "just for those few". A good developer
experience makes for a better phone for all users.

If we can have a better solution, then let's have one. But some solution
is better than having no solution at all.

[Brian Murray (brian-murray)]

On Tue, Aug 18, 2015 at 09:05:32AM -0000, Oliver Grawert wrote:
> note that we are not shipping *any* package lists today, it is 0 vs
> whatever tens of MB we add by removing the patch, there is no "slightly
> bigger"
>
> also this patch isnt enough i think, IIRC there are other places in
> livecd-rootfs where the package lists are forcefully removed to make
> sure we dont waste space on them, please also make the product team
> aware before making this change, since this limits the allowed space
> the custom tarball can occupy.
>
> also check if there are any developer tools that might use the deb-src
> entries for pulling build deps or whatnot (and add another patch to
> remove all teh deb-src entries). using apport-cli will be done by
> perhaps 1% of the users, since you need to use a terminal or developer
> mode to even execute it, do we really want to make everyone suffer by
> this instead of working out a proper solution that doesnt waste our
> extremely limited free space for a feature used by a fraction of users
> ?

Keep in mind that, as I mentioned in comment #13, this does not just
affect apport-cli but also prevents people who have opted into reporting
crashes from sending their core dumps because the packages will have an
unknown origin.

--
Brian Murray

[Brian Murray (brian-murray)]

On Tue, Aug 18, 2015 at 09:59:24AM -0000, Oliver Grawert wrote:
> @lukasz
> wow, that is seriously wrong, there was code that completely wiped them on purpose, someone must have removed the livecd-rootfs patches the OEM team added in the beginning.
> this clearly cuts down the numbers that we used to compute the partition sizes. back when we did that there were no such files, i see 66M on my phone used currently, this needs to go.
> we can make /var/lib/apt/lists writable instead and have a wrapper for apport-cli that updates the packages lists on the first run.
>
> @robie
> out of 10 million users, how many do you expect to use apport-cli ? 1000 ? 5000?

Please reread comment #13 this does not only apply to apport-cli.

--
Brian Murray

[Łukasz Zemczak (sil2100)]

Checked on the phone: -security and -updates lists in summary weight 4.9MB right now. This shouldn't be too bad in overall.

[Łukasz Zemczak (sil2100)]

Anyone willing to sponsor this for me? Once it's in we can SRU it to all other required distros. I suppose the general consensus is that for the time being this would be the right thing to do.

[Robie Basak (racb)]

ogra asked on IRC to wait until the phone product team can comment on Monday.

Here's how it sounds to me (my attempt at a summary):

0) apport-cli and user crash reporting is broken because the -security and -update apt lists are missing.

1) It isn't ideal for apt lists to be shipped in the image since they use up space and can be downloaded later.

2) Downloading later would involve medium-sized changes to apport.

3) We're already shipping most of the apt lists already anyway, just not the -security and -updates lists that would add ~4.9MB to the size.

4) So the options to fix this are:
a) Add the -security and -updates lists with a simple patch
b) Add support for apport to handle apt list updates on the fly (including making /var/lib/apt/lists writeable or something)

5) ogra is opposed to 4a right now for size reasons.

6) Nobody has proposed to implement 4b.

7) rbasak wants to see this bug fixed, and wants to sponsor 4a to Wily assuming that it won't directly break something, since some fix is better than no fix.

[Oliver Grawert (ogra)]

note that i'm assuming you also want this in the vivid-overlay PPA for the phones to fix the bug there too.

[Brian Murray (brian-murray)]

Missing list files also affects the ability to use ubuntu-bug on Live CDs for point releases (which have packages from -updates or -security). See duplicate bug 1423382.

[Ubuntu QA Website (ubuntuqa)]

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
http://iso.qa.ubuntu.com/qatracker/reports/bugs/1471903

[Brian Murray (brian-murray)]

Who will be contacting the phone product team?

[Robie Basak (racb)]

11:18 <rbasak> pmcgowan: could you comment on bug 1471903 please? ogra_ wanted you to have an opportunity to comment before we go ahead and upload the proposed fix there to Wily, since we want the fix to end up in the vivid-overlay PPA.

11:18 <ubottu> bug 1471903 in live-build (Ubuntu) "-updates, -security missing from apt lists" [High,In progress] https://launchpad.net/bugs/1471903

11:18 <rbasak> bdmurray: ^^

11:19 <rbasak> pmcgowan: my attempt at a summary is in comment #27

[Pat McGowan (pat-mcgowan)]

Here is my two cents

Go ahead and land for Wily and I assume a Vivid SRU to unblock the issue
For phone images we want a new solution with a smaller initial footprint or with the list generated dynamically in order to reclaim the @70MB+ of space used in the update partition. This should then be provided in the overlay for OTA7. I will enter a separate bug on apport to track it.

[Pat McGowan (pat-mcgowan)]

Fix for phone images reported as bug #1489410

[Łukasz Zemczak (sil2100)]

I agree that we need a proper fix for the bug of requiring the apt lists in the first place. Thanks for the bug report Pat! It's a separate issue in overall, so no use discussing it here - in my opinion if we include apt lists anyway, there's no real point in removing only some of them.

And as Robie already mentioned, any fix is better than no fix at all. This fix could be around in use until we get the right thing in place, as the size difference looks acceptable for now.

[Brian Murray (brian-murray)]

Martin has made a change to apport in bug 1489410 that allows apport to work from devices where the apt lists do not exist. However, this will not help with the situation I've described in comment #29 (point release Live CDs). So I think this is still worth fixing.

[Łukasz Zemczak (sil2100)]

Ok, I'm also +1 on releasing the change I proposed. This shouldn't have any effect on ubuntu-touch since with Martin's apport change landed we can re-add the livecd-rootfs changes that were removing the apt lists completely. This way ubuntu-touch would re-gain the lost disk space while others can merit from the newly added apt lists.

Anyone wants to sponsor my change?

[Brian Murray (brian-murray)]

This might be worth fixing in Trusty so that one can use ubuntu-bug from the Live CD if there is another point release.