/var/lib/apt/lists has content in partner-images/Docker tarballs, which becomes stale quickly

[Tianon Gravi]

In the Docker builds, we explicitly include "rm -rf /var/lib/apt/lists/*" to remove the APT list files, because they're quite large, and they get stale very quickly.

We instead recommend users always couple "apt-get update" with their "apt-get install" lines (see https://docs.docker.com/engine/userguide/eng-image/dockerfile_best-practices/#apt-get).

Is there any chance of getting those files excluded or removed from the tarballs themselves? Uncompressed, this would result in a ~30MB size decrease (obviously likely less dramatic for the compressed tarballs). The user-facing result is that users are forced to run "apt-get update" before they can "apt-get install" (which for the Docker images, we already do, so there will be no user-facing effect except that the on-disk image layers will be ~30MB smaller).

See also https://github.com/tianon/docker-brew-ubuntu-core/issues/90 for more discussion on this (which is what led to filing this issue). :)

[Ed Morley (edmorley)]

Where do the build scripts for these images live? I can't seem to find the source.

[Davim (davim)]

in https://github.com/tianon/docker-brew-ubuntu-core/blob/8ff098f80b47a127e0ccfb8a1400bb40ab28e23a/update.sh#L90 there's a layer that removes /var/lib/apt/lists/* but having this on it's own layer is pointless because it will sill be in the previous layers, not only these files should not be in the tarball but also "rm -rf /var/lib/apt/lists/*" should be merged into the previous layer.

[Ed Morley (edmorley)]

Hi! It would be great if this change could be made for the Ubuntu 18.04 release.

I'm presuming part of the hold-up here was the concern over breaking people who rely on the cloud images but don't run `apt-get update` themselves (whether that's a good security feature or not aside!). However with a new major version this should be less of a concern, right?

If there is anyone on the subscriber list who knows who else could be CCed here that is closer to where the change needs to be made - I'd much appreciate you adding them.

Many thanks :-)

[Tianon Gravi (tianon)]

@mwhudson I'm concerned about https://bazaar.launchpad.net/~mwhudson/livecd-rootfs/lp-1699913/revision/1675 -- this leaves around the index files that aren't going to change, but I think this is going to cause subtle security issues for some users, isn't it?

With this change, "RUN apt-get install -y git" will install old vulnerable git while "RUN apt-get update && apt-get install -y git" will install security-updated git, which is the other half of why we remove the lists entirely in the base image (IMO, getting "no such package" is a much better indicator of the need for "apt-get update" than just getting an outdated/insecure package quietly).

[Scott Moser (smoser)]

@Tianon,

You are correct that a user then would inevitably get an out of date package.

However, removing the release pocket content
a.) means every user in every image has to download that data which does not change.
b.) only defers the potential problem until later.

Essentially, the problem is that in order to *ever* run:
  apt-get install package
you have to have run recently run "apt-get update" for some value of recently.

default cloud images (I believe) end up running apt-get update within 24 hours and
then do so on a daily-ish basis, which probably suffices as a definition of
"recently".

I think the ideal solution is to have 'apt-get install' (or 'apt install') to
run an update by default if it detects that sources are out of date or modified.

Related bug wit some other info and discussion at bug 1429285 .

[Tianon Gravi (tianon)]

++ Absolutely agree that having "apt-get install" auto-update stale indexes would be ideal (or even a "--update" flag to install for combining them); looks like that discussion is over in bug 1709603. :)

[Launchpad Janitor (janitor)]

This bug was fixed in the package livecd-rootfs - 2.529

---------------
livecd-rootfs (2.529) cosmic; urgency=medium

  * Improvements to the ubuntu-base (i.e. Docker) build:
    - Customize the unminimize script to not install ubuntu-minimal.
    - Divert /usr/bin/man to a script that points to the unminimize script.
    - Remove apt lists that are currently removed downstream anyway.
      (LP: #1699913)

 -- Michael Hudson-Doyle <email address hidden> Thu, 07 Jun 2018 05:59:12 +1200