dist-upgrader should clean up downloaded debs

[Brian Murray]

During the release upgrade process packages for the new release are downloaded to the apt configuration "Dir::Cache::archives" (usually /var/cache/apt/archives/), but those are not removed from disk until some time later. With the dist-upgrader growing support for transitioning from debian packages to snaps for some software it's possible that there may not be enough free space on disk (as the upgrader doesn't currently calculate the space need which is bug 1783597) to support installing the snaps. Subsequently, it'd be good to remove the debs from disk after the upgrade has completed since they are likely on /var which is where the snaps will need space.

[Test Case]
1) Run 'du -sh /var/cache/apt/archives'
2) Upgrade from Ubuntu 16.04 to Ubuntu 18.04
3) Run 'du -sh /var/cache/apt/archives'
4) Observe that /var/cache/apt/archives is taking up more space than it did.

With the version of the dist-upgrader in -proposed /var/cache/apt/archives should be nearly empty - some debs may be left because they are left over from regular updates in Ubuntu 16.04 and were not part of the release upgrade process.

[Regression Potential]
It's possible that the debs are removed too early and if there were a failure to upgrade it'd be hard to recover from. One way to test this would be to use the apt-clone file from 1766890 which would create an upgrade failure.

[Brian Murray (brian-murray)]

ubuntu-release-upgrader (1:18.10.6) cosmic; urgency=medium

  * DistUpgradeQuirks.py:
    - make sure that snapd is installed before trying to use it.
      (LP: #1783589)
    - update the view with information regarding the progress of snaps being
      installed. (LP: #1783593)
    - when checking for connectivity to the snap store use C.UTF-8 for the
      language so error message matching works. (LP: #1783738)
  * DistUpgradeController.py:
    - Remove debs from apt's "Dir::Cache::archives" folder after the upgrade
      has completed successfully.
    - Add a telemetry marker to report the time to process migration from deb
      to snaps.
  * DistUpgrade/DistUpgradeViewGtk3.py: call vte's terminal.feed_child() with
    the correct parameters thereby fixing a crash. (LP: #1780501)

 -- Brian Murray <email address hidden> Wed, 01 Aug 2018 15:06:53 -0700

[Łukasz Zemczak (sil2100)]

Hello Brian, or anyone else affected,

Accepted ubuntu-release-upgrader into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-release-upgrader/1:18.04.22 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

I tested an upgrade from Ubuntu 16.04 to Ubuntu 18.04 today using the dist-upgrader from -proposed (manually downloaded and extracted b/c you can't use do-release -p and -d) before the upgrade I had installed an updated version of snapd so /var/cache/apt/archives was 14 MB. After the upgrade it was still 14 MB indicating that the debs downloaded for the upgrade had been removed from disk.

bdmurray@clean-xenial-amd64:~/u-r-u$ du -sh /var/cache/apt/archives
du: cannot read directory '/var/cache/apt/archives/partial': Permission denied
14M /var/cache/apt/archives
bdmurray@clean-xenial-amd64:~/u-r-u$ ls -lh /var/cache/apt/archives/
total 14M
-rw-r----- 1 root root 0 Jul 19 2016 lock
drwx------ 2 _apt root 4.0K Aug 6 11:22 partial
-rw-r--r-- 1 root root 1.7K Jul 19 03:58 snap-confine_2.34.2_amd64.deb
-rw-r--r-- 1 root root 14M Jul 19 03:58 snapd_2.34.2_amd64.deb
-rw-r--r-- 1 root root 1.6K Jul 19 03:58 ubuntu-core-launcher_2.34.2_amd64.deb

Setting to verification-done.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-release-upgrader - 1:18.04.22

---------------
ubuntu-release-upgrader (1:18.04.22) bionic; urgency=medium

  * DistUpgradeQuirks.py:
    - make sure that snapd is installed before trying to use it.
      (LP: #1783589)
    - update the view with information regarding the progress of snaps being
      installed. (LP: #1783593)
    - when checking for connectivity to the snap store use C.UTF-8 for the
      language so error message matching works. (LP: #1783738)
  * DistUpgradeController.py:
    - Remove debs from apt's "Dir::Cache::archives" folder after the upgrade
      has completed successfully. (LP: #1785096)
    - Add a telemetry marker to report the time to process the migration from
      deb to snaps.

 -- Brian Murray <email address hidden> Thu, 02 Aug 2018 11:20:06 -0700

[Steve Langasek (vorlon)]

The verification of the Stable Release Update for ubuntu-release-upgrader has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.