release upgrader can restore sources.list file from a previous release upgrade

[Patricia Domingues]

Impact
------
The distribution upgrader creates a backup of your sources.list named sources.list.distUpgrade before upgrading your system. This file is not cleaned up after the upgrade and under the right set of circumstances (re-attaching to a screen session which starts the dist-upgrade again, or having a strange sources.list file) the sources.list.distUpgrade file will be restored which can result in a mismatch between what you are running and what is in sources.list.

Test Case
---------
A simple test case is the following:
1) Copy /etc/apt/sources.list to /etc/apt/sources.list.distUpgrade
2) Modify /etc/apt/sources.list.distUpgrade to contain a previous release of Ubuntu e.g. 's/focal/eoan/'. (This simulates that you upgraded to your current release of Ubuntu.)
3) Modify /etc/apt/sources.list to contain the release to which you want to upgrade (strange right‽)
4) Run do-release-upgrade (with or w/o -d as necessary)
5) When prompted about "No valid sources.list entry found" choose not to upgrade
6) Observe that /etc/apt/sources.list has the same content as /etc/apt/sources.list.distUpgrade and that it doesn't match your current release.

With the version of the release upgrader in -proposed sources.list and sources.list.distUpgrade will still have the same content but it will be your current release.

A more involved test case would have one complete a release upgrade choose not to reboot then reattach to the screen session (by pressing r). After that they'd see the same thing as in Step 5 and Step 6 but given that its the same code path it seems unnecessary.

Regression Potential
--------------------
The fix involves backing up your sources.list file before presenting the error dialog regarding "No valid sources.list entry found". So its literally moving the same two lines before the dialog but the copy and paste could have gone wrong so be observant for any Tracebacks.

Original Description
--------------------
I've upgraded a server via `do-release-upgrade` from Ubuntu Bionic to Focal, with molly-guard installed on it - molly-guard configured to always ask for the hostname.

After it has finished:

```
System upgrade is complete.

Restart required

To finish the upgrade, a restart is required.
If you select 'y' the system will be restarted.

Continue [yN] y
I: molly-guard: reboot is always molly-guarded on this system.
Please type in hostname of the machine to reboot:
Good thing I asked; I won't reboot bates ...

=== Command detached from window (Tue Jul 21 15:23:36 2020) ===

=== Command terminated normally (Tue Jul 21 15:23:46 2020) ===
```

It doesn't alert you it was restoring original state of sources.list.
After rebooting the system and it came back up showing `Welcome to Ubuntu 20.04 LTS`, I've just noticed it was pointing out to bionic instead of focal installing a package.

[Patricia Domingues (patriciasd)]

attaching dist-upgrade/screenlog.0

[Brian Murray (brian-murray)]

The release upgrader should be modified so that a failure to reboot is not considered as a fail to upgrade and subsequently the sources.list files should not be restored.

[Dimitri John Ledkov (xnox)]

Failure to call reboot, should not trigger rollback to previous suite.

[Brian Murray (brian-murray)]

I tried recreating this and while the system did not reboot sources.list was not restored to bionic.

bdmurray@clean-bionic-amd64:~$ tail /var/log/dist-upgrade/screenlog.0
System upgrade is complete.

Restart required

To finish the upgrade, a restart is required.
If you select 'y' the system will be restarted.

Continue [yN] y

=== Command terminated normally (Mon Aug 3 18:44:27 2020) ===
bdmurray@clean-bionic-amd64:~$ cat /etc/apt/sources.list
deb http://192.168.10.7/ubuntu focal main restricted universe multiverse
deb http://192.168.10.7/ubuntu focal-updates main restricted universe multiverse
# deb http://archive.ubuntu.com/ubuntu bionic-proposed main restricted universe multiverse
deb http://192.168.10.7/ubuntu focal-security main restricted universe multiverse
deb-src http://192.168.10.7/ubuntu focal main restricted universe multiverse
deb-src http://192.168.10.7/ubuntu focal-updates main restricted universe multiverse
# deb-src http://archive.ubuntu.com/ubuntu bionic-proposed main restricted universe multiverse
deb-src http://192.168.10.7/ubuntu focal-security main restricted universe multiverse
bdmurray@clean-bionic-amd64:~$ ls -lh /sbin/reboot
lrwxrwxrwx 1 root root 28 Jul 9 2019 /sbin/reboot -> /lib/molly-guard/molly-guard

[Brian Murray (brian-murray)]

Ah my test in comment #4 was not via an ssh session, so that explains why I didn't receive a message from molly-guard but it its not clear why the system didn't reboot.

[Brian Murray (brian-murray)]

I tried this both with an Ubuntu desktop install of 18.04 LTS and an lxc cloud image of 18.04 LTS and during neither upgrade process did I encounter the error described in this bug report. Here's some relevant information from the lxc upgrade.

ubuntu@gorgeous-molly:~$ tail /var/log/dist-upgrade/screenlog.0

To finish the upgrade, a restart is required.
If you select 'y' the system will be restarted.

Continue [yN] y
W: molly-guard: SSH session detected!
Please type in hostname of the machine to reboot:
Good thing I asked; I won't reboot gorgeous-molly ...

=== Command detached from window (Tue Aug 4 03:29:45 2020) ===

ubuntu@gorgeous-molly:~$ cat /etc/apt/sources.list
## Note, this file is written by cloud-init on first boot of an instance
## modifications made here will not survive a re-bundle.
## if you wish to make changes you can:
## a.) add 'apt_preserve_sources_list: true' to /etc/cloud/cloud.cfg
## or do the same in user-data
## b.) add sources in /etc/apt/sources.list.d
## c.) make changes to template file /etc/cloud/templates/sources.list.tmpl

# See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to
# newer versions of the distribution.
deb http://192.168.10.7/ubuntu focal main restricted
# deb-src http://192.168.10.7/ubuntu bionic main restricted

## Major bug fix updates produced after the final release of the
## distribution.
deb http://192.168.10.7/ubuntu focal-updates main restricted
# deb-src http://192.168.10.7/ubuntu bionic-updates main restricted
...

[Brian Murray (brian-murray)]

Has some configuration of molly-guard been done that I also need to make to recreate this bug?

[Patricia Domingues (patriciasd)]

Brian, yes:
Configure to always ask for the hostname:
```
/etc/molly-guard/rc:
ALWAYS_QUERY_HOSTNAME=true
```

[Brian Murray (brian-murray)]

I tried upgrading again with the /etc/molly-guard/rc changes on both an Ubuntu 18.04 LTS desktop install (logged in via GUI) and a lxc container (connected via ssh) and in neither case were the sources.list files set back to bionic.

To finish the upgrade, a restart is required.
If you select 'y' the system will be restarted.

Continue [yN] y
I: molly-guard: reboot is always molly-guarded on this system.
Please type in hostname of the machine to reboot:
Good thing I asked; I won't reboot precious-dove ...

=== Command detached from window (Tue Aug 4 15:22:14 2020) ===ubuntu@precious-dove:~$ grep bionic /etc/apt/sources.list
# deb-src http://192.168.10.7/ubuntu bionic main restricted
# deb-src http://192.168.10.7/ubuntu bionic-updates main restricted
# deb-src http://192.168.10.7/ubuntu bionic universe
# deb-src http://192.168.10.7/ubuntu bionic-updates universe
# deb-src http://192.168.10.7/ubuntu bionic multiverse
# deb-src http://192.168.10.7/ubuntu bionic-updates multiverse
# deb-src http://192.168.10.7/ubuntu bionic-backports main restricted universe multiverse
# deb http://archive.canonical.com/ubuntu bionic partner
# deb-src http://archive.canonical.com/ubuntu bionic partner
# deb-src http://192.168.10.7/ubuntu bionic-security main restricted
# deb-src http://192.168.10.7/ubuntu bionic-security universe
# deb-src http://192.168.10.7/ubuntu bionic-security multiverse

bdmurray@clean-bionic-amd64:~$ tail /var/log/dist-upgrade/screenlog.0

To finish the upgrade, a restart is required.
If you select 'y' the system will be restarted.

Continue [yN] y
I: molly-guard: reboot is always molly-guarded on this system.
Please type in hostname of the machine to reboot:
Good thing I asked; I won't reboot clean-bionic-amd64 ...

=== Command detached from window (Tue Aug 4 08:52:24 2020) ===bdmurray@clean-bionic-amd64:~$ cat /etc/apt/sources.list
deb http://192.168.10.7/ubuntu focal main restricted universe multiverse
deb http://192.168.10.7/ubuntu focal-updates main restricted universe multiverse
# deb http://archive.ubuntu.com/ubuntu bionic-proposed main restricted universe multiverse
deb http://192.168.10.7/ubuntu focal-security main restricted universe multiverse
deb-src http://192.168.10.7/ubuntu focal main restricted universe multiverse
deb-src http://192.168.10.7/ubuntu focal-updates main restricted universe multiverse
# deb-src http://archive.ubuntu.com/ubuntu bionic-proposed main restricted universe multiverse
deb-src http://192.168.10.7/ubuntu focal-security main restricted universe multiverse

Additionally, the dist-upgrader code wouldn't restore sources.list (afaict) if the reboot failed.

[Patricia Domingues (patriciasd)]

How did you detach from the screen session?
I was able to reproduce the issue in a container pressing `x` on the screen session.

[Brian Murray (brian-murray)]

Yes, I pressed 'x' to destroy the screen session. Could you provide /var/log/dist-upgrade/main.log and /var/log/dist-upgrade/screenlog.0 so I can try and sort out what's going on?

[Patricia Domingues (patriciasd)]

Sure. Comment #1 has screenlog.0. I'm adding the main.log:

[Patricia Domingues (patriciasd)]

just wondering something, now we have the point release 20.04.1, when I've upgraded the server it was not released yet, so I ran with `-d`: `do-release-upgrade -d`

[Brian Murray (brian-murray)]

The 20.04.1 point release is not out yet so I was running 'do-release-upgrade -d'. I still don't see anything obvious in the log files the directory '/var/log/dist-upgrade' should also contain a file name apt-clone_system_state.tar.gz which is a snapshot of what the system looked like before the upgrade. If I had that I could restore the clone file and try the upgrade myself.

[Patricia Domingues (patriciasd)]

Brian, sorry, I noticed that I've sent the wrong 'main.log' file
from `/var/log/dist-upgrade/20200721-1523/` instead of `/var/log/dist-upgrade/`.
I'm attaching the correct one. Sorry!

[Brian Murray (brian-murray)]

Thanks for the apt-clone file. I'm still not having any luck recreating this issue though.

[Brian Murray (brian-murray)]

Okay, I've one more idea. When the release upgrade process runs it creates a backup of the sources.list file (/etc/apt/sources.list.distUpgrade). Could you compare the "restored" /etc/apt/sources.list file to the contents of /etc/apt/sources.list.distUpgrade? I'd like to see a diff of the two files. Thanks!

[Patricia Domingues (patriciasd)]

Ok. I see what I might have made wrong: trying to resurrect the session (pressing 'r')
I was able to reproduce it just this way:

After it asks for upgrade:
```
If you select 'y' the system will be restarted.

Continue [yN] y
I: molly-guard: reboot is always molly-guarded on this system.
Please type in hostname of the machine to reboot:
Good thing I asked; I won't reboot upgteste ...
=== Command detached from window (Wed Aug 19 21:24:36 2020) ===
=== Command terminated normally (Wed Aug 19 21:24:46 2020) ===

Reading cache

Checking package manager
Reading package lists... Done
Building dependency tree
Reading state information... Done
Hit http://ports.ubuntu.com/ubuntu-ports focal InRelease
Hit http://ports.ubuntu.com/ubuntu-ports focal-updates InRelease
Hit http://ports.ubuntu.com/ubuntu-ports focal-backports InRelease
Hit http://ports.ubuntu.com/ubuntu-ports focal-security InRelease
Fetched 0 B in 0s (0 B/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done

Updating repository information

No valid sources.list entry found

While scanning your repository information no entry about eoan could
be found.

An upgrade might not succeed.

Do you want to continue anyway?

Continue [yN] n

Restoring original system state

Aborting
Reading package lists... Done
Building dependency tree
Reading state information... Done
=== Command detached from window (Wed Aug 19 21:26:11 2020) ===
=== Command terminated with exit status 1 (Wed Aug 19 21:26:21 2020) ===
```
after that sources.list had bionic entries:
```
# See http://help.ubuntu.com/community/UpgradeNotes for how to upgra...

[Patricia Domingues (patriciasd)]

Brian, answering comment#18, server upgraded has finished ~ 15h30`:
from logs: `=== Command detached from window (Tue Jul 21 15:23:36 2020) ===`

I just noticed it had bionic entries on sources.list when I was installing bridge-utils:

```
Start-Date: 2020-07-21 16:28:23
Commandline: apt install bridge-utils
Requested-By: ubuntu (1000)
Install: bridge-utils:amd64 (1.5-15ubuntu1)
End-Date: 2020-07-21 16:28:33
```

This is Bionic version, focal is 1.6-2ubuntu1, so after that I've replaced `sources.list` suite from bionic to focal.

[Brian Murray (brian-murray)]

Ah, thanks for that additional information! I was able to recreate this issue by upgrading from bionic to focal and then running 'sudo ./focal --frontend DistUpgradeViewText' in the folder in which the release upgrader was extracted e.g. '/tmp/ubuntu-release-upgrader-nbgn2n5o'.

Perhaps the distribution upgrade process should remove the sources.list.distUpgrade file after the release upgrade successfully completes.

bdmurray@clean-bionic-amd64:/tmp/ubuntu-release-upgrader-nbgn2n5o$ ls -lh /etc/apt/
total 36K
drwxr-xr-x 2 root root 4.0K Aug 22 09:47 apt.conf.d
drwxr-xr-x 2 root root 4.0K Mar 11 2019 auth.conf.d
drwxr-xr-x 2 root root 4.0K Apr 20 2018 preferences.d
-rw-rw-r-- 1 root root 676 Aug 24 10:42 sources.list
drwxr-xr-x 2 root root 4.0K Aug 22 07:28 sources.list.d
-rw-rw-r-- 1 root root 676 Aug 22 07:28 sources.list.distUpgrade
-rw-rw-r-- 1 root root 2.9K Aug 13 2018 sources.list.ori
-rw-rw-r-- 1 root root 674 Aug 22 07:24 sources.list.save
drwxr-xr-x 2 root root 4.0K Aug 22 07:37 trusted.gpg.d
bdmurray@clean-bionic-amd64:/tmp/ubuntu-release-upgrader-nbgn2n5o$ md5sum /etc/apt/sources.list /etc/apt/sources.list.distUpgrade
1d22517b154947f8a818e36f6edf7c62 /etc/apt/sources.list
1d22517b154947f8a818e36f6edf7c62 /etc/apt/sources.list.distUpgrade

[Brian Murray (brian-murray)]

Actually, just choosing to resurrect the screen window will restart the upgrade process which then will warn you about an upgrade not succeeding and then if you exit that sources.list.distUpgrade will be restored.

I tried messing with screenrc and adding "onerror" to zombie (https://www.gnu.org/software/screen/manual/html_node/Zombie.html) but that did not help.

[Brian Murray (brian-murray)]

Come to find out that adding "onerror" to zombie in the screenrc file does help there is just what is approximately a 10 second delay between the "Command detached from window" message and the upgrade process exiting 0 which then causes screen to disappear.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-release-upgrader - 1:20.10.10

---------------
ubuntu-release-upgrader (1:20.10.10) groovy; urgency=medium

  * DistUpgrade/DistUpgradeController.py: do what the comment says and really
    back up /etc/apt/sources.list first. (LP: #1888916)
  * DistUpgrade/screenrc: add 'onerror' to the zombie command so that if the
    process exits normally the screen window will disappear.

 -- Brian Murray <email address hidden> Tue, 25 Aug 2020 14:17:55 -0700

[Łukasz Zemczak (sil2100)]

Hello Patricia, or anyone else affected,

Accepted ubuntu-release-upgrader into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-release-upgrader/1:20.04.25 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Patricia Domingues (patriciasd)]

Thanks. I've tested upgrading from focal to groovy:

`ubuntu-release-upgrader-core/focal-proposed 1:20.04.25 all [upgradable from: 1:20.04.24]`

```patricia@f-g:~$ dpkg -l |grep ubuntu-release
ii ubuntu-release-upgrader-core 1:20.04.25 all manage release upgrades
```

Now, I don't see the same issue anymore even if I press `R` to resurrect the screen session before it closes automatically `[screen is terminating]`.

[Brian Murray (brian-murray)]

I tested a release upgrade from Bionic to Focal and confirm that my sources.list file was not reverted back to Artful.

bdmurray@clean-bionic-amd64:/tmp/20.04$ head /var/log/dist-upgrade/main.log
2020-08-31 08:01:42,527 INFO Using config files '['./DistUpgrade.cfg.bionic']'
2020-08-31 08:01:42,527 INFO uname information: 'Linux clean-bionic-amd64 5.4.0-42-generic #46~18.04.1-Ubuntu SMP Fri Jul 10 07:21:24 UTC 2020 x86_64'
2020-08-31 08:01:42,819 INFO apt version: '1.6.12ubuntu0.1'
2020-08-31 08:01:42,820 INFO python version: '3.6.9 (default, Jul 17 2020, 12:50:27)
[GCC 8.4.0]'
2020-08-31 08:01:42,822 INFO release-upgrader version '20.04.25' started

bdmurray@clean-bionic-amd64:/tmp/20.04$ grep -A12 "no entry" /var/log/dist-upgrade/screenlog.0
While scanning your repository information no entry about bionic
could be found.

An upgrade might not succeed.

Do you want to continue anyway?

Continue [yN] n

Restoring original system state

Aborting
Reading package lists... Done

bdmurray@clean-bionic-amd64:/tmp/20.04$ cat /etc/apt/sources.list
deb http://192.168.10.7/ubuntu focal main restricted universe multiverse
deb http://192.168.10.7/ubuntu focal-updates main restricted universe multiverse
#deb http://192.168.10.7/ubuntu focal-proposed main restricted universe multiverse
deb http://192.168.10.7/ubuntu focal-security main restricted universe multiverse
deb-src http://192.168.10.7/ubuntu focal main restricted universe multiverse
deb-src http://192.168.10.7/ubuntu focal-updates main restricted universe multiverse
#deb-src http://192.168.10.7/ubuntu focal-proposed main restricted universe multiverse
deb-src http://192.168.10.7/ubuntu focal-security main restricted universe multiverse

Keep in mind the sources.list file is correct as the test case had you modify it to the release which you are upgrading.

bdmurray@clean-bionic-amd64:/tmp/20.04$ md5sum /etc/apt/sources.list
6d4439d7fd3f142f81b70e00abd7b95a /etc/apt/sources.list
bdmurray@clean-bionic-amd64:/tmp/20.04$ md5sum /etc/apt/sources.list.distUpgrade
6d4439d7fd3f142f81b70e00abd7b95a /etc/apt/sources.list.distUpgrade

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-release-upgrader - 1:20.04.25

---------------
ubuntu-release-upgrader (1:20.04.25) focal; urgency=medium

  * DistUpgrade/DistUpgradeQuirks.py: Add a check for ROS packages being
    installed and warn that upgrades with them installed are not likely to
    work. Thanks to Kyle Fazzari for the patch. (LP: #1611737)
  * DistUpgrade/DistUpgradeController.py: do what the comment says and really
    back up /etc/apt/sources.list first. (LP: #1888916)

 -- Brian Murray <email address hidden> Wed, 26 Aug 2020 08:36:39 -0700

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for ubuntu-release-upgrader has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.