RELEASE_UPGRADER_ALLOW_THIRD_PARTY env variable not passed to dist-upgrader tarball

[Brian Murray]

Impact
------
It is not possible to override the dist-upgrader's third party mirror check by an environmental variable, subsequently users are asked a question when they shouldn't be.

Test Case
---------
The change is in do-release-upgrade and DistUpgradeFetcherCore.py so affects upgrades to the release after the one receiving the SRU e.g. if the SRU is for cosmic test upgrading to disco not cosmic. This needs to be tested for check-new-release-gtk and do-release-upgrade.

1) Run RELEASE_UPGRADER_ALLOW_THIRD_PARTY=1 /usr/lib/ubuntu-release-upgrader/check-new-release-gtk (or do-release-upgrade and with -d to upgrade to disco)
2) Choose to proceed with the upgrade

With the version of ubuntu-release-upgrader in the archive you'll receive a dialog (or message for the text upgrader) saying "No valid mirror found" and "Do you want to rewrite 'sources.list' anyway?". With the version of ubuntu-release-upgrader from -proposed you'll receive no such dialog and you can find "WARNING mirror check skipped" in /var/log/dist-upgrade/main.log.

There is no need to complete the full upgrade.

Regression Potential
--------------------
With the pkexec change we are only passing the RELEASE_UPGRADER_ALLOW_THIRD_PARTY env variable on to the release upgrade process but with the do-release-upgrade (DistUpgradeFetcherCore.py / sudo) one we are passing all existing environment variables on which could cause an issue for the release upgrade process.

[Original Description]
ubuntu-release-upgrader indicates that RELEASE_UPGRADER_ALLOW_THIRD_PARTY can be set in the environment so that release upgrades will be attempted without commenting out third party mirrors or repositories, however in practice this is untrue. This doesn't work because pkexec is used to call do-release-upgrade, and it doesn't pass on env, and because sudo (used for DistUpgradeViewText) is called with -E. This should be sorted out so that the documentation is not lying and so upgrades with PPAs enabled can be attempted.

Its possible to confirm the fix for this bug by looking at /var/log/dist-upgrade/main.log for a PPA not being commented out or the following line:

2018-12-05 14:23:29,008 WARNING mirror check skipped, *overriden* via config

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-release-upgrader - 1:19.04.7

---------------
ubuntu-release-upgrader (1:19.04.7) disco; urgency=medium

  [ Brian Murray ]
  * DistUpgrade/DistUpgradeCache.py: When calculating free space needed for
    mount points don't use a negative number as the buffer. (LP: #1786484)
  * fix more typos re RELEASE_UPRADER_ALLOW_THIRD_PARTY
  * DistUpgrade/DistUpgradeCache.py: When calculating free space needed for
    mount points don't use a negative number as the buffer. (LP: #1786484)
  * fix more typos re RELEASE_UPRADER_ALLOW_THIRD_PARTY
  * DistUpgrade/DistUpgradeController.py: When rewriting sources.list entries
    check to see if the source provides packages for the release to which the
    upgrade is occurring. If it doesn't the entry is disabled thereby
    improving upgrades with PPAs. (LP: #1807043)
  * do-release-upgrade: add a parameter to allow third party mirrors and
    repositories, additionally pass along the environmental variable
    RELEASE_UPGRADER_ALLOW_THIRD_PARTY via pkexec and sudo. (LP: #1807032)
  * DistUpgrade/DistUpgradeCache.py: improve messaging regarding failures to
    calculate the upgrade and suggest using ppa-purge if PPAs are an issue.
  * DistUpgrade/DistUpgradeCache.py: in the event there is a failure to
    calculate the upgrade provide information about the log files in
    /var/log/dist-upgrade. (LP: #1771387)

  [ Jeremy Bicha ]
  * autopkgtest: Update dependency from gir1.2-webkit-3.0 to gir1.2-webkit2-4.0

 -- Brian Murray <email address hidden> Tue, 11 Dec 2018 15:35:34 -0800

[Łukasz Zemczak (sil2100)]

Hello Brian, or anyone else affected,

Accepted ubuntu-release-upgrader into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-release-upgrader/1:18.10.11.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

I tested this on an Ubuntu Cosmic system on which I installed ubuntu-release-upgrader-gtk version 18.10.11.3 from -proposed. I use a local mirror of the archive and used the right environmental variable when calling check-new-release-gtk to upgrade to disco. I ended up seeing the correct message regarding the mirror check in /var/log/dist-upgrade/main.log and did not receive the dialog about no valid mirror.

[Brian Murray (brian-murray)]

I then tested using the environment variable with /usr/bin/do-release-upgrade and saw the same mirror check skipped message in /var/log/dist-upgrade/main.log, so I'm setting this to verification done.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-release-upgrader - 1:18.10.11.3

---------------
ubuntu-release-upgrader (1:18.10.11.3) cosmic; urgency=medium

  * DistUpgrade/DistUpgradeCache.py: When calculating free space needed for
    mount points don't use a negative number as the buffer. (LP: #1786484)
  * fix more typos re RELEASE_UPRADER_ALLOW_THIRD_PARTY.
  * DistUpgrade/DistUpgradeController.py: When rewriting sources.list entries
    check to see if the source provides packages for the release to which the
    upgrade is occurring. If it doesn't the entry is disabled thereby
    improving upgrades with PPAs. (LP: #1807043)
  * do-release-upgrade: add a parameter to allow third party mirrors and
    repositories, additionally pass along the environmental variable
    RELEASE_UPGRADER_ALLOW_THIRD_PARTY via pkexec and sudo. (LP: #1807032)
  * DistUpgrade/DistUpgradeCache.py: improve messaging regarding failures to
    calculate the upgrade and suggest using ppa-purge if PPAs are an issue.
  * DistUpgrade/DistUpgradeCache.py: in the event there is a failure to
    calculate the upgrade provide information about the log files in
    /var/log/dist-upgrade. (LP: #1771387)
  * debian/source_ubuntu-release-upgrader.py: Include xorg_fixup.log if it
    exists.
  * DistUpgrade/xorg_fix_proprietary.py: modify how the system is checked to
    see if nvidia is being used, drop fglrx check since it has been
    deprecated. (LP: #1773637)

 -- Brian Murray <email address hidden> Thu, 20 Dec 2018 12:31:37 -0800

[Chris Halse Rogers (raof)]

The verification of the Stable Release Update for ubuntu-release-upgrader has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Łukasz Zemczak (sil2100)]

Hello Brian, or anyone else affected,

Accepted ubuntu-release-upgrader into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-release-upgrader/1:18.04.30 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

I tested a release upgrade from bionic to cosmic with the versions of update-manager and ubuntu-release-upgrader from -proposed and my third party mirror enabled - during the upgrade I did not receive the "No valid mirror" found error message and my sources.list was just rewritten from bionic to cosmic. Additionally, we can see the new line in /var/log/dist-upgrade/main.log:

bdmurray@clean-bionic-amd64:~$ apt-cache policy ubuntu-release-upgrader-core
ubuntu-release-upgrader-core:
  Installed: 1:18.04.30
  Candidate: 1:18.04.30
  Version table:
 *** 1:18.04.30 100
        100 /var/lib/dpkg/status
     1:18.04.29 500
        500 http://192.168.10.7/ubuntu bionic-updates/main amd64 Packages
        500 http://192.168.10.7/ubuntu bionic-updates/main i386 Packages
     1:18.04.17 500
        500 http://192.168.10.7/ubuntu bionic/main amd64 Packages
        500 http://192.168.10.7/ubuntu bionic/main i386 Packages
bdmurray@clean-bionic-amd64:~$ grep "overriden" /var/log/dist-upgrade/main.log
2019-01-23 06:27:37,048 WARNING mirror check skipped, *overriden* via config

[Brian Murray (brian-murray)]

I also tested using /usr/lib/ubuntu-release-upgrader/check-new-release-gtk with the environment variable and it also worked.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-release-upgrader - 1:18.04.30

---------------
ubuntu-release-upgrader (1:18.04.30) bionic; urgency=medium

  * data/release-upgrades: Clarify documentation regarding the behavior for
    different Prompt settings. (LP: #1795024)
  * do-release-upgrade: Utilize information regarding what Prompt is set to so
    that a more informative error message can be displayed.
    (LP: #1798618, LP: #1795024)
  * DistUpgrade/DistUpgradeCache.py: When calculating free space needed for
    mount points don't use a negative number as the buffer. (LP: #1786484)
  * fix more typos re RELEASE_UPRADER_ALLOW_THIRD_PARTY
  * DistUpgrade/DistUpgradeController.py: When rewriting sources.list entries
    check to see if the source provides packages for the release to which the
    upgrade is occurring. If it doesn't the entry is disabled thereby
    improving upgrades with PPAs. (LP: #1807043)
  * do-release-upgrade: add a parameter to allow third party mirrors and
    repositories, additionally pass along the environmental variable
    RELEASE_UPGRADER_ALLOW_THIRD_PARTY via pkexec and sudo. (LP: #1807032)
  * DistUpgrade/DistUpgradeCache.py: improve messaging regarding failures to
    calculate the upgrade and suggest using ppa-purge if PPAs are an issue.
  * DistUpgrade/DistUpgradeCache.py: in the event there is a failure to
    calculate the upgrade provide information about the log files in
    /var/log/dist-upgrade. (LP: #1771387)
  * debian/source_ubuntu-release-upgrader.py: Include xorg_fixup.log if it
    exists.
  * DistUpgrade/xorg_fix_proprietary.py: modify how the system is checked to
    see if nvidia is being used, drop fglrx check since it has been
    deprecated. (LP: #1773637)

 -- Brian Murray <email address hidden> Wed, 16 Jan 2019 15:53:58 -0800