do-release-upgrade fails with "Your python3 install is corrupted" if /usr/bin/python points to /etc/alternatives/python

[Dmitrii Shcherbakov]

do-release-upgrade from 18.04 to 18.10 fails (likewise, when I upgraded with a workaround the same happened for 18.10 -> 19.04):

"
Reading cache

Checking package manager

Can not upgrade

Your python3 install is corrupted. Please fix the '/usr/bin/python3'
symlink.
=== Command terminated with exit status 1 (Sat Apr 20 17:19:49 2019) ===
"

There is nothing wrong with /usr/bin/python3

➜ ~ ls -al /usr/bin/python3
lrwxrwxrwx 1 root root 9 окт 25 14:11 /usr/bin/python3 -> python3.6

I also have python2 installed for which Debian alternatives are configured which causes the problem as far as I can see:

➜ ~ update-alternatives --display python
python - auto mode
  link best version is /usr/bin/python2.7
  link currently points to /usr/bin/python2.7
  link python is /usr/bin/python
/usr/bin/python2.7 - priority 100

https://git.launchpad.net/ubuntu/+source/ubuntu-release-upgrader/tree/DistUpgrade/DistUpgradeController.py?h=ubuntu/bionic-updates#n361

(Pdb) l
 11 debian_defaults = '/usr/share/%s/debian_defaults' % dirname
 12 if os.path.exists(debian_defaults):
 13 config = SafeConfigParser()
 14 with open(debian_defaults) as f:
 15 config.readfp(f)
 16 -> try:
 17 expected_default = config.get('DEFAULT', 'default-version')
 18 except NoOptionError:
 19 logging.debug("no default version for %s found in '%s'" %
 20 (binary, config))
 21 return False
(Pdb) n
> <ipython-input-13-93c4105e7796>(17)_pythonSymlinkCheck()
-> expected_default = config.get('DEFAULT', 'default-version')
(Pdb) n
> <ipython-input-13-93c4105e7796>(22)_pythonSymlinkCheck()
-> try:
(Pdb) expected_default
'python2.7'

> <ipython-input-13-93c4105e7796>(23)_pythonSymlinkCheck()
-> fs_default_version = os.readlink('/usr/bin/%s' % binary)
(Pdb) n
> <ipython-input-13-93c4105e7796>(27)_pythonSymlinkCheck()
-> if not fs_default_version in (expected_default, os.path.join('/usr/bin', expected_default)) and not (binary == 'python' and fs_default_version in ('python2', '/usr/bin/python2')):
(Pdb) fs_default_version
'/etc/alternatives/python'
(Pdb) os.path.join('/usr/bin', expected_default)
'/usr/bin/python2.7'

(Pdb) not fs_default_version in (expected_default, os.path.join('/usr/bin', expected_default))
True

(Pdb) print("%s symlink points to: '%s', but expected is '%s' or '%s'" % (binary, fs_default_version, expected_default, os.path.join('/usr/bin', expected_default)))
python symlink points to: '/etc/alternatives/python', but expected is 'python2.7' or '/usr/bin/python2.7'

A workaround is:

sudo rm /usr/bin/python
sudo ln -s /usr/bin/python2.7 /usr/bin/python
do-release-upgrade

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-release-upgrader - 1:19.10.15

---------------
ubuntu-release-upgrader (1:19.10.15) eoan; urgency=medium

  * Don't check for validity of /usr/bin/python symlink because this is no
    longer relevant to the upgrades (and the printed error message was
    inaccurate). LP: #1825655.
  * Clean up remaining references to /usr/bin/python (tests and utilities)
    in the source.

 -- Steve Langasek <email address hidden> Fri, 11 Oct 2019 16:54:15 -0700

[Nikita Koval (nikodll)]

Just hit the exactly same issue while trying to upgrade from 19.04 to 19.10

root@ferma:~# ls -l /usr/bin/python*
lrwxrwxrwx 1 root root 24 Oct 23 22:37 /usr/bin/python -> /etc/alternatives/python
lrwxrwxrwx 1 root root 9 Mar 4 2019 /usr/bin/python2 -> python2.7
-rwxr-xr-x 1 root root 3691008 Oct 7 19:36 /usr/bin/python2.7
lrwxrwxrwx 1 root root 25 Oct 23 22:52 /usr/bin/python3 -> /etc/alternatives/python3
-rwxr-xr-x 1 root root 1797 Jul 12 2017 /usr/bin/python3-unidiff
-rwxr-xr-x 2 root root 4877888 Oct 7 14:56 /usr/bin/python3.7
-rwxr-xr-x 2 root root 4877888 Oct 7 14:56 /usr/bin/python3.7m
lrwxrwxrwx 1 root root 10 Mar 26 2019 /usr/bin/python3m -> python3.7m
root@ferma:~# ls -l /etc/alternatives/python
lrwxrwxrwx 1 root root 18 Oct 23 22:53 /etc/alternatives/python -> /usr/bin/python2.7
root@ferma:~# ls -l /etc/alternatives/python3
lrwxrwxrwx 1 root root 18 Oct 23 22:55 /etc/alternatives/python3 -> /usr/bin/python3.7
root@ferma:~# rm /usr/bin/python

[Steve Langasek (vorlon)]

> Just hit the exactly same issue while trying to upgrade from 19.04 to 19.10

No, that is not the same issue. In your case, the error message is *accurately* reporting that you have corrupted your python3 system install.

/usr/bin/python3 and /usr/bin/python are not alternatives in Ubuntu, and changing them to be so is unsupported. The difference is that /usr/bin/python is not used by do-release-upgrade, so is not relevant to report on; and /usr/bin/python3 is.

[wolfgang rumpf (wolfgang-rumpf)]

I'm experiencing the same issue as Nikita above. I have tried to reinstall python3, I have tried replacing the symlinks, nothing has fixed this. Help?

[Steve Langasek (vorlon)]

On Thu, Oct 24, 2019 at 12:44:25AM -0000, wolfgang rumpf wrote:
> I'm experiencing the same issue as Nikita above. I have tried to
> reinstall python3, I have tried replacing the symlinks, nothing has
> fixed this. Help?

The correct symlink is /usr/bin/python3 -> /usr/bin/python3.7.

[wolfgang rumpf (wolfgang-rumpf)]

Yes. Tried that. Still getting the corrupt python3 error message.

I can run python3, however, it works fine. Any suggestions on how to trouble-shoot this?

[Dmitrii Shcherbakov (dmitriis)]

If I am not mistaken, Disco/19.04 doesn't have the fix backported yet:

https://git.launchpad.net/ubuntu/+source/ubuntu-release-upgrader/tree/DistUpgrade/DistUpgradeController.py?h=ubuntu/disco-updates#n362
        binaries_and_dirnames = [("python", "python"), ("python2", "python"),
                                 ("python3", "python3")]

While 19.10 does:

https://git.launchpad.net/ubuntu/+source/ubuntu-release-upgrader/tree/DistUpgrade/DistUpgradeController.py?h=ubuntu/eoan#n362
        binaries_and_dirnames = [("python3", "python3")]

wolfgang-rumpf,

* do you have python2 installed by any chance?
* does /usr/bin/python point to /usr/bin/python2.7 or /etc/alternatives/python?
* if it points to /etc/alternatives/python you are hitting the original bug I filed about the incorrect check performed by the release upgrader. Without the fix, it looks at /usr/bin/python while it only actually cares about python3. So if you fix this symlink everything will work as expected.

If you just need to do it quickly on one system (i.e. upgrade your laptop):

sudo rm /usr/bin/python
sudo ln -s /usr/bin/python2.7 /usr/bin/python
do-release-upgrade

Also, do you remember why an alternative was set up in the first place? It would be helpful to understand where this kind of setup is coming from.

[wolfgang rumpf (wolfgang-rumpf)]

I do have python2 installed, but my /usr/bin/python points to /usr/bin/python3

Changing the default (/usr/bin/python) symbolic link as you suggested is enabling me to upgrade my laptop - so I guess the problem can be ignored for now....

[Brian Murray (brian-murray)]

Hello Dmitrii, or anyone else affected,

Accepted ubuntu-release-upgrader into eoan-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-release-upgrader/1:19.10.15.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-eoan to verification-done-eoan. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-eoan. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[drohhyn (drohhyn)]

I'm not really sure, if I am doing Brian's proposal right.

I am currently on disco 19.04 and have the same error as wolfgang. I have configured using -proposed packages like documented in https://wiki.ubuntu.com/Testing/EnableProposed. My file /etc/apt/preferences.d/proposed-updates looks like this:
Package: *
Pin: release a=eoan-proposed
Pin-Priority: 400

Is this really correct? Shouldn't is be
Package: *
Pin: release a=disco-proposed
Pin-Priority: 400
as I am still on disco?

And in addition the only packages listed are ubuntu-release-upgrader-gtk, *-qt and *-core with both variations, but not ubuntu-release-upgrader itself.
What am I doing wrong?

[Brian Murray (brian-murray)]

Testing the distribution upgrade tarball is a bit different - you need to run "do-release-upgrade -p". Modifying /etc/apt/preferences.d/ is unnecessary.

[Brian Murray (brian-murray)]

I tested this by installing python2 on an Ubuntu 19.04 system then symlinking /usr/bin/python to /etc/alternatives/python - an empty file but that's fine. I then ran 'do-release-upgrade' and received the error message in the description. I then ran 'do-release-upgrade -p' and the upgrade proceeded.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-release-upgrader - 1:19.10.15.2

---------------
ubuntu-release-upgrader (1:19.10.15.2) eoan; urgency=medium

  * Fix binary and binary-indep targets to work as intended (LP: #1850851)

ubuntu-release-upgrader (1:19.10.15.1) eoan; urgency=medium

  * do-release-upgrade: also check for linux-image- in reboot-required.pkgs,
    refusing upgrading before a reboot is performed. (LP: #1844805)

ubuntu-release-upgrader (1:19.10.15) eoan; urgency=medium

  * Don't check for validity of /usr/bin/python symlink because this is no
    longer relevant to the upgrades (and the printed error message was
    inaccurate). LP: #1825655.
  * Clean up remaining references to /usr/bin/python (tests and utilities)
    in the source.

ubuntu-release-upgrader (1:19.10.14) eoan; urgency=medium

  * DistUpgrade/DistUpgradeCache.py: New python3-apt bindings no longer provide
    Package.section, so switch to using Package.{candidate,installed}.section
    whenever applicable. This fixes broken partial upgrades. (LP: #1841675)

ubuntu-release-upgrader (1:19.10.13) eoan; urgency=medium

  [ Brian Murray ]
  * DistUpgrade/DistUpgradeQuirks.py: do not include gnome-system-monitor in
    the deb to snap transition anymore. (LP: #1845485)
  * DistUpgrade/DistUpgradeQuirks.py: improve wording regarding not being able
    to upgrade from a system with an apt architecture of i386.

  [ Łukasz 'sil2100' Zemczak ]
  * DistUpgrade/DistUpgradeQuirks.py: slightly refactor code to fix related
    unit tests.

 -- Adam Conrad <email address hidden> Thu, 31 Oct 2019 13:38:35 -0600

[Adam Conrad (adconrad)]

The verification of the Stable Release Update for ubuntu-release-upgrader has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.