do-release-upgrade (19.04 -> 19.10) aborts when snapd is disabled

[Chippo Elder]

Running sudo do-release-upgrade I get:
=========================
Checking for a new Ubuntu release
Get:1 Upgrade tool signature [1,554 B]
Get:2 Upgrade tool [1,319 kB]
Fetched 1,320 kB in 6s (21.3 kB/s)
authenticate 'eoan.tar.gz' against 'eoan.tar.gz.gpg'
extracting 'eoan.tar.gz'

Reading cache

Checking package manager
Reading package lists... Done
Building dependency tree
Reading state information... Done
Hit http://archive.ubuntu.com/ubuntu disco InRelease
Hit http://archive.canonical.com/ubuntu disco InRelease
Get:1 http://archive.ubuntu.com/ubuntu disco-updates InRelease [97.5 kB]
Get:2 http://archive.ubuntu.com/ubuntu disco-security InRelease [97.5 kB]
Get:3 http://archive.ubuntu.com/ubuntu disco-backports InRelease [88.8 kB]
Fetched 284 kB in 0s (0 B/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done

Restoring original system state

Aborting
Reading package lists... Done
Building dependency tree
Reading state information... Done
=========================

I've had the problem for several days, but the system otherwise seems completely cool. I keep getting and successfully installing all the usual security updates (firefox this morning).

The only logs which seem relevant are in /var/log/dist-upgrade/main.log
=========================
2019-10-24 14:15:14,716 INFO Using config files '['./DistUpgrade.cfg']'
2019-10-24 14:15:14,716 INFO uname information: 'Linux chippo-Aspire-V3-731 5.0.0-32-generic #34-Ubuntu SMP Wed Oct 2 02:06:48 UTC 2019 x86_64'
2019-10-24 14:15:15,262 INFO apt version: '1.8.3'
2019-10-24 14:15:15,262 INFO python version: '3.7.3 (default, Oct 7 2019, 12:56:13)
[GCC 8.3.0]'
2019-10-24 14:15:15,267 INFO release-upgrader version '19.10.12' started
2019-10-24 14:15:15,279 INFO locale: 'en_ZA' 'UTF-8'
2019-10-24 14:15:15,344 INFO screen could not be run
2019-10-24 14:15:15,388 DEBUG Using 'DistUpgradeViewText' view
2019-10-24 14:15:15,451 DEBUG enable dpkg --force-overwrite
2019-10-24 14:15:15,495 DEBUG creating statefile: '/var/log/dist-upgrade/apt-clone_system_state.tar.gz'
2019-10-24 14:15:25,436 DEBUG lsb-release: 'disco'
2019-10-24 14:15:25,436 DEBUG _pythonSymlinkCheck run
2019-10-24 14:15:25,438 DEBUG openCache()
2019-10-24 14:15:25,438 DEBUG quirks: running PreCacheOpen
2019-10-24 14:15:25,438 DEBUG running Quirks.PreCacheOpen
2019-10-24 14:15:26,288 DEBUG /openCache(), new cache size 95789
2019-10-24 14:15:26,288 DEBUG need_server_mode(): run in 'desktop' mode, (because of pkg 'ubuntu-desktop')
2019-10-24 14:15:26,288 DEBUG checkViewDepends()
2019-10-24 14:15:26,290 DEBUG running doUpdate() (showErrors=False)
2019-10-24 14:15:29,239 DEBUG openCache()
2019-10-24 14:15:30,217 DEBUG /openCache(), new cache size 95789
2019-10-24 14:15:30,217 DEBUG doPostInitialUpdate
2019-10-24 14:15:30,217 DEBUG quirks: running eoanPostInitialUpdate
2019-10-24 14:15:30,217 DEBUG running Quirks.eoanPostInitialUpdate
2019-10-24 14:15:35,238 DEBUG abort called
2019-10-24 14:15:35,238 DEBUG openCache()
2019-10-24 14:15:36,226 DEBUG /openCache(), new cache size 95789
=========================

lsb_release -rd
Description: Ubuntu 19.04
Release: 19.04

apt-cache policy ubuntu-release-upgrader-core
ubuntu-release-upgrader-core:
  Installed: 1:19.04.16.7
  Candidate: 1:19.04.16.7
  Version table:
 *** 1:19.04.16.7 500
        500 http://archive.ubuntu.com/ubuntu disco-updates/main amd64 Packages
        500 http://archive.ubuntu.com/ubuntu disco-updates/main i386 Packages
        100 /var/lib/dpkg/status
     1:19.04.16 500
        500 http://archive.ubuntu.com/ubuntu disco/main amd64 Packages
        500 http://archive.ubuntu.com/ubuntu disco/main i386 Packages
---
ProblemType: Bug
ApportVersion: 2.20.10-0ubuntu27.2
Architecture: amd64
CrashDB: ubuntu
CurrentDesktop: LXDE
DistroRelease: Ubuntu 19.04
InstallationDate: Installed on 2016-12-05 (1059 days ago)
InstallationMedia: Ubuntu 16.10 "Yakkety Yak" - Release amd64 (20161012.2)
Package: ubuntu-release-upgrader (not installed)
ProcVersionSignature: Ubuntu 5.0.0-34.36-generic 5.0.21
Tags: disco dist-upgrade
Uname: Linux 5.0.0-34-generic x86_64
UpgradeStatus: Upgraded to disco on 2019-10-31 (0 days ago)
UserGroups: adm cdrom dip lpadmin plugdev sambashare sudo vboxusers
VarLogDistupgradeAptlog:
 Log time: 2019-10-31 14:27:58.805832
 Log time: 2019-10-31 14:28:05.066042
 Log time: 2019-10-31 14:28:11.218874
_MarkForUpload: True

[Chippo Elder (chiphog)]

Arul in bug <https://bugs.launchpad.net/ubuntu/+source/ubuntu-release-upgrader/+bug/1848829> confirms this bug, but that bug seems to be different from this one. So, just check that bug too, for fixes/workarounds etc.

[Chippo Elder (chiphog)]

Here are some things that have helped others with similar problems.

1) Upgrade ubuntu-release-upgrader to version 1:19.10.15.
2) cd /usr/bin; rm python; ln -s python2 python
3) Uninstall distro-info 0.0.0 Python package via pip. Install/reinstall distro-info package from Ubuntu repos.

They didn't help me; may you have better luck.

From:
https://bugs.launchpad.net/ubuntu/+source/ubuntu-release-upgrader/+bug/1849704
https://bugs.launchpad.net/ubuntu/+source/ubuntu-release-upgrader/+bug/1848829

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Arul (alarulrajan)]

Yes. I have exact same bug. Still unable to upgrade :(

[Brian Murray (brian-murray)]

Could you please run 'apport-collect 1849669' in a terminal? This will collect some log files from your system which will help sorting out what is going on. Additionally, if you could include the output of 'apt-cache policy distro-info' that would be great.

[Chippo Elder (chiphog)]

distro-info:
  Installed: (none)
  Candidate: 0.21ubuntu2
  Version table:
     0.21ubuntu2 500
        500 http://za.archive.ubuntu.com/ubuntu disco/main amd64 Packages

[Chippo Elder (chiphog)]

I did the 'apport-collect 1849669' and gave permission (here and on my desktop) and everything seemed to go fine. I was expecting to see various log files and info about my system to appear here, in this bug report. Which I don't. Should I run it again?

[Chippo Elder (chiphog)]

I should also mention that when I did run 'apport-collect 1849669', I got the following error:
dpkg-query: no packages found matching ubuntu-release-upgrader

Which seems correct 'cos I don't have it installed. I have:
ubuntu-release-upgrader-core
ubuntu-release-upgrader-gtk
ubuntu-release-upgrader-qt
All are version 1:19.10.15.

And when I raised this bug, I'd already done a "dpkg -S /usr/bin/do-release-upgrade" and learned that that program was in ubuntu-release-upgrader-core. This website wouldn't let me raise a bug report for ubuntu-release-upgrader-core, because that package didn't exist (according to it). It did suggest that I used package ubuntu-release-upgrader.

I pondered about this for a while, and then I saw that the source package ubuntu-release-upgrader created the 3 binary packages listed above, and decided that it was OK.

[Chippo Elder (chiphog)]

I don't know why you wanted the output of 'apt-cache policy distro-info', but seeing as it was missing, I installed it.

That didn't change anything about the results of the attempted upgrade. It did run the upgrade, now, in screen, which is damn cool. It should have done that before I installed distro-info.

[Arul (alarulrajan)]

distro-info:
  Installed: 0.21ubuntu2
  Candidate: 0.21ubuntu2
  Version table:
 *** 0.21ubuntu2 500
        500 http://in.archive.ubuntu.com/ubuntu disco/main amd64 Packages
        100 /var/lib/dpkg/status

This is what i got

[Chippo Elder (chiphog)]

@Arul: Brian asked me to run "apport-collect 1849669" above. Also above, you'll see that it didn't work for me. However, it might work for you. Why don't you try it?

[Chippo Elder (chiphog)]

apport information

[Chippo Elder (chiphog)]

apport information

[Chippo Elder (chiphog)]

apport information

[Chippo Elder (chiphog)]

apport information

[Chippo Elder (chiphog)]

apport information

[Chippo Elder (chiphog)]

apport information

[Chippo Elder (chiphog)]

apport information

[Chippo Elder (chiphog)]

Aaah! Finally, my "apport-collect 1849669" is working. Must have been those python3-problem-report updates I got earlier. After dozens of updates from Ubuntu, sudo do-release-upgrade still doesn't work. Behaviour unchanged.

[Arul (alarulrajan)]

apport-collect 1849669 says the below...

"You are not the reported or subscriber of this problem report, or the report is a duplicate or already closed.

Please create a new report using apport-bug"

[Arul (alarulrajan)]

i some how upgraded with reinstalling all *ubuntu-desktop versions and reinstalling gdm3/lightdm/python3 etc.. which one exactly worked.. i did't know..

[Brian Murray (brian-murray)]

@Chippo - please try running 'do-release-upgrade -p' this will use the dist-upgrader tarball from -proposed. We recently discovered that not all the fixes we thought were in the dist-upgrader were actually being included. Thanks in advance!

[Chippo Elder (chiphog)]

Wilco. Should I have 'disco-proposed' enabled in software-properties-gtk?

[Brian Murray (brian-murray)]

On Fri, Nov 01, 2019 at 07:19:33PM -0000, Chippo Elder wrote:
> Wilco. Should I have 'disco-proposed' enabled in software-properties-
> gtk?

Nope.

--
Brian Murray

[Chippo Elder (chiphog)]

@Brian: behaviour is unchanged (from without the -p):
===========================
root@chippo-Aspire-V3-731:~# do-release-upgrade -p
Checking for a new Ubuntu release
Get:1 Upgrade tool signature [1,554 B]
Get:2 Upgrade tool [1,327 kB]
Fetched 1,328 kB in 0s (0 B/s)
authenticate 'eoan.tar.gz' against 'eoan.tar.gz.gpg'
extracting 'eoan.tar.gz'
...
...
...
Hit http://archive.ubuntu.com/ubuntu disco-security InRelease
Hit http://archive.ubuntu.com/ubuntu disco-backports InRelease
Fetched 0 B in 0s (0 B/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done

Restoring original system state

Aborting
Reading package lists... Done
Building dependency tree
Reading state information... Done
=== Command detached from window (Fri Nov 1 22:12:43 2019) ===
=== Command terminated with exit status 1 (Fri Nov 1 22:12:53 2019) ===
===========================

I tried this with and without disco-proposed enabled. I also changed to the Main Server, in case your changes hadn't propagated. I also tried with -d and without any option. All no go, but other than the upgrade failure the system seems fine and I get security updates daily, without trouble.

[Brian Murray (brian-murray)]

Try 'DEBUG_UPDATE_MANAGER=1 do-release-upgrade -p'. Thanks again for testing!

[Chippo Elder (chiphog)]

I get this:
===========================
root@chippo-Aspire-V3-731:~# DEBUG_UPDATE_MANAGER=1 do-release-upgrade -p
Checking for a new Ubuntu release
MetaRelease.__init__() useDevel=False useProposed=True
/etc/update-manager/meta-release: https://changelogs.ubuntu.com/meta-release
/etc/update-manager/meta-release: https://changelogs.ubuntu.com/meta-release-lts
/etc/update-manager/meta-release: -development
/etc/update-manager/meta-release: -proposed
metarelease-uri: https://changelogs.ubuntu.com/meta-release-proposed
MetaRelease.download()
reading file '/var/lib/update-manager/meta-release-proposed'
have self.metarelease_information
MetaRelease.parse()
current dist name: 'disco'
found distro name: 'warty'
found distro name: 'hoary'
found distro name: 'breezy'
found distro name: 'dapper'
found distro name: 'edgy'
found distro name: 'feisty'
found distro name: 'gutsy'
found distro name: 'hardy'
found distro name: 'intrepid'
found distro name: 'jaunty'
found distro name: 'karmic'
found distro name: 'lucid'
found distro name: 'maverick'
found distro name: 'natty'
found distro name: 'oneiric'
found distro name: 'precise'
found distro name: 'quantal'
found distro name: 'raring'
found distro name: 'saucy'
found distro name: 'trusty'
found distro name: 'utopic'
found distro name: 'vivid'
found distro name: 'wily'
found distro name: 'xenial'
found distro name: 'yakkety'
found distro name: 'zesty'
found distro name: 'artful'
found distro name: 'bionic'
found distro name: 'cosmic'
found distro name: 'disco'
found distro name: 'eoan'
new dist: <UpdateManager.Core.MetaRelease.Dist object at 0x7fd7c1cb3fd0>
trying to find suitable mirror
mirror_from_sources_list: disco
url_downloadable: http://za.archive.ubuntu.com/ubuntu//dists/eoan-proposed/main/dist-upgrader-all/current/eoan.tar.gz.gpg
s='http' n='za.archive.ubuntu.com' p='/ubuntu//dists/eoan-proposed/main/dist-upgrader-all/current/eoan.tar.gz.gpg' q='' f=''
trying to find suitable mirror
mirror_from_sources_list: disco
url_downloadable: http://za.archive.ubuntu.com/ubuntu//dists/eoan-proposed/main/dist-upgrader-all/current/eoan.tar.gz
s='http' n='za.archive.ubuntu.com' p='/ubuntu//dists/eoan-proposed/main/dist-upgrader-all/current/eoan.tar.gz' q='' f=''
0% [Connecting to za.archive.ubuntu.com (197.155.77.2)]
GET /ubuntu//dists/eoan-proposed/main/dist-upgrader-all/current/eoan.tar.gz.gpg HTTP/1.1
Host: za.archive.ubuntu.com
User-Agent: Debian APT-HTTP/1.3 (1.8.4)

Answer for: http://za.archive.ubuntu.com/ubuntu//dists/eoan-proposed/main/dist-upgrader-all/current/eoan.tar.gz.gpg
HTTP/1.1 200 OK
Server: nginx
Date: Fri, 01 Nov 2019 22:12:44 GMT
Content-Type: application/octet-stream
Content-Length: 1554
Last-Modified: Thu, 31 Oct 2019 22:08:11 GMT
Connection: keep-alive
ETag: "5dbb5b4b-612"
Accept-Ranges: bytes

GET /ubuntu//dists/eoan-proposed/main/dist-upgrader-all/current/eoan.tar.gz HTTP/1.1
Host: za.archive.ubuntu.com
User-Agent: Debian APT-HTTP/1.3 (1.8.4)

Answer for: http://za.archive.ubuntu.com/ubuntu//dists/eoan-proposed/main/dist-upgrader-all/current/eoan.tar.gz
HTTP/1.1 200 OK
Server: nginx
Date: Fri, 01 Nov 2019 22:12:44 GMT
Content-Type: application/x-gzip
Content-Length: 1326858
Last-Mo...

[Chippo Elder (chiphog)]

I found the rest of the output in screen. It's the same as before:
===========================
Reading cache

Checking package manager
Reading package lists... Done
Building dependency tree
Reading state information... Done
Hit http://linux.teamviewer.com/deb stable InRelease
Hit https://dl.yarnpkg.com/debian stable InRelease
Hit http://za.archive.ubuntu.com/ubuntu disco InRelease
Hit http://za.archive.ubuntu.com/ubuntu disco-updates InRelease
Hit http://archive.canonical.com/ubuntu disco InRelease
Hit http://za.archive.ubuntu.com/ubuntu disco-security InRelease
Hit http://za.archive.ubuntu.com/ubuntu disco-backports InRelease
Hit http://ppa.launchpad.net/lutris-team/lutris/ubuntu disco InRelease
Hit http://za.archive.ubuntu.com/ubuntu disco-proposed InRelease
Ign http://dl.google.com/linux/earth/deb stable InRelease
Hit http://ppa.launchpad.net/rael-gc/rvm/ubuntu disco InRelease
Hit http://ppa.launchpad.net/stjr/srb2/ubuntu disco InRelease
Hit http://debian.drdteam.org stable InRelease
Hit http://ppa.launchpad.net/ubuntuhandbook1/avidemux/ubuntu disco InRelease
Hit http://dl.google.com/linux/earth/deb stable Release
Hit https://deb.nodesource.com/node_12.x disco InRelease
Fetched 0 B in 0s (0 B/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done

Restoring original system state

Aborting
Reading package lists... Done
Building dependency tree
Reading state information... Done
===========================

[Chippo Elder (chiphog)]

@Brian: I see that my tests aren't really giving you any decent info to solidly work on. Would you like ssh access to my box (there isn't anything too private on it)? That way you can do a strace on do-release-upgrade and other things that are difficult to explain over email to lusers (like me).

If that's of interest to you, email me a public key and I'll work out how to configure an incoming port on my girlfriend's router.

[Chippo Elder (chiphog)]

I'm upgrading right now! Yay!!!

I followed my own advice and started doing straces on upgrade related programs. My breakthrough came when less'ing through the output of:
strace -f -s 200 update-manager > /tmp/um.strace 2>&1 &
I saw these lines in the dump just before it starts closing dozens of files and munmapping dozens of memory areas:
----------
[pid 25685] connect(5, {sa_family=AF_UNIX, sun_path="/run/snapd.socket"}, 20) = -1 ENOENT (No such file or directory)
pid 25685] write(2, "error: cannot communicate with server: Get http://localhost/v2/debug?aspect=connectivity: dial unix /run/snapd.socket: connect: no such file or directory\n", 154 <unfinished ...>
----------

Snapd? I remember snapd. I turned it off many months ago, when I was cleaning up my boot process (which used to take 10 minutes, with all the rubbish I was running). It seemed (at the time) that snapd was yet another jeOS installer for games, like lutris and steam-installer. So I turned it off and nothing bad happened - 'till trying to upgrade.

I can't remember how I turned it off, but I woke it up a bit brute-force-ishly with:
systemctl --system enable snapd.autoimport.service snapd.core-fixup.service snapd.seeded.service snapd.service snapd.system-shutdown.service snapd.socket snapd.snap-repair.timer
systemctl --system start snapd.autoimport.service snapd.core-fixup.service snapd.seeded.service snapd.service snapd.system-shutdown.service snapd.socket snapd.snap-repair.timer

So, with an early python test for /run/snapd.socket plus an error message that mentions SNAPD, I think we can close this bug.

[Etienne URBAH (eurbah)]

When snapd is not in a good state, do-release-upgrade systematically fails on upgrade from eoan (19.10) to fossa (20.04).

The only messages displayed on the terminal by do-release-upgrade are :
Restoring original system state
Aborting

Before using snapd, I think that do-release-upgrade must test if snapd is in a good state, and display an explicit error message if not.

For me, the successful workaround was to uninstall then reinstall the snapd package.

[Brian Murray (brian-murray)]

Could you explain a bit about how snapd was in a "not good state"? What error messages did you see?

[Etienne URBAH (eurbah)]

snapd messages inside syslog :

systemd[19415]: Listening on REST API socket for snapd user session agent.
systemd[19415]: snapd.session-agent.socket: Succeeded.
systemd[19415]: Closed REST API socket for snapd user session agent.
snapd[1729]: AppArmor status: apparmor is enabled and all features are available
snapd[1729]: cannot run daemon: cannot read state: invalid character '\x13' in string literal
systemd[1]: snapd.service: Main process exited, code=exited, status=1/FAILURE
systemd[1]: snapd.service: Failed with result 'exit-code'.