error: cannot find current revision for snap subiquity: readlink /snap/subiquity/current: no such file or directory

[dann frazier]

I booted the 20200728.1 ISO on an arm64 server and selected "Install Ubuntu Server" but, when I ssh'd in, it reported the above error. A few notes
 - I'm doing the install using a BMC-emulated CD-ROM device, so accesses are relatively slow, possibly holding race windows open longer than normal.
 - While I verified the sha256sum of the ISO file externally, I cancelled the runtime integrity
   check because it just takes too long in this install type.
 - I was not able to reproduce on a subsequent attempt, but I'll try a few more times.

$ ssh installer@<serverip1>
Warning: Permanently added '<serverip1>' (ECDSA) to the list of known hosts.
installer@<serverip1>'s password:
Welcome to Ubuntu 20.04.1 LTS (GNU/Linux 5.4.0-42-generic aarch64)

 * Documentation: https://help.ubuntu.com
 * Management: https://landscape.canonical.com
 * Support: https://ubuntu.com/advantage

  System information as of Wed Jul 29 16:16:00 UTC 2020

  System load: 13.52 Processes: 2272
  Usage of /home: unknown Users logged in: 0
  Memory usage: 1% IPv4 address for enp9s0f0: <serverip1>
  Swap usage: 0% IPv4 address for enp9s0f1: <serverip2>
  Temperature: 46.1 C

0 updates can be installed immediately.
0 of these updates are security updates.

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

error: cannot find current revision for snap subiquity: readlink /snap/subiquity/current: no such file or directory
Shared connection to <serverip1> closed.

[Michael Hudson-Doyle (mwhudson)]

I think that's a message from snapd. Is it possible you ssh-ed in before snap seeding had completed? (A process which I imagine is also rather slow in this setup)

[dann frazier (dannf)]

Maybe - I don't know how to confirm. I can say that a second attempt to ssh into that install also failed with the same error, and that was 346s later.

[Michael Hudson-Doyle (mwhudson)]

Hmm. I wonder if this is the same as bug 1888497 then.

[dann frazier (dannf)]

In case it helps, here's the log of the console prior to me ssh'ing in.

[Ubuntu QA Website (ubuntuqa)]

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
http://iso.qa.ubuntu.com/qatracker/reports/bugs/1889466

[dann frazier (dannf)]

On Wed, Jul 29, 2020 at 2:40 PM Michael Hudson-Doyle
<email address hidden> wrote:
>
> Hmm. I wonder if this is the same as bug 1888497 then.

I haven't seen that exact symptom - however, I have seen something similar.
*Most* of the time, the serial console ends here, and I ssh in to continue:

[ 1727.688186] cloud-init[4492]: Set the following 'random' passwords
[ 1727.704128] cloud-init[4492]: installer:6MqFhjF8JNehXbHttvva
ci-info: no authorized SSH keys fingerprints found for user installer.
<14>Jul 30 19:38:25 ec2:
<14>Jul 30 19:38:25 ec2:
#############################################################
<14>Jul 30 19:38:25 ec2: -----BEGIN SSH HOST KEY FINGERPRINTS-----
<14>Jul 30 19:38:25 ec2: 1024
SHA256:SSW96YInIbi59aBCSd+4wphIyZ9Owl3xnk8pHRW7170 root@ubuntu-server
(DSA)
<14>Jul 30 19:38:25 ec2: 256
SHA256:ARVfj7WP1u7jjbvHShX7TX19znZNoRq/W+gw+Ucywvs root@ubuntu-server
(ECDSA)
<14>Jul 30 19:38:25 ec2: 256
SHA256:TLjJ6EBMbrdHepDPU5TAprbOfHvQoXrgZ62dDCk5kw8 root@ubuntu-server
(ED25519)
<14>Jul 30 19:38:25 ec2: 3072
SHA256:bH73OUJKls0/2slJTjVXFRQD4WSx5ljZKR8V1xpStUg root@ubuntu-server
(RSA)
<14>Jul 30 19:38:25 ec2: -----END SSH HOST KEY FINGERPRINTS-----
<14>Jul 30 19:38:25 ec2:
#############################################################
-----BEGIN SSH HOST KEY KEYS-----
ecdsa-sha2-nistp256
AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOIasHO6Iub9x7GpCVrwsf2jBBoT7ONYl777lGQIF/esfgXbw53n8kO3QKqkxYWIdh3D08F/7HZsUpgGZElBWbo=
root@ubuntu-server
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILUbnXrMhWnSnZTbRKAFONCNhp7YosY2W7tyjiE5vQ6M
root@ubuntu-server
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCj4CXbgndKoJRb4tnlhxb0cH5fRrEd8GEQXaifLxqNT7SStHitk49xZap8Mw4mzm18c4D4GK0uvdEn+frc1KU58WOmfmylCZOCz+K3yBdF+HxHMul+pREJCHCkSaIyyx8GiJ986IMOOA1nudJABMVQ2kXZPlSEjCgEIsDnQI/ogo6JfOCSRlznfXR5aVI8Y2XJocbMqI2vbG3Svsa5RYaVR7soTiZlqOAXVPfh+1n0eIdOjAmuBryPvkPGLUKxQJy1dSI7uTtb1jsUMUDvXHwGgo1m8kcuuBH7WqQuQarMsEsMWFOtobomZ9bsQYU5mWUZoz5Z2Ye87ilCiHY+yB71HkaiEr+/cT7t2zDRytwfSHrEjW8Pwuhi5+VXjB0TMKvHj8CGQ1wKQIqXMsVyt6MfY5UKhjCCTDHDxJdr2UaafrSd6XyLat0wYHLpVEUCoCSR7hvnL7jtJiJA5FNGwGROuUAmTQ8bFSOzGbFUfFFB7DuMz1zjbY+NHjxpM1wxV08=
root@ubuntu-server
-----END SSH HOST KEY KEYS-----
[ 1792.534519] cloud-init[4657]: Cloud-init v.
20.2-45-g5f7825e2-0ubuntu1~20.04.1 running 'modules:final' at Thu, 30
Jul 2020 19:38:25 +0000. Up 1792.14 seconds.
[ 1792.556155] cloud-init[4657]: ci-info: no authorized SSH keys
fingerprints found for user installer.
[ 1792.572145] cloud-init[4657]: Cloud-init v.
20.2-45-g5f7825e2-0ubuntu1~20.04.1 finished at Thu, 30 Jul 2020
19:38:25 +0000. Datasource DataSourceNoCloud
[seed=/var/lib/cloud/seed/nocloud][dsmode=net]. Up 1792.52 seconds
[ 1792.600149] cloud-init[4657]: Welcome to Ubuntu Server Installer!
[ 1792.620182] cloud-init[4657]: Above you will find SSH host keys and
a random password set for the `installer` user. You can use these
credentials to ssh-in and complete the installation. If you provided
SSH keys in the cloud-init datasource, they were also provisioned to
the installer user.
[ 1792.652163] cloud-init[4657]: If you have access to the graphical
console, like TTY1 or HMC ASCII terminal you can complete the
installation there too.

But th...

[dann frazier (dannf)]

I reproduced this again, by attempting to ssh in as soon as I saw this message appear on the console:

[ 1727.688186] cloud-init[4492]: Set the following 'random' passwords
[ 1727.704128] cloud-init[4492]: installer:6MqFhjF8JNehXbHttvva

and ssh failed w/ "error: cannot find current revision for snap subiquity: readlink /snap/subiquity/current: no such file or directory"

I noticed at this time there was nothing on the graphical display either. I retried over several minutes, and continued to see this error. At one point, several minutes later, I saw the subiquity menu appear on the graphical display. I tried ssh'ing in again and voila, I also see the subiquity menu appear when ssh'ing in.

[dann frazier (dannf)]

So, to summarize, the problem seems to be that the serial console gives me instructions on how to SSH in and complete the installation. But it took, in my case, about 20 minutes before I could actually do so. In the meantime, ssh just failed with the "readlink /snap/subiquity/current" error.

Once I could ssh in, I was able to copy off /var/log - attached.

[Michael Hudson-Doyle (mwhudson)]

It looks like snap seeding is taking aaaaaaaaages on this system. The first line in the journal is from Jul 31 07:23:54, cloud-init finishes at Jul 31 07:38:25 (already appallingly slow) and snapd doesn't start mounting things until Jul 31 07:56:01. So yeah, things are very, terribly slow but I don't think there's much subiquity can do about this. I assume the BMC-emulated ISO is to blame but obviously can't be sure. You could try adding 'toram' to the kernel command line, which will copy the the ISO contents to ram before exiting the initramfs and see if that helps?

I think the groovy images have the preseeding optimization enabled now which also might help but I'm not sure of the current status of that work.

[dann frazier (dannf)]

Yeah, there are certainly performance improvements to be had - thanks for the 'toram' tip. (Perhaps it'd even make sense to have a toram option in the default GRUB menu?) But even if things were faster, it seems like there's still a race that should be fixed. That is, would it be possible to block the "go ahead and ssh in" message until the snap is ready, and/or make the ssh rejection provide a "Still working, try again later" message instead of the opaque readlink error message?

[Avio (aviopene)]

I have the same problem: I've just uninstalled snap VLC (because it can't play video files outside my home directory, so it's pretty useless) and installed standard VLC packaged by Ubuntu (20.04 amd64). And now I get:

error: cannot find current revision for snap vlc: readlink /snap/vlc/current: no such file or directory

This makes no sense to me given that `whereis vlc` gives:

vlc: /usr/bin/vlc /usr/lib/x86_64-linux-gnu/vlc /usr/share/vlc /usr/share/man/man1/vlc.1.gz

and that `/snap/bin` is one of the last directories in $PATH

UPDATE: apt purging snapd causes:

Command 'snap' not found, but can be installed with:

sudo apt install snapd

There is something pretty wrong here...

[Avio (aviopene)]

Ok, sorry, my fault, I forgot I put an alias for vlc: `snap run vlc`