subiquity

Bug #1888497
Comment #0

Comment 0 for bug 1888497

Revision history for this message

Frank Heimes (fheimes) wrote on 2020-07-22: Sporadically the installer system does not boot-up completely

From time to time (sporadically and very rarely - maybe in one or two attempt out of twenty) I face a situation where the installer system (on s390x) does not boot-up completely.

This happened to me in the past already, but since it happened only one or twice I thought it's due to resource constraints on the system or so.
But since I faced it now again on LPAR (before it was on z/VM), I'm opening this ticket now.

In the latest case I used the focal daily live image from the 20th of July with installer 20.06.1 (but that also happened with previous versions).

The situation is like this:

In boot of the installer ends here (LPAR):
...
"[ 128.200711] cloud-init[1375]: The key's randomart image is:"
"[ 128.200735] cloud-init[1375]: +--[ED25519 256]--+"
"[ 128.200758] cloud-init[1375]: |o .....ooo |"
"[ 128.200781] cloud-init[1375]: |.= . . +. o. .|"
"[ 128.200804] cloud-init[1375]: |+ * . . * o.o . |"
"[ 128.200826] cloud-init[1375]: |.= o . = = = + |"
"[ 128.200849] cloud-init[1375]: |o + . S o + |"
"[ 128.200876] cloud-init[1375]: | + o = . . |"
"[ 128.200900] cloud-init[1375]: | o + . |"
"[ 128.200925] cloud-init[1375]: | o.=. E |"
"[ 128.200947] cloud-init[1375]: | .+.o+o. |"
"[ 128.200977] cloud-init[1375]: +----[SHA256]-----+"
"[ 138.898906] cloud-init[2217]: Cloud-init v. 20.2-45-g5f7825e2-0ubuntu1~20.04."
"1 running 'modules:config' at Wed, 22 Jul 2020 11:27:39 +0000. Up 138.77 seconds"
.
"[ 138.898966] cloud-init[2217]: Set the following 'random' passwords"
"[ 138.899001] cloud-init[2217]: installer:aecmewaoicnai"

But it is not complete at this point.
A completed boot of the installer system ends like this:

"It is possible to connect to the installer over the network, which"
"might allow the use of a more capable terminal and can offer more languages"
"than can be rendered in the Linux console."

"To connect, SSH to installer@<IP address>."

"The password you should use is "ydnjdnciu"kZ4tR4vRvPxHPerCNU8g""."

"The host key fingerprints are:"

"RSA SHA256:n+6TJsfdCBII2PO89GMU10mG1oFvEIFBT2v6uPN0Jz0"
"ECDSA SHA256:VcDS5ac8xswXxFEghjo1ZIcue38AM6HJg0poIxdeeec"
"ED25519 SHA256:lb/DVVhj1obDPhfo3M8oPqeAyduvlLcPFJCC8ZaiCJY"

"Ubuntu 20.04 LTS ubuntu-server sclp_line0"

In such an above situation I also can't reach the subiquity UI:

fheimes@T570:~$ ssh-keygen -f "/home/fheimes/.ssh/known_hosts" -R s1lp14
# Host s1lp14 found: line 165
/home/fheimes/.ssh/known_hosts updated.
Original contents retained as /home/fheimes/.ssh/known_hosts.old
fheimes@T570:~$ ssh installer@s1lp14
The authenticity of host 's1lp14 (10.245.236.14)' can't be established.
ECDSA key fingerprint is SHA256:AoYrJhWOFaaEtUun/aZAT8/VgNP+yq//Jhg7vzUwvxE.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 's1lp14,10.245.236.14' (ECDSA) to the list of known hosts.
installer@s1lp14's password:
Welcome to Ubuntu 20.04 LTS (GNU/Linux 5.4.0-40-generic s390x)

* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage

System information as of Wed Jul 22 11:28:32 UTC 2020

System load: 0.44 Memory usage: 4% Processes: 180
Usage of /home: unknown Swap usage: 0% Users logged in: 0

0 updates can be installed immediately.
0 of these updates are security updates.

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

Hence even gathering the logs is unfortunately not easily possible.

From time to time (sporadically and very rarely - maybe in one or two attempt out of twenty) I face a situation where the installer system (on s390x) does not boot-up completely.

In the latest case I used the focal daily live image from the 20th of July with installer 20.06.1 (but that also happened with previous versions).

The situation is like this:

In boot of the installer ends here (LPAR):
...
"[  128.200711] cloud-init[1375]: The key's randomart image is:"
"[  128.200735] cloud-init[1375]: +--[ED25519 256]--+"
"[  128.200758] cloud-init[1375]: |o       .....ooo |"
"[  128.200781] cloud-init[1375]: |.= .   . +. o.  .|"
"[  128.200804] cloud-init[1375]: |+ * . . * o.o .  |"
"[  128.200826] cloud-init[1375]: |.= o . = = = +   |"
"[  128.200849] cloud-init[1375]: |o   + . S o +    |"
"[  128.200876] cloud-init[1375]: |   + o = . .     |"
"[  128.200900] cloud-init[1375]: |    o + .        |"
"[  128.200925] cloud-init[1375]: |     o.=. E      |"
"[  128.200947] cloud-init[1375]: |     .+.o+o.     |"
"[  128.200977] cloud-init[1375]: +----[SHA256]-----+"
"[  138.898906] cloud-init[2217]: Cloud-init v. 20.2-45-g5f7825e2-0ubuntu1~20.04."
"1 running 'modules:config' at Wed, 22 Jul 2020 11:27:39 +0000. Up 138.77 seconds"
.
"[  138.898966] cloud-init[2217]: Set the following 'random' passwords"
"[  138.899001] cloud-init[2217]: installer:aecmewaoicnai"

But it is not complete at this point.
A completed boot of the installer system ends like this:

"It is possible to connect to the installer over the network, which"
"might allow the use of a more capable terminal and can offer more languages"
"than can be rendered in the Linux console."

"To connect, SSH to installer@<IP address>."

"The password you should use is "ydnjdnciu"kZ4tR4vRvPxHPerCNU8g""."

"The host key fingerprints are:"

"RSA     SHA256:n+6TJsfdCBII2PO89GMU10mG1oFvEIFBT2v6uPN0Jz0"
"ECDSA   SHA256:VcDS5ac8xswXxFEghjo1ZIcue38AM6HJg0poIxdeeec"
"ED25519 SHA256:lb/DVVhj1obDPhfo3M8oPqeAyduvlLcPFJCC8ZaiCJY"

"Ubuntu 20.04 LTS ubuntu-server sclp_line0"

In such an above situation I also can't reach the subiquity UI:

fheimes@T570:~$ ssh-keygen -f "/home/fheimes/.ssh/known_hosts" -R s1lp14
# Host s1lp14 found: line 165
/home/fheimes/.ssh/known_hosts updated.
Original contents retained as /home/fheimes/.ssh/known_hosts.old
fheimes@T570:~$ ssh installer@s1lp14
The authenticity of host 's1lp14 (10.245.236.14)' can't be established.
ECDSA key fingerprint is SHA256:AoYrJhWOFaaEtUun/aZAT8/VgNP+yq//Jhg7vzUwvxE.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 's1lp14,10.245.236.14' (ECDSA) to the list of known hosts.
installer@s1lp14's password: 
Welcome to Ubuntu 20.04 LTS (GNU/Linux 5.4.0-40-generic s390x)

* Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

System information as of Wed Jul 22 11:28:32 UTC 2020

System load:    0.44      Memory usage: 4%   Processes:       180
  Usage of /home: unknown   Swap usage:   0%   Users logged in: 0

0 updates can be installed immediately.
0 of these updates are security updates.

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

Hence even gathering the logs is unfortunately not easily possible.