| 2020-07-22 11:51:43 |
Frank Heimes |
bug |
|
|
added bug |
| 2020-07-22 11:51:55 |
Frank Heimes |
bug task added |
|
ubuntu-z-systems |
|
| 2020-07-22 11:52:33 |
Frank Heimes |
ubuntu-z-systems: assignee |
|
Canonical Foundations Team (canonical-foundations) |
|
| 2020-07-22 11:53:09 |
Frank Heimes |
bug |
|
|
added subscriber Michael Hudson-Doyle |
| 2020-07-22 11:53:18 |
Frank Heimes |
bug |
|
|
added subscriber Dimitri John Ledkov |
| 2020-07-22 11:53:51 |
Frank Heimes |
summary |
Sporadically the installer system does not boot-up completely |
Sporadically the installer system does not boot-up completely (or doesn't start all services) |
|
| 2020-07-31 15:23:01 |
Frank Heimes |
ubuntu-z-systems: importance |
Undecided |
High |
|
| 2020-12-08 07:42:37 |
Frank Heimes |
attachment added |
|
hwe0007-07122020.tgz https://bugs.launchpad.net/ubuntu-z-systems/+bug/1888497/+attachment/5441908/+files/hwe0007-07122020.tgz |
|
| 2020-12-10 08:49:49 |
Frank Heimes |
ubuntu-z-systems: status |
New |
In Progress |
|
| 2021-01-21 12:00:18 |
Frank Heimes |
subiquity: status |
New |
Fix Released |
|
| 2021-01-21 12:00:22 |
Frank Heimes |
ubuntu-z-systems: status |
In Progress |
Fix Released |
|
| 2021-04-20 07:35:03 |
Frank Heimes |
subiquity: status |
Fix Released |
New |
|
| 2021-04-20 07:35:06 |
Frank Heimes |
ubuntu-z-systems: status |
Fix Released |
New |
|
| 2021-04-20 08:23:22 |
Frank Heimes |
attachment added |
|
console_output.txt https://bugs.launchpad.net/ubuntu-z-systems/+bug/1888497/+attachment/5490069/+files/console_output.txt |
|
| 2021-04-21 06:14:45 |
Frank Heimes |
summary |
Sporadically the installer system does not boot-up completely (or doesn't start all services) |
Booting the install system does not always succeed, hence a remote ssh login is not always possible |
|
| 2021-04-21 09:12:21 |
Ubuntu QA Website |
tags |
installer s390x |
installer iso-testing s390x |
|
| 2021-04-22 07:45:18 |
Frank Heimes |
attachment added |
|
21.04-April_21st-bootlogs.zip https://bugs.launchpad.net/ubuntu-z-systems/+bug/1888497/+attachment/5490901/+files/21.04-April_21st-bootlogs.zip |
|
| 2021-05-05 05:41:37 |
Frank Heimes |
subiquity: status |
New |
In Progress |
|
| 2021-05-05 05:41:40 |
Frank Heimes |
ubuntu-z-systems: status |
New |
In Progress |
|
| 2021-05-05 05:45:51 |
Frank Heimes |
merge proposal linked |
|
https://code.launchpad.net/~mwhudson/livecd-rootfs/+git/livecd-rootfs/+merge/401501 |
|
| 2021-07-23 00:05:57 |
Michael Hudson-Doyle |
subiquity: status |
In Progress |
Invalid |
|
| 2021-07-23 00:06:04 |
Michael Hudson-Doyle |
bug task added |
|
livecd-rootfs (Ubuntu) |
|
| 2021-07-23 00:06:12 |
Michael Hudson-Doyle |
livecd-rootfs (Ubuntu): status |
New |
Fix Released |
|
| 2021-07-23 00:06:15 |
Michael Hudson-Doyle |
nominated for series |
|
Ubuntu Focal |
|
| 2021-07-23 00:06:15 |
Michael Hudson-Doyle |
bug task added |
|
livecd-rootfs (Ubuntu Focal) |
|
| 2021-07-23 00:09:58 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~mwhudson/livecd-rootfs/+git/livecd-rootfs/+merge/406126 |
|
| 2021-07-23 00:19:31 |
Michael Hudson-Doyle |
description |
From time to time (sporadically and very rarely - maybe in one or two attempt out of twenty) I face a situation where the installer system (on s390x) does not boot-up completely.
This happened to me in the past already, but since it happened only one or twice I thought it's due to resource constraints on the system or so.
But since I faced it now again on LPAR (before it was on z/VM), I'm opening this ticket now.
In the latest case I used the focal daily live image from the 20th of July with installer 20.06.1 (but that also happened with previous versions).
The situation is like this:
In boot of the installer ends here (LPAR):
...
"[ 128.200711] cloud-init[1375]: The key's randomart image is:"
"[ 128.200735] cloud-init[1375]: +--[ED25519 256]--+"
"[ 128.200758] cloud-init[1375]: |o .....ooo |"
"[ 128.200781] cloud-init[1375]: |.= . . +. o. .|"
"[ 128.200804] cloud-init[1375]: |+ * . . * o.o . |"
"[ 128.200826] cloud-init[1375]: |.= o . = = = + |"
"[ 128.200849] cloud-init[1375]: |o + . S o + |"
"[ 128.200876] cloud-init[1375]: | + o = . . |"
"[ 128.200900] cloud-init[1375]: | o + . |"
"[ 128.200925] cloud-init[1375]: | o.=. E |"
"[ 128.200947] cloud-init[1375]: | .+.o+o. |"
"[ 128.200977] cloud-init[1375]: +----[SHA256]-----+"
"[ 138.898906] cloud-init[2217]: Cloud-init v. 20.2-45-g5f7825e2-0ubuntu1~20.04."
"1 running 'modules:config' at Wed, 22 Jul 2020 11:27:39 +0000. Up 138.77 seconds"
.
"[ 138.898966] cloud-init[2217]: Set the following 'random' passwords"
"[ 138.899001] cloud-init[2217]: installer:aecmewaoicnai"
or another example (z/VM):
...
¬ 93.463680| cloud-init¬1282|: +--¬ED25519 256|--+
¬ 93.463713| cloud-init¬1282|: !Eo=o .... !
¬ 93.463749| cloud-init¬1282|: !.Bo.o ... o !
¬ 93.463782| cloud-init¬1282|: !**.*... o = !
¬ 93.463818| cloud-init¬1282|: !*=O o. o . . !
¬ 93.463849| cloud-init¬1282|: !**++ S !
¬ 93.463886| cloud-init¬1282|: !§o+.. !
¬ 93.463918| cloud-init¬1282|: !+*o. !
¬ 93.463954| cloud-init¬1282|: !.o. !
¬ 93.463988| cloud-init¬1282|: !. !
¬ 93.464028| cloud-init¬1282|: +----¬SHA256|-----+
¬ 104.841438| cloud-init¬2004|: Cloud-init v. 20.2-45-g5f7825e2-0ubuntu1ß20.04.
1 running 'modules:config' at Mon, 20 Jul 2020 10:46:38 +0000. Up 104.63 seconds
.
¬ 104.841490| cloud-init¬2004|: Set the following 'random' passwords
¬ 104.841516| cloud-init¬2004|: installer:U9NJDuvXFw6X2fxG7pP8
But it is not complete at this point.
A completed boot of the installer system ends like this:
"It is possible to connect to the installer over the network, which"
"might allow the use of a more capable terminal and can offer more languages"
"than can be rendered in the Linux console."
"To connect, SSH to installer@<IP address>."
"The password you should use is "ydnjdnciu"kZ4tR4vRvPxHPerCNU8g""."
"The host key fingerprints are:"
"RSA SHA256:n+6TJsfdCBII2PO89GMU10mG1oFvEIFBT2v6uPN0Jz0"
"ECDSA SHA256:VcDS5ac8xswXxFEghjo1ZIcue38AM6HJg0poIxdeeec"
"ED25519 SHA256:lb/DVVhj1obDPhfo3M8oPqeAyduvlLcPFJCC8ZaiCJY"
"Ubuntu 20.04 LTS ubuntu-server sclp_line0"
In such an above situation I also can't reach the subiquity UI:
fheimes@T570:~$ ssh-keygen -f "/home/fheimes/.ssh/known_hosts" -R s1lp14
# Host s1lp14 found: line 165
/home/fheimes/.ssh/known_hosts updated.
Original contents retained as /home/fheimes/.ssh/known_hosts.old
fheimes@T570:~$ ssh installer@s1lp14
The authenticity of host 's1lp14 (10.245.236.14)' can't be established.
ECDSA key fingerprint is SHA256:AoYrJhWOFaaEtUun/aZAT8/VgNP+yq//Jhg7vzUwvxE.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 's1lp14,10.245.236.14' (ECDSA) to the list of known hosts.
installer@s1lp14's password:
Welcome to Ubuntu 20.04 LTS (GNU/Linux 5.4.0-40-generic s390x)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
System information as of Wed Jul 22 11:28:32 UTC 2020
System load: 0.44 Memory usage: 4% Processes: 180
Usage of /home: unknown Swap usage: 0% Users logged in: 0
0 updates can be installed immediately.
0 of these updates are security updates.
The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.
Hence even gathering the logs is unfortunately not easily possible. |
[impact]
When there is a serial console configured, there was a unit cycle
serial-getty@$TTY.service -> cloud-final.service -> multi-user.target -> getty.target -> serial-getty@$TTY.service
(or something like that)
Depending on which unit systemd kills to resolve the cycle, this can result in cloud-init never completing which leads to the subiquity server waiting forever for it and nothing useful can be done (other than restarting and hoping for better luck next time).
Because subiquity itself waits for cloud-init (and this has been true for a long time now) there is no need for serial-getty@$TTY.service to wait on cloud-final.service.
[regression potential]
This change results in shuffling the systemd units around a fair bit, but the new arrangement has been tested in devel for a few months now and works well there. It's also much more straightforward than the current setup.
[test case]
This is a bit tricky as it's an intermittent failure. Basically, boot the live installer with a serial console configured a bunch of times and (10?) check that the installer starts up properly each time.
[original description]
From time to time (sporadically and very rarely - maybe in one or two attempt out of twenty) I face a situation where the installer system (on s390x) does not boot-up completely.
This happened to me in the past already, but since it happened only one or twice I thought it's due to resource constraints on the system or so.
But since I faced it now again on LPAR (before it was on z/VM), I'm opening this ticket now.
In the latest case I used the focal daily live image from the 20th of July with installer 20.06.1 (but that also happened with previous versions).
The situation is like this:
In boot of the installer ends here (LPAR):
...
"[ 128.200711] cloud-init[1375]: The key's randomart image is:"
"[ 128.200735] cloud-init[1375]: +--[ED25519 256]--+"
"[ 128.200758] cloud-init[1375]: |o .....ooo |"
"[ 128.200781] cloud-init[1375]: |.= . . +. o. .|"
"[ 128.200804] cloud-init[1375]: |+ * . . * o.o . |"
"[ 128.200826] cloud-init[1375]: |.= o . = = = + |"
"[ 128.200849] cloud-init[1375]: |o + . S o + |"
"[ 128.200876] cloud-init[1375]: | + o = . . |"
"[ 128.200900] cloud-init[1375]: | o + . |"
"[ 128.200925] cloud-init[1375]: | o.=. E |"
"[ 128.200947] cloud-init[1375]: | .+.o+o. |"
"[ 128.200977] cloud-init[1375]: +----[SHA256]-----+"
"[ 138.898906] cloud-init[2217]: Cloud-init v. 20.2-45-g5f7825e2-0ubuntu1~20.04."
"1 running 'modules:config' at Wed, 22 Jul 2020 11:27:39 +0000. Up 138.77 seconds"
.
"[ 138.898966] cloud-init[2217]: Set the following 'random' passwords"
"[ 138.899001] cloud-init[2217]: installer:aecmewaoicnai"
or another example (z/VM):
...
¬ 93.463680| cloud-init¬1282|: +--¬ED25519 256|--+
¬ 93.463713| cloud-init¬1282|: !Eo=o .... !
¬ 93.463749| cloud-init¬1282|: !.Bo.o ... o !
¬ 93.463782| cloud-init¬1282|: !**.*... o = !
¬ 93.463818| cloud-init¬1282|: !*=O o. o . . !
¬ 93.463849| cloud-init¬1282|: !**++ S !
¬ 93.463886| cloud-init¬1282|: !§o+.. !
¬ 93.463918| cloud-init¬1282|: !+*o. !
¬ 93.463954| cloud-init¬1282|: !.o. !
¬ 93.463988| cloud-init¬1282|: !. !
¬ 93.464028| cloud-init¬1282|: +----¬SHA256|-----+
¬ 104.841438| cloud-init¬2004|: Cloud-init v. 20.2-45-g5f7825e2-0ubuntu1ß20.04.
1 running 'modules:config' at Mon, 20 Jul 2020 10:46:38 +0000. Up 104.63 seconds
.
¬ 104.841490| cloud-init¬2004|: Set the following 'random' passwords
¬ 104.841516| cloud-init¬2004|: installer:U9NJDuvXFw6X2fxG7pP8
But it is not complete at this point.
A completed boot of the installer system ends like this:
"It is possible to connect to the installer over the network, which"
"might allow the use of a more capable terminal and can offer more languages"
"than can be rendered in the Linux console."
"To connect, SSH to installer@<IP address>."
"The password you should use is "ydnjdnciu"kZ4tR4vRvPxHPerCNU8g""."
"The host key fingerprints are:"
"RSA SHA256:n+6TJsfdCBII2PO89GMU10mG1oFvEIFBT2v6uPN0Jz0"
"ECDSA SHA256:VcDS5ac8xswXxFEghjo1ZIcue38AM6HJg0poIxdeeec"
"ED25519 SHA256:lb/DVVhj1obDPhfo3M8oPqeAyduvlLcPFJCC8ZaiCJY"
"Ubuntu 20.04 LTS ubuntu-server sclp_line0"
In such an above situation I also can't reach the subiquity UI:
fheimes@T570:~$ ssh-keygen -f "/home/fheimes/.ssh/known_hosts" -R s1lp14
# Host s1lp14 found: line 165
/home/fheimes/.ssh/known_hosts updated.
Original contents retained as /home/fheimes/.ssh/known_hosts.old
fheimes@T570:~$ ssh installer@s1lp14
The authenticity of host 's1lp14 (10.245.236.14)' can't be established.
ECDSA key fingerprint is SHA256:AoYrJhWOFaaEtUun/aZAT8/VgNP+yq//Jhg7vzUwvxE.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 's1lp14,10.245.236.14' (ECDSA) to the list of known hosts.
installer@s1lp14's password:
Welcome to Ubuntu 20.04 LTS (GNU/Linux 5.4.0-40-generic s390x)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
System information as of Wed Jul 22 11:28:32 UTC 2020
System load: 0.44 Memory usage: 4% Processes: 180
Usage of /home: unknown Swap usage: 0% Users logged in: 0
0 updates can be installed immediately.
0 of these updates are security updates.
The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.
Hence even gathering the logs is unfortunately not easily possible. |
|
| 2021-07-23 00:30:49 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~mwhudson/livecd-rootfs/+git/livecd-rootfs/+merge/406127 |
|
| 2021-07-30 03:34:13 |
Michael Hudson-Doyle |
livecd-rootfs (Ubuntu Focal): status |
New |
In Progress |
|
| 2021-08-03 15:37:18 |
Brian Murray |
livecd-rootfs (Ubuntu Focal): status |
In Progress |
Fix Committed |
|
| 2021-08-03 15:37:20 |
Brian Murray |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2021-08-03 15:37:22 |
Brian Murray |
bug |
|
|
added subscriber SRU Verification |
| 2021-08-03 15:37:26 |
Brian Murray |
tags |
installer iso-testing s390x |
installer iso-testing s390x verification-needed verification-needed-focal |
|
| 2021-08-05 16:49:59 |
Frank Heimes |
ubuntu-z-systems: status |
In Progress |
Fix Committed |
|
| 2021-08-05 22:32:02 |
Michael Hudson-Doyle |
tags |
installer iso-testing s390x verification-needed verification-needed-focal |
installer iso-testing s390x verification-done-focal |
|
| 2021-08-16 14:25:23 |
Łukasz Zemczak |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2021-08-16 14:25:21 |
Launchpad Janitor |
livecd-rootfs (Ubuntu Focal): status |
Fix Committed |
Fix Released |
|
| 2021-08-16 15:31:06 |
Frank Heimes |
ubuntu-z-systems: status |
Fix Committed |
Fix Released |
|
