Temporary installer password and key fingerprints are not printed with latest impish daily ISOs

Bug #1933523 reported by Frank Heimes
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Ubuntu on IBM z Systems
High
Skipper Bug Screeners
subiquity
Undecided
Dan Bungert
livecd-rootfs (Ubuntu)
Undecided
Canonical Foundations Team
Focal
Undecided
Unassigned

Bug Description

[impact]
cloud-init no longer puts the password of a user with a RANDOM password anywhere subiquity can find it, so while the password chosen ends up printed to the console, subiquity cannot display it from within the UI as it used to.

[regression potential]
If something goes wrong here, it's possible that subiquity will no longer be able to display the password for the installer user at all, which will make continuing the install via SSH impossible (this is very important for installing on z/vm)

[test case]
Make an ISO with this change and boot it and check that the installer password is available in the UI. Make a s390x ISO, boot it in z/vm and check that the installer password is displayed on the console.

[original description]
Using the latest Impish daily ISO (timestamp) 2021-06-24 I faced the situation that I didn't got a temporary installation password not the key fingerprints at the end of the installer boot-up process.

Hence I am not able to perform an installation from remote via ssh.
However, subiqity is started at the console (e.g. Integrated ASCII console) and an installation can be successfully completed there, but a console installation is not in all cases possible, for example with z/VM guests.

As one can read between the lines I tried this on s390x (Integrated ASCII console and z/VM), but 'paride' tried the same on amd64 and ran into the same symptoms, so it does not seem to be architecture specific.

This must have happened recently, since I already was able to do ssh-based Impish installations on s390x in the past (for example with the daily ISO from 06-14-2021).
___

This is not the case for the 20.04 dailies - I can do remote ssh installation with the 20.04 live daily ISOs.
___

With Impish on s390x the kernel parameter 'quiet' is still set by default.

Related branches

Revision history for this message
Frank Heimes (fheimes) wrote :
Revision history for this message
Frank Heimes (fheimes) wrote :
description: updated
Changed in livecd-rootfs (Ubuntu):
assignee: nobody → Canonical Foundations Team (canonical-foundations)
Changed in ubuntu-z-systems:
assignee: nobody → Skipper Bug Screeners (skipper-screen-team)
summary: - Temp installer password and key fingerprints are not printed with latest
- impish daily ISO
+ Temporäry installer password and key fingerprints are not printed with
+ latest impish daily ISOs
summary: - Temporäry installer password and key fingerprints are not printed with
+ Temporary installer password and key fingerprints are not printed with
latest impish daily ISOs
Frank Heimes (fheimes)
Changed in ubuntu-z-systems:
importance: Undecided → High
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in livecd-rootfs (Ubuntu):
status: New → Confirmed
Frank Heimes (fheimes)
Changed in ubuntu-z-systems:
status: New → Confirmed
Dan Bungert (dbungert)
tags: added: fr-1477
Dan Bungert (dbungert)
Changed in subiquity:
assignee: nobody → Dan Bungert (dbungert)
status: New → In Progress
Changed in livecd-rootfs (Ubuntu):
status: Confirmed → In Progress
Revision history for this message
Dan Bungert (dbungert) wrote :

Hi @Frank,

So I looked into this a bit, and with some feedback help from mwhudson, I see that we attempt to print the ssh info. Presumably this isn't going in the place you're looking, however.

Something that changed relatively recently was that we no longer have cloud-init handle setting the temp password of the installer user.

Is that how you get the ssh key / installer user password historically? From the cloud-init output? Some message like
'Above you will find SSH host keys and a random password set for the `installer` user'

Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote :

Hmm looking into our livecd-rootfs stuff a bit, I'm a bit confused about tty names. We have special handling for sclp_line0 and (on s390x only) ttyS0 but from your logs it seems the tty in z/vm is called ttysclp0, is that correct? Which terminal are you looking at and which can run the subiquity tui?

Revision history for this message
Frank Heimes (fheimes) wrote :

Regarding the (main) s390x ttys:
SCLP line-mode terminal device driver /dev/sclp_line0 ('Operating Systems Messages')
SCLP full-screen mode VT220 terminal device driver /dev/ttysclp0 ('Integrated ASCII console', or z/VM)

Subiquity tui can only run in full-screen, not in line-mode.

There are two "consoles" in case of an LPAR - the "Operating Systems Messages" (line-mode) where only low-level console messages are displayed, including the installer boot messages that contain the temporary installer password,
and the "Integrated ASCII Console" (the emulated VT220 full-screen-mode) where the subiquity tui can run - and is in fact started there automatically (this still works and allows to complete an installation).

The z/VM console (by default) uses the 3215 line-mode, for the console messages that include temporary installer password, but the Subiquity tui can not run there (since it's line-mode), hence a remote ssh installation is needed here.

Just notice that all this worked fine in the past - and even with the early Impish images (about 06-14-2021 and before) and I do not think that things changed that much.

And since paride tried a current impish amd64 ISO image, where the temp installer password is also missing,things seems to be platform agnostic and should not be cause by any of the special s390x consoles - I think.

What's missing is an output like this (see attached example.txt taken from a 20.04 installation), that provides lines like:
"
"Set the following 'random' passwords"
installer:x3Z9hgeXCPNDQC8VFeqK
"
and
"
It is possible to connect to the installer over the network, which
might allow the use of a more capable terminal and can offer more languages
than can be rendered in the Linux console.

To connect, SSH to installer@10.245.236.14.

You should use the preconfigured password passed to cloud-init.

The host key fingerprints are:

RSA SHA256:YTmNejNSf1OlzQbOcNzkcYrDzqfy9aoT1laAs+oca5k
ECDSA SHA256:MKGkFuNzdQ3aNBL33bhqapn8clpDD9ZIL2SLr24IDAI
ED25519 SHA256:NitkBIaBVl97j1E3AXOrQRNrfR89UChEFxbJWnYhtVE
"
and yes, I believe the output came from cloud-init.

And even earlier the random / temp. password was not only given "somewhere" in the boot log, but also at the end (close to "To connect, SSH to installer@10.245.236.14."), so that one didn't needed to scroll through the messages to find it.

Revision history for this message
Frank Heimes (fheimes) wrote :
Changed in ubuntu-z-systems:
status: Confirmed → In Progress
Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote : Re: [Bug 1933523] Re: Temporary installer password and key fingerprints are not printed with latest impish daily ISOs

On Tue, 29 Jun 2021 at 19:50, Frank Heimes <email address hidden>
wrote:

> Regarding the (main) s390x ttys:
> SCLP line-mode terminal device driver /dev/sclp_line0
> ('Operating Systems Messages')
> SCLP full-screen mode VT220 terminal device driver /dev/ttysclp0
> ('Integrated ASCII console', or z/VM)
>
> Subiquity tui can only run in full-screen, not in line-mode.
>
> There are two "consoles" in case of an LPAR - the "Operating Systems
> Messages" (line-mode) where only low-level console messages are displayed,
> including the installer boot messages that contain the temporary installer
> password,
> and the "Integrated ASCII Console" (the emulated VT220 full-screen-mode)
> where the subiquity tui can run - and is in fact started there
> automatically (this still works and allows to complete an installation).
>
> The z/VM console (by default) uses the 3215 line-mode, for the console
> messages that include temporary installer password, but the Subiquity
> tui can not run there (since it's line-mode), hence a remote ssh
> installation is needed here.
>
> Just notice that all this worked fine in the past - and even with the
> early Impish images (about 06-14-2021 and before) and I do not think
> that things changed that much.
>
> And since paride tried a current impish amd64 ISO image, where the temp
> installer password is also missing,things seems to be platform agnostic
> and should not be cause by any of the special s390x consoles - I think.
>
>
> What's missing is an output like this (see attached example.txt taken from
> a 20.04 installation), that provides lines like:
> "
> "Set the following 'random' passwords"
> installer:x3Z9hgeXCPNDQC8VFeqK
> "
>

So I expect this to no longer be present; basically cloud-init has changed
to make it much harder / impossible to find this password later.

> and
> "
> It is possible to connect to the installer over the network, which
> might allow the use of a more capable terminal and can offer more languages
> than can be rendered in the Linux console.
>
>
> To connect, SSH to installer@10.245.236.14.
>
> You should use the preconfigured password passed to cloud-init.
>
> The host key fingerprints are:
>
> RSA SHA256:YTmNejNSf1OlzQbOcNzkcYrDzqfy9aoT1laAs+oca5k
> ECDSA SHA256:MKGkFuNzdQ3aNBL33bhqapn8clpDD9ZIL2SLr24IDAI
> ED25519 SHA256:NitkBIaBVl97j1E3AXOrQRNrfR89UChEFxbJWnYhtVE
> "
>

This output should still be there though, and that's what we should focus
on figuring out. The serial-getty service running on sclp_line0 is what is
_supposed_ to be producing this output.

I guess you should try this on an LPAR install where you can actually get
into the system: can you post the output of systemctl
status serial-getty@sclp_line0.service? And maybe systemctl
show serial-getty@sclp_line0.service? TIA

Revision history for this message
Frank Heimes (fheimes) wrote :

To be able to better compare between focal-daily and impish I now started LPAR installations:
1) on focal daily (from pending) - where things work, means where I get the temp/randon pw
2) and on impish (daily) (again from pending) - where I do not get a temp/randon pw

I used LPAR installs to be able to get into the system via the Integrated ASCII console.

The attached file "lpar_OS_messages.zip" contains the following corresponding two files:
lpar_OS_messages_console_focal_daily.txt
lpar_OS_messages_console_impish.txt

At the beginning of each file one can find the console installer boot-up messages
and after a separator "__________" as desired the output of:
"systemctl status serial-getty@sclp_line0.service"
and
"systemctl show serial-getty@sclp_line0.service"

Revision history for this message
Paride Legovini (paride) wrote :

@Michael: the installer:<password> output should still be present, but only as a printout in the console output (/dev/console), cloud-init doesn't log it anymore in cloud-init-output.log. This is the relevant change:

https://github.com/canonical/cloud-init/commit/b794d426b9ab43ea9d6371477466070d86e10668

Revision history for this message
Frank Heimes (fheimes) wrote :

just did the following quick test:
from the "Integrated ASCII console", where impish/subiquity ran I opened the live shell and did:
root@ubuntu-server:/# echo "x" > /dev/console
and
root@ubuntu-server:/# echo "x" > /dev/sclp_line0
and I can see two "x" coming through to the "Operating Systems messages" which is sclp_line0
that lets me think that /dev/console is properly 'mapped' to /dev/sclp_line0

and fwiw:
- Operating System Messages:
   $ tty
   /dev/sclp_line0
- Integrated ASCII Console:
   $ tty
   /dev/ttysclp0

Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote :

Oh, I see the problem: the process that prints the network connection information is running, but does not have its stdout connected to the right place so the output is going nowhere. Easy fix fortunately (easier than figuring out what is going on!)

Revision history for this message
Frank Heimes (fheimes) wrote :

Ok, sounds promising.
Please give me a heads-up when it lands in Impish daily, so that I can update to it and retry.

Revision history for this message
Dan Bungert (dbungert) wrote :

Marking invalid on Subiquity as I believe that no change is required there. We can of course reverse that position if we find that Subiquity changes are indeed desired.

Changed in subiquity:
status: In Progress → Invalid
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package livecd-rootfs - 2.728

---------------
livecd-rootfs (2.728) impish; urgency=medium

  [ Dan Bungert ]
  * Remove the customized final message from the live server cloud-init config
    as it is no longer displayed.

  [ Michael Hudson-Doyle ]
  * Make sure the process that prints SSH connection info on s390x has
    its output connected to the console. (LP: #1933523)

  [ Brian Murray ]
  * Add a subarch of intel-iotg which is used for creating images with a
    kernel optimized for Intel IOT devices.

 -- Michael Hudson-Doyle <email address hidden> Thu, 01 Jul 2021 11:00:08 +1200

Changed in livecd-rootfs (Ubuntu):
status: In Progress → Fix Released
Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote :

The fix should be in the next images that get built.

Frank Heimes (fheimes)
Changed in ubuntu-z-systems:
status: In Progress → Fix Committed
Revision history for this message
Frank Heimes (fheimes) wrote :

@mwhudson @dbungert
Looks like the line in subiquity-serial.conf did the trick!
With this in place and as part of the Impish daily ISO (I used the image from today, July 2nd) I can now see the temp./random password again - and even more nicely at the end of the installer boot-up process - great !

I've attached the last lines from an LPAR as well as from a z/VM installation console log.

I'm now happily closing this ticket as Fix Released - thx!

Changed in ubuntu-z-systems:
status: Fix Committed → Fix Released
Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote :

Yay thanks. Now to get back to backporting this whole mess to focal.

Revision history for this message
Frank Heimes (fheimes) wrote :

That sounds like 'fun'...
If there is once a candidate available that can be used by focal (or an appropriate focal daily) - I'm happy to do some more testing on focal, too.

description: updated
Changed in livecd-rootfs (Ubuntu Focal):
status: New → In Progress
Revision history for this message
Paride Legovini (paride) wrote :

Hi, now as [1] landed if I don't go wrong we'll have to:

- Wait for livecd-rootfs 2.664.25 to be accepted and installed
  in the Focal archive.
- Wait for new ISOs to be built and check from the cd-build-log [2]
  which livecd build was used.
- Check that livecd build log at [3] actually uses livecd-rootfs 2.664.25.
  (The livecd-rootfs version is not included in the cd-build-log [2]).
- If everything is as expected => Test the ISO.

[1] https://code.launchpad.net/~mwhudson/livecd-rootfs/+git/livecd-rootfs/+merge/406127
[2] https://people.canonical.com/~ubuntu-archive/cd-build-logs/ubuntu-server/focal/
[3] https://launchpad.net/~ubuntu-cdimage/+livefs/ubuntu/focal/ubuntu-server-live/

Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote :

On Fri, 30 Jul 2021 at 21:11, Paride Legovini <email address hidden>
wrote:

> Hi, now as [1] landed if I don't go wrong we'll have to:
>
> - Wait for livecd-rootfs 2.664.25 to be accepted and installed
> in the Focal archive.
> - Wait for new ISOs to be built and check from the cd-build-log [2]
> which livecd build was used.
> - Check that livecd build log at [3] actually uses livecd-rootfs 2.664.25.
> (The livecd-rootfs version is not included in the cd-build-log [2]).
> - If everything is as expected => Test the ISO.
>

More or less yes. But no ISO will be built using the livecd-rootfs from
proposed automatically, we'll have to ask someone to do that.

Cheers,
mwh

Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Frank, or anyone else affected,

Accepted livecd-rootfs into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.664.26 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in livecd-rootfs (Ubuntu Focal):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-focal
Revision history for this message
Brian Murray (brian-murray) wrote :

Hello Frank, or anyone else affected,

Accepted livecd-rootfs into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.664.27 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Revision history for this message
Łukasz Zemczak (sil2100) wrote :

As requested, I ran a server-live build with proposed enabled:

http://cdimage.ubuntu.com/ubuntu-server/focal/daily-live/20210805.1/

Revision history for this message
Frank Heimes (fheimes) wrote :

I tested the updated livecd-rootfs that is part of the new live-server image that Łukasz created:
http://cdimage.ubuntu.com/ubuntu-server/focal/daily-live/20210805.1/
and did a couple of installations (mainly LPAR, but DASD and FCP) and so far the boot-up of the installer always succeeded, and the temporary installer password and key fingerprints were always present and I was (so far) always able to login to the installer via ssh.
So I consider this now as successfully verified and think that we can close this ticket.

tags: added: verification-done-focal
removed: verification-needed verification-needed-focal
Revision history for this message
Paride Legovini (paride) wrote :

+1 on the verification done by Frank: I did a couple of non-s390x installs using the -proposed ISO and everything looks good.

Revision history for this message
Łukasz Zemczak (sil2100) wrote :

Thanks! We will be probably re-spinning livecd-rootfs with some additional fixes on top, but this verification still stands.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package livecd-rootfs - 2.664.27

---------------
livecd-rootfs (2.664.27) focal; urgency=medium

  * And whoops, we missed adding ARCH in the SUBARCH ubuntu-image handling for
    intel-iot.

livecd-rootfs (2.664.26) focal; urgency=medium

  * Revert previous change of fixing /dev sharing - this causes weird
    autopkgtest issues.

livecd-rootfs (2.664.25) focal; urgency=medium

  [ Brian Murray ]
  * Add support for creating images (ubuntu-core and classic) with a kernel
    optimized for Intel IoT devices. (LP: #1938338)

  [ Michael Hudson-Doyle ]
  * Simplify how the subiquity client is run on the serial console in the live
    server environment, breaking a unit cycle that sometimes prevents
    subiquity from starting up at all. (LP: #1888497)
  * Do not set the password for the installer user via cloud-init as subiquity
    can now do this itself. (LP: #1933523)

  [ Łukasz 'sil2100' Zemczak ]
  * Fix sharing of the /dev tree to make sure we can safely umount the chroot
    when needed. This fixes local non-livefs-builder image builds.
    (LP: #1938414)

 -- Łukasz 'sil2100' Zemczak <email address hidden> Wed, 04 Aug 2021 17:32:37 +0200

Changed in livecd-rootfs (Ubuntu Focal):
status: Fix Committed → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for livecd-rootfs has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers