Support serial-port based install

[Mark Shuttleworth]

Since we have all the benefits of a full live environment, it would be good to support a serial port based install experience. This bug serves as a placeholder for that work.

I think the key pieces would be:

 * offer boot loader menu on serial as well as console
 * perhaps have some mechanism to choose ttyS0 / ttyS1 ?
 * have a boot loader menu entry which puts console=ttySx in the kernel parameters?
 * make sure that subiquity works in VT100?

[Michael Hudson-Doyle (mwhudson)]

This mostly works already (or did, I haven't tested it in a while), you "just" need to make sure the right console=tty$whatever is on the kernel command line. Unfortunately when I talked to people about this, the consensus seemed to be that there is no way to know in general where the serial console is. If you can get to isolinux or grub you can add it yourself but if you can get to those you can probably install on tty1 anyway.

[Mark Shuttleworth (sabdfl)]

Well, yes. I managed to get stuff to work after a few hours of trawling
through the pre-history of computing.

Shouldn't we aim a little higher for human beings?

;)

[Adam Conrad (adconrad)]

The problem on a generic modular platform like x86 is we have no safe way to determine what is wired to ttyS0 and just starting a getty there can do wonderfully weird things to other serial devices that might be hanging off it.

Maybe for a server-only installer in the modern world where modems are rare and who has a fingerprint scanner on a server anyway, the risk is a sensible trade-off to get it right "most of the time", but I dunno. Other installer codebases (like d-i) that chose not to automatically do this on x86 certainly hail from a time when lots of random serial devices were a lot more common.

Now, above discussion aside, there is a devicetree node used for this in OpenFirmware (which is always correct), and there is meant to be a way in ACPI to tag an actual serial console chosen from EFI/BIOS. As far as I know, the ACPI one is much less reliably used than the OF one, but maybe we can still figure out how to poke it and use it when it is.

[Adam Conrad (adconrad)]

And ignoring all the above bits about sketchy auto-detection, we could just add a grub menu entry for "use serial" that enables consoles on ttyS0 and S1 and random devices be damned, cause the user pressed that button.

[Mark Shuttleworth (sabdfl)]

Yes, in grub I think it should be an option which the user selects.

The question is more how we get that option displayed in a serial-only
world.

As I understand it, grub can display its menu on the serial, but we
don't do that by default on the install for the reasons Adam described.

I think we could either:

 * take the view that the *install* disk is something people use
explicitly and we can JFDI there
 * we could add a second boot target for the BIOS which has something
explicit in the name

I don't know if the latter is feasible. Can you effectively have a USB
disk with TWO bootable partitions, one called "Install Ubuntu" and
another called "Install Ubuntu over ttyS0"? And for good measure, a
third called "Install Ubuntu over ttyS1"?

If this is possible, then we would have it all nicely mapped out:

 * the BIOS is told to redirect to serial
 * the disk partitions with console / COM0 / COM1 are displayed as BIOS
boot options
 * if the user chooses the correct one they will then see the grub menu
 * the user again chooses the correct one (or maybe the menu items for
that boot partition automatically are correct)
 * and off you go

Mark

[Steve Langasek (vorlon)]

At the grub level, this appears to be sufficient to let grub auto-configure its display on both graphical and serial console:

  serial
  terminal_input serial console
  terminal_output serial console

("terminal_{in,out}put" take a list as arguments)

Works in testing for me in a VM with either UEFI (ovmf) or BIOS (seabios).

Is there any reason for us not to configure this by default on x86 in GRUB? (Not just for the installer image specifically; the serial console autodetection would appear to be based on the firmware interfaces Adam mentions, so should be equally safe generally) Maybe we should turn it on for the subiquity image first, then gather feedback? OTOH what's the worst that happens if we try to use a wrong serial device as a console from within GRUB?

[Mathieu Trudel-Lapierre (cyphermox)]

You could arguably say that serial grub is a good idea everywhere. For one thing, there's some chance of letting people using braille displays drive the boot menu. Modems probably won't care about it too much, since the output would look nothing like AT commands (and the set of modem users is pretty limited).

Also, given the decisions made at install-time, we can pick whether or not we should add "console=ttS0" on the installed system's commandline.

[Adam Conrad (adconrad)]

In the BIOS world, 99% of serial consoles were a hackish thing that mirrored the VGA text output to the serial console, hence lilo/grub "just worked", and then the kernel could DTRT with a console= given to it. I'm guessing EFI is doing things more correctly here, which leads to grub actually needing serial drivers all of a sudden.

If grub's detecting serial consoles based on the ACPI console node's setting, I see exactly zero danger in enabling this globally. In fact, if it's doing so, I suspect we can also go one better and propagate that detection down via a conditional in grub.cfg, if we really want to be clever (may require patches to upstream, may not be as simple as it sounds, YMMV, no promises, warranty void upon reading this comment).

[Steve Langasek (vorlon)]

If I delete the serial device from my VM (with either ovmf or seabios), the 'serial' command returns an error. So that also seems useful. OTOH, the error message refers to com0, so maybe it's not all /that/ smart about what device it's talking to.

If I configure *two* serial ports in a VM, grub output appears only on the first.

Regardless, I think it's reasonable for the installer image in particular to output to both serial and graphical out of the box, even if it's doing so blindly.

[Mark Shuttleworth (sabdfl)]

Let's start this on the subiquity image and see how it goes. Thanks all
for weighing in.

[Dimitri John Ledkov (xnox)]

I have a user on IRC that tried to install onto https://www.pcengines.ch/apu3a2.htm using https://www.pcengines.ch/usbcom1a.htm as the console, but failed to get to the installer, but did see the boot-loader which guided to use broken console kernel cmdline which fails to boot to subiquity on the serial console.

Separately, the isolinux bootsplash screens there show options which are not valid for subiquity.

I think we should be integrating with the systemd-serial-tty getty generator, to start as many subiquities as needed in all the places.

<giaco> I confirm. I was able to see the bootloader menu, preff F-keys, type at boot: prompt, start memtest. But install, expert, cli and other were not accepting console options. Without them I was that it was loading something but hey no feedback no party.

[Steve Langasek (vorlon)]

Dimitri, isn't this what ./live-build/ubuntu-server/includes.binary/overlay/lib/systemd/system/serial-subiquity@.service + ./live-build/ubuntu-server/includes.binary/overlay/lib/systemd/system/serial-getty@.service.d/subiquity-serial.conf is already supposed to do? Michael reported that this was previously working. What "broken" kernel commandline, specifically, was being used; and what is a correct kernel commandline that is not working?

If there has been a regression in subiquity wrt serial console support, please file a separate bug report for that so that we can keep this bug focused on the question of serial console integration from bootloader on.

[Mark Shuttleworth (sabdfl)]

No feedback, no party. That's bug comment of the week, for sure :)

[Michael Hudson-Doyle (mwhudson)]

Coming back to this after far too long, my reading of this bug is that we should make a change to the grub config for the live server installer iso, which to the best of my knowledge is a change we would make in debian-cd. The change, if it detects a serial line, should make should cause grub to display both on serial and console and append console=whatever to the kernel command line.

What about BIOS boot? Those systems run isolinux currently. Can that display on serial too, or should we switch to grub for BIOS boot too?

[Dimitri John Ledkov (xnox)]

So in parallel to this bug report there have been several other things done and improvement in subiquity/ubuntu since the original bug report:

- grub was attempted to co-present it simultaniously multiplexed on all serial and graphical consoles
- that was reverted as there were console corruptions resulting from that, meaning one couldn't use grub on any of the consoles
- instead grub is running on the only console available, or the best one / same one as firmware for example. This works for majority cases, but may need tweaking on edge cases (ie. multiple serial consoles)
- since 2018 May, a lot of work has gone into the serial subiquity UI, once we are passed grub - subiquity is started on all the consoles we can detect, and one will get getty on all consoles post install too
- there might be some corner cases where interactive dmesg goes to only one out of several available serial consoles

Overall the serial console experience is really great (and with colors!) on modern serial-only servers like BMC enabled ARM servers.

Some improvements we can do in subiquity:
- remember which console is used to complete the install, which persists to the installed system for interactive dmesg
- for autoinstall, we do need a way to specify which console should be used for the "autoinstall" progress/crashes

[Michael Hudson-Doyle (mwhudson)]

Can we still get grub to add console=whatever to the kernel command line when it thinks there is a serial console?

> - remember which console is used to complete the install, which persists to the installed system for interactive dmesg

Does this essentially mean propagating console=whatever into the grub config for the installed system? That sounds like it would need a curtin change.

> - for autoinstall, we do need a way to specify which console should be used for the "autoinstall" progress/crashes

As I mentioned on IRC, my plan here is to use _all_ consoles for non-interactive messages.

[Ryan Harper (raharper)]

On Thu, Nov 14, 2019 at 12:56 AM Michael Hudson-Doyle <
<email address hidden>> wrote:

> Can we still get grub to add console=whatever to the kernel command line
> when it thinks there is a serial console?
>
> > - remember which console is used to complete the install, which
> persists to the installed system for interactive dmesg
>
> Does this essentially mean propagating console=whatever into the grub
> config for the installed system? That sounds like it would need a curtin
> change.
>

Curtin already copies any parameters used to boot the system after '---'
That doesn't help if you want additional parameters that aren't already on
the command line.

[xinliang (xin3liang)]

Does this bug means the new installer doesn't support serial console install? We don't any graphic console in our server.
I try ubuntu 20.04.2 live iso install in serial console and can't see the subiquity installer welcome page.

Here is the last log:
[ 112.173157] cloud-init[2414]: Cloud-init v. 20.4.1-0ubuntu1~20.04.1 running 'modules:final' at Tue, 30 Mar 2021 09:14:24 +0000. Up 111.59 seconds.
[ 112.196289] cloud-init[2414]: ci-info: no authorized SSH keys fingerprints found for user installer.
[ 112.220316] cloud-init[2414]: Cloud-init v. 20.4.1-0ubuntu1~20.04.1 finished at Tue, 30 Mar 2021 09:14:24 +0000. Datasource DataSourceNoCloud [seed=/var/lib/cloud/seed/nocloud][dsmode=net]. Up 112.15 seconds
[ 112.252300] cloud-init[2414]: Welcome to Ubuntu Server Installer!
[ 112.268275] cloud-init[2414]: Above you will find SSH host keys and a random password set for the `installer` user. You can use these credentials to ssh-in and complete the installation. If you provided SSH keys in the cloud-init datasource, they were also provisioned to the installer user.
[ 112.304281] cloud-init[2414]: If you have access to the graphical console, like TTY1 or HMC ASCII terminal you can complete the installation there too.
[ OK ] Finished Execute cloud user/final scripts.
[FAILED] Failed to start Service fo…snap application lxd.activate.
See 'systemctl status snap.lxd.activate.service' for details.
[ OK ] Reached target Cloud-init target.
[ OK ] Started Subiquity, the ins…ler for Ubuntu Server ttyAMA0.

Ubuntu 20.04.2 LTS ubuntu-server ttyAMA0

Traceback (most recent call last):
  File "/snap/subiquity/2281/usr/bin/subiquity", line 33, in <module>
  File "/snap/subiquity/2281/usr/bin/subiquity", line 25, in importlib_load_entry_point
  File "/snap/subiquity/2281/lib/python3.6/site-packages/importlib_metadata/__init__.py", line 96, in load
  File "/snap/subiquity/2281/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._boo
Ubuntu 20.04.2 LTS ubuntu-server ttyAMA0