kinetic ppc64le reporting Failed to send WATCHDOG=1 notification message: Transport endpoint is not connected

[Colin Ian King]

Upgraded from Jammy to Kinetic (14 Oct 2022) on ppc64le in QEMU, single CPU, 1GB memory, rebooted, can NO longer login. Stuck in systems boot phase.

Attached is an image of the hang

[Andrea Righi (arighi)]

It could be related to #1991691 if you're using the kernel currently in release.

[Colin Ian King (colin-king)]

Getting the same issue on the kinetic server daily image during the install phase, see attached

[Colin Ian King (colin-king)]

Using power9 arch, attached is the xml config

<domain type="qemu">
  <name>ubuntu22.10-kinetic-ppc64le</name>
  <uuid>e410f72a-6555-4b47-bb18-8bb61eb52de2</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://ubuntu.com/ubuntu/22.04"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit="KiB">8388608</memory>
  <currentMemory unit="KiB">8388608</currentMemory>
  <vcpu placement="static">1</vcpu>
  <os>
    <type arch="ppc64le" machine="pseries-jammy">hvm</type>
  </os>
  <cpu mode="custom" match="exact" check="none">
    <model fallback="forbid">POWER9</model>
  </cpu>
  <clock offset="utc"/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-ppc64le</emulator>
    <disk type="file" device="cdrom">
      <driver name="qemu" type="raw" cache="unsafe"/>
      <source file="/home/cking/virt/ubuntu-22.04-live-server-ppc64el.iso"/>
      <target dev="sdb" bus="scsi"/>
      <readonly/>
      <boot order="2"/>
      <address type="drive" controller="0" bus="0" target="0" unit="1"/>
    </disk>
    <disk type="file" device="disk">
      <driver name="qemu" type="qcow2" cache="unsafe"/>
      <source file="/home/cking/virt/ubuntu22.04-ppc64le-clone"/>
      <target dev="vda" bus="virtio"/>
      <boot order="1"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x07" function="0x0"/>
    </disk>
    <controller type="usb" index="0" model="qemu-xhci">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0"/>
    </controller>
    <controller type="pci" index="0" model="pci-root">
      <model name="spapr-pci-host-bridge"/>
      <target index="0"/>
    </controller>
    <controller type="scsi" index="0" model="ibmvscsi">
      <address type="spapr-vio" reg="0x00002000"/>
    </controller>
    <controller type="virtio-serial" index="0">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x04" function="0x0"/>
    </controller>
    <interface type="network">
      <mac address="52:54:00:27:88:a3"/>
      <source network="default"/>
      <model type="rtl8139"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x0"/>
    </interface>
    <serial type="pty">
      <target type="spapr-vio-serial" port="0">
        <model name="spapr-vty"/>
      </target>
      <address type="spapr-vio" reg="0x30000000"/>
    </serial>
    <console type="pty">
      <target type="serial" port="0"/>
      <address type="spapr-vio" reg="0x30000000"/>
    </console>
    <channel type="spicevmc">
      <target type="virtio" name="com.redhat.spice.0"/>
      <address type="virtio-serial" controller="0" bus="0" port="1"/>
    </channel>
    <input type="keyboard" bus="usb">
      <address type="usb" bus="0" port="1"/>
    </input>
    <input type="mouse" bus="usb">
      <address type="usb" bus="0" port="2"/>
    </input>
    <graphics type="spice" autoport="yes">
      <listen type="address"/>
    </graphics>
    <sound model="ich6">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0"/>
    </sound...

[Nick Rosbrook (enr0n)]

This makes me think pid1 is dead, since the notification socket is apparently down. Can you please try booting with these options appended to the kernel command line?

systemd.log_level=debug systemd.log_target=console console=ttyS0,38400 console=tty1

Hopefully that will give us more information to debug this.

[Colin Ian King (colin-king)]

I can't install Kinetic server from the LiveCD because of this issue. It's just broken.

[Colin Ian King (colin-king)]

This looks related: https://bugs.launchpad.net/ubuntu/+source/ubiquity/+bug/1754328

[Colin Ian King (colin-king)]

systemd error is being emitted from dispatch_notify_event() in src/journal/journald-server.c, which is called from server_connect_notify(). This function contains a large hunk of comment block describing a potential race condition. I wonder if this is some kind of race condition that only occurs on some systems due to boot speed, in this case, emulated powerpc in QEMU.

[Nick Rosbrook (enr0n)]

Is this issue also present in Lunar? At this point I do not think we will do another systemd SRU for Kinetic.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Colin Ian King (colin-king)]

Why won't fix? It occurs on other releases too.

[Nick Rosbrook (enr0n)]

Only marked won't fix for Kinetic. Thanks for confirming it happens on other releases.

[Brian Murray (brian-murray)]

Ubuntu 23.04 (Lunar Lobster) has reached end of life, so this bug will not be fixed for that specific release.