Comment 37 for bug 1986781

Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :

Reproducer/Verification for SRU to Jammy,
based on strace delay injection on read().

Uploading to jammy.
Attaching debdiff for reference.

...

Launch a VM with the Jammy daily live server:

 $ wget https://cdimage.ubuntu.com/ubuntu-server/jammy/daily-live/current/jammy-live-server-amd64.iso

 $ ISO=jammy-live-server-amd64.iso
 $ VM=casper-jammy
 $ virt-install --name $VM --cdrom $ISO --vcpus 2 --memory 2048 --disk none --osinfo ubuntu-stable-latest

Press e to edit, append "break=init console=ttyS0", press ctrl-x, close window.

Open the serial console and chroot:

 $ virsh console $VM
 ...
 (initramfs) chroot /root /bin/bash

Test strace delay injection:

 # time strace --trace read cat /dev/null
 read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\3206\2\0\0\0\0\0"..., 832) = 832
 read(3, "", 131072) = 0
 +++ exited with 0 +++

 real 0m0.041s
 user 0m0.009s
 sys 0m0.030s

 # time strace --trace read --inject read:delay_enter=5s cat /dev/null
 read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\3206\2\0\0\0\0\0"..., 832) = 832 (DELAYED)
 read(3, "", 131072) = 0 (DELAYED)
 +++ exited with 0 +++

 real 0m10.041s
 user 0m0.010s
 sys 0m0.033s

Modify the casper-md5check service:

 # sed -i '/^ExecStart=/ s,=,=/usr/bin/strace --inject read:delay_enter=60s ,' /usr/lib/systemd/system/casper-md5check.service

 # cat /usr/lib/systemd/system/casper-md5check.service
 [Unit]
 Description=casper-md5check Verify Live ISO checksums

 [Service]
 Type=oneshot
 ExecStart=/usr/bin/strace --inject read:delay_enter=60s /usr/lib/casper/casper-md5check /cdrom /cdrom/md5sum.txt
 RemainAfterExit=yes

 [Install]
 WantedBy=multi-user.target

Exit and wait; the issue happens:

 # exit
 (initramfs) exit

 ...

 Ubuntu 22.04.1 ubuntu-server ttyS0

 connecting...
 waiting for cloud-init... /

 <10 minutes later>

 ================================================================================
   Serial [ Help ]
 ================================================================================

   As the installer is running on a serial console, it has started in basic
   mode, using only the ASCII character set and black and white colours.

    ┌────────────────────────────────────────────────────────────────────────
    │ │
    │ cloud-init failed to complete after 10 minutes of waiting. This │
    │ suggests a bug, which we would appreciate help understanding. If │
    │ you could file a bug at │
    │ https://bugs.launchpad.net/subiquity/+filebug and attach the │
    │ contents of /var/log, it would be most appreciated. │
    │ │
    │ [ Switch to a shell ] │
    │ [ Close ] │
    │ │
    └────────────────────────────────────────────────────────────────────────

                    [ Continue in rich mode > ]
                    [ Continue in basic mode > ]

...

Similarly, repeat, this time with test packages from ppa:mfo/lp1986781:

...

Open the serial console and chroot:

 $ virsh console $VM
 ...
 (initramfs) chroot /root /bin/bash

Install the test package:

 # dhclient enp1s0
 # wget https://launchpad.net/~mfo/+archive/ubuntu/lp1986781/+build/25512466/+files/casper_1.470.2_amd64.deb
 # dpkg -i casper_*.deb

Modify the casper-md5check service:

 # sed -i '/^ExecStart=/ s,=,=/usr/bin/strace --inject read:delay_enter=60s ,' /usr/lib/systemd/system/casper-md5check.service

 # cat /usr/lib/systemd/system/casper-md5check.service
 Description=casper-md5check Verify Live ISO checksums
 After=multi-user.target

 [Service]
 Type=oneshot
 ExecStart=/usr/bin/strace --inject read:delay_enter=60s /usr/lib/casper/casper-md5check /cdrom /cdrom/md5sum.txt
 RemainAfterExit=yes

 [Install]
 WantedBy=multi-user.target

Exit and wait; the issue does _not_ happen anymore:

 # exit
 (initramfs) exit

...

 Ubuntu 22.04.1 ubuntu-server ttyS0

 connecting...
 waiting for cloud-init... /

 <some seconds later>

 ================================================================================
   Serial [ Help ]
 ================================================================================

   As the installer is running on a serial console, it has started in basic
   mode, using only the ASCII character set and black and white colours.

   If you are connecting from a terminal emulator such as gnome-terminal that
   supports unicode and rich colours you can switch to "rich mode" which uses
   unicode, colours and supports many languages.

   You can also connect to the installer over the network via SSH, which will
   allow use of rich mode.

                    [ Continue in rich mode > ]
                    [ Continue in basic mode > ]
                    [ View SSH instructions ]

Checking the casper-md5sum service is still running:

 Help > Enter shell.

 # systemctl status casper-md5check.service --no-pager | grep 'Active:'
      Active: activating (start) since Mon 2023-01-23 18:56:50 UTC; 3min 26s ago

And it should not be a problem, as its start timeout is not limited.

 # systemctl show casper-md5check.service | grep -i timeout
 TimeoutStartUSec=infinity
 TimeoutStopUSec=1min 30s
 TimeoutAbortUSec=1min 30s
 TimeoutStartFailureMode=terminate
 TimeoutStopFailureMode=terminate
 TimeoutCleanUSec=infinity
 JobTimeoutUSec=infinity
 JobRunningTimeoutUSec=infinity
 JobTimeoutAction=none

Quit, cleanup the VM.

 Press ctrl-]

 $ virsh destroy $VM
 $ virsh undefine $VM