autopkgtest ignores --architecture flag

[Heinrich Schuchardt]

Description
-----------
The cloud images produced by livecd-rootfs do not properly configure the tty for armhf and subsequently the terminal output is not available.

Test Case
---------
1) Run `autopkgtest-buildvm-ubuntu-cloud -v --arch armhf --release mantic` substituting the release name as appropriate
2) Observe the last console message `EFI stub: Exiting boot services...`

With a cloud image build with the livecd-rootfs version in -proposed you'll see much more output from `autopkgtest-buildvm-ubuntu-cloud` including cloud-init configuring the system.

Regression Potential
--------------------
The serial console is being set to ttyS0 differently that it was for architectures other than arm64 or armhf. However, this has been in place in noble for an extended period of time and has not cause any regressions for those architectures.

Original Description
--------------------
On an arm64 as well as on amd64 I saw this behavior:

$ autopkgtest-buildvm-ubuntu-cloud -v --arch armhf --release mantic
$ autopkgtest -U --apt-pocket=proposed -a armhf -s *.dsc -- qemu autopkgtest-mantic-armhf.img
autopkgtest [13:18:32]: starting date and time: 2023-09-20 13:18:32+0200
autopkgtest [13:18:32]: version 5.28ubuntu1
autopkgtest [13:18:32]: host mcbin; command line: /usr/bin/autopkgtest -U --apt-pocket=proposed -a armhf -s forensics-all_3.47.dsc -- qemu autopkgtest-mantic-armhf.img
qemu-system-aarch64: terminating on signal 15 from pid 3520 (/usr/bin/python3)
<VirtSubproc>: failure: timed out waiting for 'login prompt on serial console'
autopkgtest [13:19:33]: ERROR: testbed failure: unexpected eof from the testbed

When trying to boot into the image on an arm64 system with

qemu-system-aarch64 \
-machine virt -accel kvm -m 4G -smp 4 -cpu host,aarch64=off -nographic \
-object rng-random,filename=/dev/urandom,id=rng0 \
-device virtio-rng-pci,rng=rng0,id=rng-device0 \
-drive if=pflash,format=raw,unit=0,read-only=on,file=/usr/share/AAVMF/AAVMF32_CODE.fd \
-device virtio-net-pci,netdev=eth0 \
-netdev user,id=eth0,hostfwd=tcp::8022-:22 \
-drive file=autopkgtest-mantic-armhf.img,if=virtio,format=qcow2

the last messages displayed are

EFI stub: Booting Linux Kernel...
EFI stub: Entering in SVC mode with MMU enabled
EFI stub: Loaded initrd from LINUX_EFI_INITRD_MEDIA_GUID device path
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services...

qemu-system-aarch64 seems to be close to idling at less than 0.5 % CPU load. So booting obviously stalled.

[Heinrich Schuchardt (xypron)]

A login into the image is possible via U-Boot:

qemu-system-arm -machine virt -cpu cortex-a15 -m 1G -smp cores=2 -bios /usr/lib/u-boot/qemu_arm/u-boot.bin -nographic -object rng-random,filename=/dev/urandom,id=rng0 -device virtio-rng-pci,rng=rng0,id=rng-device0 -device virtio-net-pci,netdev=eth0 -netdev user,id=eth0,hostfwd=tcp::8022-:22 -drive file=autopkgtest-mantic-armhf.img,if=virtio,format=qcow2

In the U-Boot console use

virtio scan
load virtio 0:1 $kernel_addr_r boot/vmlinuz
load virtio 0:1 $ramdisk_addr_r boot/initrd.img
setenv bootargs root=/dev/vda1
bootz $kernel_addr_r $ramdisk_addr_r:$filesize $fdtcontroladdr

grub.cfg has this line:

linux /boot/vmlinuz-6.5.0-5-generic root=LABEL=cloudimg-rootfs ro console=tty1 console=ttyS0

But the last command indicates that the serial console is ttyAMA0:

ubuntu@autopkgtest:~$ last
ubuntu ttyAMA0 Wed Sep 20 13:50 still logged in
reboot system boot 6.5.0-5-generic Wed Sep 20 13:43 still running

File /etc/default/grub.d/50-cloudimg-settings.cfg introduces the console=ttyS0 commandline argument.

[Heinrich Schuchardt (xypron)]

On arm64 the command 'autopkgtest -U --apt-pocket=proposed -a armhf -s *.dsc -- qemu autopkgtest-mantic-armhf.img' invokes QEMU with:

qemu-system-aarch64 -machine virt -cpu host -m 1024 -smp 1 -nographic -net nic,model=virtio -net user,hostfwd=tcp:127.0.0.1:10022-:22 -object rng-random,filename=/dev/urandom,id=rng0 -device virtio-rng-pci,rng=rng0,id=rng-device0 -monitor unix:/tmp/autopkgtest-qemu.vpg40tc_/monitor,server=on,wait=off -virtfs local,id=autopkgtest,path=/tmp/autopkgtest-qemu.vpg40tc_/shared,security_model=none,mount_tag=autopkgtest -device virtio-serial -chardev socket,path=/tmp/autopkgtest-qemu.vpg40tc_/hvc0,server=on,wait=off,id=hvc0 -device virtconsole,chardev=hvc0 -chardev socket,path=/tmp/autopkgtest-qemu.vpg40tc_/hvc1,server=on,wait=off,id=hvc1 -device virtconsole,chardev=hvc1 -serial unix:/tmp/autopkgtest-qemu.vpg40tc_/ttyS0,server=on,wait=off -drive index=0,file=/tmp/autopkgtest-qemu.vpg40tc_/overlay.img,format=qcow2,cache=unsafe,if=virtio,discard=unmap -drive if=pflash,format=raw,unit=0,read-only=on,file=/usr/share/AAVMF/AAVMF_CODE.fd -drive if=pflash,format=raw,unit=1,file=/tmp/autopkgtest-qemu.vpg40tc_/efivars.fd -enable-kvm

The CPU argument is wrong. It needs to be: -cpu host,aarch64=off.

[Heinrich Schuchardt (xypron)]

On amd64 'autopkgtest -U --apt-pocket=proposed -a armhf -s *.dsc -- qemu autopkgtest-mantic-armhf.img' invokes QEMU with:

qemu-system-x86_64 -m 1024 -smp 1 -nographic -net nic,model=virtio -net user,hostfwd=tcp:127.0.0.1:10022-:22 -object rng-random,filename=/dev/urandom,id=rng0 -device virtio-rng-pci,rng=rng0,id=rng-device0 -monitor unix:/tmp/autopkgtest-qemu.h0flvsby/monitor,server=on,wait=off -virtfs local,id=autopkgtest,path=/tmp/autopkgtest-qemu.h0flvsby/shared,security_model=none,mount_tag=autopkgtest -device virtio-serial -chardev socket,path=/tmp/autopkgtest-qemu.h0flvsby/hvc0,server=on,wait=off,id=hvc0 -device virtconsole,chardev=hvc0 -chardev socket,path=/tmp/autopkgtest-qemu.h0flvsby/hvc1,server=on,wait=off,id=hvc1 -device virtconsole,chardev=hvc1 -serial unix:/tmp/autopkgtest-qemu.h0flvsby/ttyS0,server=on,wait=off -serial unix:/tmp/autopkgtest-qemu.h0flvsby/ttyS1,server=on,wait=off -drive index=0,file=/tmp/autopkgtest-qemu.h0flvsby/overlay.img,format=qcow2,cache=unsafe,if=virtio,discard=unmap -enable-kvm -cpu host

The architecture flag is completely ignored.

[Heinrich Schuchardt (xypron)]

On arm64 the following seems to work:

autopkgtest -U --apt-pocket=proposed -a armhf -s *.dsc -- qemu --qemu-options="-cpu host,aarch64=off" autopkgtest-mantic-armhf.img

But the '-cpu host,aarch64=off' QEMU option should be derived automatically from the '-a armhf' argument.

[Mateus Rodrigues de Morais (mateus-morais)]

I ran into this. Found out that I need to specify --qemu-architecture [1] to autopkgtest-virt-qemu, so I believe your command would then be:

$ autopkgtest -U --apt-pocket=proposed -a armhf -s *.dsc -- qemu --qemu-architecture=$arch autopkgtest-mantic-armhf.img

Not sure if $arch should be arm or armhf.

But yes, I think this is redundant and should somehow be derived from autopkgtest's own -a flag.

[1] https://manpages.ubuntu.com/manpages/jammy/en/man1/autopkgtest-virt-qemu.1.html

[Heinrich Schuchardt (xypron)]

Thanks Mateus,

autopkgtest -U --apt-pocket=proposed -a armhf -s *.dsc -- qemu --qemu-architecture=arm autopkgtest-mantic-armhf.img

leads to usage of -cpu host,aarch64=off.

autopkgtest should add --dpkg-architecture=armhf when invoking autopkgtest-virt-qemu.

[Ubuntu QA Website (ubuntuqa)]

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
https://iso.qa.ubuntu.com/qatracker/reports/bugs/2036730

[Tim Andersson (andersson123)]

myself, Paride Legovini and Brian Murray have encountered a bug where we can't even run autopkgtest-buildvm-ubuntu-cloud with armhf. We similarly get to this:

EFI stub: Booting Linux Kernel...
EFI stub: Entering in SVC mode with MMU enabled
EFI stub: Loaded initrd from LINUX_EFI_INITRD_MEDIA_GUID device path
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services...

But can never get to the step of actually running autopkgtest.

For clarity we used the exact command line mentioned at the beginning of this bug:
autopkgtest-buildvm-ubuntu-cloud -v --arch armhf --release mantic

[Heinrich Schuchardt (xypron)]

"autopkgtest-buildvm-ubuntu-cloud -v --arch armhf --release mantic" on amd64 launches QEMU as

qemu-system-arm -machine virt -m 512 -smp 1 -nographic -net nic,model=virtio -net user,hostfwd=tcp:127.0.0.1:10022-:22 -object rng-random,filename=/dev/urandom,id=rng0 -device virtio-rng-pci,rng=rng0,id=rng-device0 -monitor unix:/tmp/autopkgtest-qemu.n4bb59zs/monitor,server=on,wait=off -virtfs local,id=autopkgtest,path=/tmp/autopkgtest-qemu.n4bb59zs/shared,security_model=none,mount_tag=autopkgtest -device virtio-serial -chardev socket,path=/tmp/autopkgtest-qemu.n4bb59zs/hvc0,server=on,wait=off,id=hvc0 -device virtconsole,chardev=hvc0 -chardev socket,path=/tmp/autopkgtest-qemu.n4bb59zs/hvc1,server=on,wait=off,id=hvc1 -device virtconsole,chardev=hvc1 -serial unix:/tmp/autopkgtest-qemu.n4bb59zs/ttyS0,server=on,wait=off -drive index=0,file=/tmp/autopkgtest-buildvm-ubuntu-cloud1bu3xpgi/mantic-server-cloudimg-armhf.img,format=qcow2,if=virtio,discard=unmap -drive index=1,file=/tmp/autopkgtest-buildvm-ubuntu-cloud1bu3xpgi/autopkgtest.seed,format=raw,if=virtio,discard=unmap,readonly -drive if=pflash,format=raw,unit=0,read-only=on,file=/usr/share/AAVMF/AAVMF32_CODE.fd -drive if=pflash,format=raw,unit=1,file=/tmp/autopkgtest-qemu.n4bb59zs/efivars.fd

So here the QEMU architecture looks right.

When I run this command I see no issue except for missing feedback to the user for many minutes:

BdsDxe: loading Boot0001 "UEFI Misc Device" from PciRoot(0x0)/Pci(0x5,0x0)
BdsDxe: starting Boot0001 "UEFI Misc Device" from PciRoot(0x0)/Pci(0x5,0x0)
EFI stub: Booting Linux Kernel...
EFI stub: Entering in SVC mode with MMU enabled
EFI stub: Loaded initrd from LINUX_EFI_INITRD_MEDIA_GUID device path
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services...
Moving image into final destination ./autopkgtest-mantic-armhf.img

The problem here seems to be using console=/dev/ttyS0 instead of console=/dev/ttyAMA0 on the kernel command line.

So we have two separate issues to resolve:

- serial console on the command line for armhf images
- setting --qemu-architecture=arm for the 'autopkgtest -a armhf' command.

[Heinrich Schuchardt (xypron)]

https://cloud-images.ubuntu.com/mantic/current/mantic-server-cloudimg-armhf.img contains file
/etc/default/grub.d/50-cloudimg-settings.cfg with line
GRUB_CMDLINE_LINUX_DEFAULT="console=tty1 console=ttyS0"

This line comes from livecd-rootfs:

live-build/ubuntu-cpc/hooks.d/chroot/999-cpc-fixes.chroot:189:
GRUB_CMDLINE_LINUX_DEFAULT="console=tty1 console=ttyS0"

So this needs to become architecture specific or maybe the line could be dropped.

[Heinrich Schuchardt (xypron)]

For fixing the serial console on armhf and arm64 a merge request was created:
https://code.launchpad.net/~xypron/livecd-rootfs/+git/livecd-rootfs/+merge/453082

[Tim Andersson (andersson123)]

https://code.launchpad.net/~xypron/livecd-rootfs/+git/livecd-rootfs/+merge/453886 new mp

[Paride Legovini (paride)]

At the moment we're waiting for Noble to be open for development, right?

@xypron once the fix lands in Noble, we'll need to SRU it to the other affected releases if we want the serial console to work there, I suppose?

It is still not entirely clear to me if the missing serial console was a regression that happened at some point, or if we never realized it was not working.

[Paride Legovini (paride)]

Fix uploaded to noble (which is still frozen, so the package is waiting for approval).

[Launchpad Janitor (janitor)]

This bug was fixed in the package livecd-rootfs - 24.04.2

---------------
livecd-rootfs (24.04.2) noble; urgency=medium

  * Enable snap preseeding with ppc64el images where /boot/vmlinux is used
    instead of /boot/vmlinuz. (LP: #2038957)

 -- Philip Roche <email address hidden> Thu, 26 Oct 2023 17:56:00 +0100

[Brian Murray (brian-murray)]

autopkgtest-buildvm-ubuntu-cloud now works with noble for armhf:

BdsDxe: loading Boot0001 "UEFI Misc Device" from PciRoot(0x0)/Pci(0x5,0x0)
BdsDxe: starting Boot0001 "UEFI Misc Device" from PciRoot(0x0)/Pci(0x5,0x0)
EFI stub: Booting Linux Kernel...
EFI stub: Entering in SVC mode with MMU enabled
EFI stub: Loaded initrd from LINUX_EFI_INITRD_MEDIA_GUID device path
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services...
Moving image into final destination ./autopkgtest-noble-armhf.img

Actually this output looks the same as comment #9 and there was an extended period of time without any feedback being provided.

[Brian Murray (brian-murray)]

I was looking at the build logs for the noble armhf CPC images and it seems that the latest daily image has not been built with the new version of livecd-rootfs.

[Brian Murray (brian-murray)]

I tested again with the latest image which is 20231031 and saw the folowing:

[ OK ] Reached target poweroff.target - System Power Off.
[ 1066.071229] evict_inodes inode 18c03077, i_count = 1, was skipped!
[ 1066.072466] evict_inodes inode f9eec572, i_count = 1, was skipped!
[ 1066.271793] reboot: Power down
Moving image into final destination ./autopkgtest-noble-armhf.img

So I was definitely seeing console output during the build process now!

[Brian Murray (brian-murray)]

Additionally, booting the image with some magic qemu command allowed me to log in:

Ubuntu Noble Numbat (development branch) autopkgtest ttyAMA0

asdf
Password:

Login incorrect
autopkgtest login: ubuntu
Password:
Welcome to Ubuntu Noble Numbat (development branch) (GNU/Linux 6.5.0-9-generic armv7l)

FTR this was how I booted the image I built:

 $ qemu-system-aarch64 \
-machine virt -m 4G -smp 4 -nographic \
-object rng-random,filename=/dev/urandom,id=rng0 \
-device virtio-rng-pci,rng=rng0,id=rng-device0 \
-drive if=pflash,format=raw,unit=0,read-only=on,file=/usr/share/AAVMF/AAVMF32_CODE.fd \
-device virtio-net-pci,netdev=eth0 \
-netdev user,id=eth0,hostfwd=tcp::8022-:22 \
-drive file=autopkgtest-noble-armhf.img,if=virtio,format=qcow2

[Brian Murray (brian-murray)]

After having built the autopkgtest image I tried to run an armhf autopkgtest and encountered the following:

 $ autopkgtest -U --apt-pocket=proposed -a armhf gzip -- qemu autopkgtest-noble-armhf.img
autopkgtest [05:28:35]: starting date and time: 2023-11-01 05:28:35-0700
autopkgtest [05:28:35]: version 5.28ubuntu1
autopkgtest [05:28:35]: host impulse; command line: /usr/bin/autopkgtest -U --apt-pocket=proposed -a armhf gzip -- qemu autopkgtest-noble-armhf.img
qemu-system-x86_64: terminating on signal 15 from pid 1286611 (/usr/bin/python3)
<VirtSubproc>: failure: timed out waiting for 'login prompt on serial console'
autopkgtest [05:29:36]: ERROR: testbed failure: unexpected eof from the testbed

I believe some of the previous comments regarding modifying how and or which qemu is called will address this.

[Tim Andersson (andersson123)]

Now that the livecd-rootfs changes have landed, I'm going to work on a fix for autopkgtest itself.

[Tim Andersson (andersson123)]

I tried a number of command combinations adding the new args, and none worked. Can you advise, xypron?

```
andersson123@duckstation7:~/canonical/code/autopkgtest$ ./runner/autopkgtest -U --apt-pocket=proposed -a armhf -s mawk -- qemu --qemu-architecture=arm autopkgtest-noble-armhf.img
autopkgtest [18:02:03]: starting date and time: 2023-11-01 18:02:03+0000
autopkgtest [18:02:03]: git checkout: f1594619 5.28ubuntu1 (patches unapplied)
autopkgtest [18:02:03]: host duckstation7; command line: ./runner/autopkgtest -U --apt-pocket=proposed -a armhf -s mawk -- qemu --qemu-architecture=arm autopkgtest-noble-armhf.img
qemu-system-arm: terminating on signal 15 from pid 1374981 (/usr/bin/python3)
<VirtSubproc>: failure: timed out waiting for 'login prompt on serial console'
autopkgtest [18:03:03]: ERROR: testbed failure: unexpected eof from the testbed
andersson123@duckstation7:~/canonical/code/autopkgtest$ ./runner/autopkgtest -U --apt-pocket=proposed -a armhf -s mawk -- qemu --dpkg-architecture=armhf autopkgtest-noble-armhf.img
autopkgtest [18:03:19]: starting date and time: 2023-11-01 18:03:19+0000
autopkgtest [18:03:19]: git checkout: f1594619 5.28ubuntu1 (patches unapplied)
autopkgtest [18:03:19]: host duckstation7; command line: ./runner/autopkgtest -U --apt-pocket=proposed -a armhf -s mawk -- qemu --dpkg-architecture=armhf autopkgtest-noble-armhf.img
qemu-system-arm: terminating on signal 15 from pid 1425854 (/usr/bin/python3)
<VirtSubproc>: failure: timed out waiting for 'login prompt on serial console'
autopkgtest [18:04:19]: ERROR: testbed failure: unexpected eof from the testbed
andersson123@duckstation7:~/canonical/code/autopkgtest$ ./runner/autopkgtest -U --apt-pocket=proposed -a armhf -s mawk -- qemu --qemu-architecture=arm autopkgtest-noble-armhf.img
autopkgtest [18:07:30]: starting date and time: 2023-11-01 18:07:30+0000
autopkgtest [18:07:30]: git checkout: f1594619 5.28ubuntu1 (patches unapplied)
autopkgtest [18:07:30]: host duckstation7; command line: ./runner/autopkgtest -U --apt-pocket=proposed -a armhf -s mawk -- qemu --qemu-architecture=arm autopkgtest-noble-armhf.img
qemu-system-arm: terminating on signal 15 from pid 1620808 (/usr/bin/python3)
<VirtSubproc>: failure: timed out waiting for 'login prompt on serial console'
autopkgtest [18:08:31]: ERROR: testbed failure: unexpected eof from the testbed
```

[Tim Andersson (andersson123)]

Invoking the earliest commented options in this thread gives the following:

```
andersson123@duckstation7:~/canonical/code/autopkgtest$ ./runner/autopkgtest -U --apt-pocket=proposed -a armhf -s mawk -- qemu --qemu-options="-cpu host,aarch64=off" autopkgtest-noble-armhf.img
autopkgtest [18:11:06]: starting date and time: 2023-11-01 18:11:06+0000
autopkgtest [18:11:06]: git checkout: f1594619 5.28ubuntu1 (patches unapplied)
autopkgtest [18:11:06]: host duckstation7; command line: ./runner/autopkgtest -U --apt-pocket=proposed -a armhf -s mawk -- qemu '--qemu-options=-cpu host,aarch64=off' autopkgtest-noble-armhf.img
qemu-system-x86_64: can't apply global host-x86_64-cpu.aarch64=off: Property 'host-x86_64-cpu.aarch64' not found
```

[Tim Andersson (andersson123)]

I am on amd64 though, so I'm not sure what the desired behaviour is there.

[Tim Andersson (andersson123)]

All that being said, here's an MP with a branch which contains what I hope is a fix for the original issue - armhf not running on arm64 with simply "-a armhf". xypron if you could test this on your machine, I'd appreciate it.

https://code.launchpad.net/~andersson123/ubuntu/+source/autopkgtest/+git/autopkgtest/+merge/455001

[Brian Murray (brian-murray)]

I'm still seeing a failure with Tim's patch and I believe how qemu is called still needs to be modified as I saw this:

 $ runner/autopkgtest -U --apt-pocket=proposed -a armhf gzip -- qemu /srv/vms/autopkgtest-noble-armhf.img
autopkgtest [23:57:52]: starting date: 2023-11-01
autopkgtest [23:57:52]: git checkout: f7cd36b ssh-setup/nova: fix ARCH having two lines of data
autopkgtest [23:57:52]: host impulse; command line: runner/autopkgtest -U --apt-pocket=proposed -a armhf gzip -- qemu /srv/vms/autopkgtest-noble-armhf.img
qemu-system-x86_64: terminating on signal 15 from pid 1302946 (/usr/bin/python3)
<VirtSubproc>: failure: timed out waiting for "login prompt on ttyS0"
autopkgtest [23:58:52]: ERROR: testbed failure: cannot send to testbed: [Errno 32] Broken pipe

[Tim Andersson (andersson123)]

Brian, did you try w/o my change using the command line args I detailed above? I couldn't get that to work...

[Skia (hyask)]

https://code.launchpad.net/~hyask/ubuntu/+source/autopkgtest/+git/autopkgtest/+ref/armhf-qemu-fix

This branch has Tim's relaxed fix that should cover more cases.

However, I'm still unable to run the autopkgtest successfully locally with armhf, due to a kernel stack-trace at boot time in the image I have (noble). I may or may not investigate further this specific issue, but in the meantime, feel free to grab the patch on my branch and give some feedback.

[Skia (hyask)]

Here is a working armhf command to manually boot a noble-armhf image:

```
qemu-system-arm \
-machine virt -m 3G -smp 2 -cpu cortex-a15 -nographic \
-object rng-random,filename=/dev/urandom,id=rng0 \
-device virtio-rng-pci,rng=rng0,id=rng-device0 \
-drive if=pflash,format=raw,unit=0,read-only=on,file=/usr/share/AAVMF/AAVMF32_CODE.fd \
-device virtio-net-pci,netdev=eth0 \
-netdev user,id=eth0,hostfwd=tcp::8022-:22 \
-drive file=autopkgtest-noble-armhf.img,if=virtio,format=qcow2
```

The main takeaways are:
* use of `qemu-system-arm`, not `qemu-system-aarch64`
* no more than 3G, as there currently is a firmware bug preventing the use of more memory
* `-smp 1` is not supported, be sure to give at least two cores

I'll now see about patching autopkgtest to craft that kind of commandline when calling `qemu` with `armhf`.

[Skia (hyask)]

Using the aforementioned branch, I managed to have a cross-architecture test run on my machine successfully.

Two things:

To get the test to pass, I need to increase the `timeout-reboot` parameter, giving me the following command line:
`autopkgtest -a armhf gzip -- qemu --timeout-reboot 90 autopkgtest-noble-armhf.img`
Indeed, running the tests in emulation can be quite long, and the default timeout is only 60 seconds, which is quite short for booting a whole system up to the login prompt. (About 65-75 seconds on my machine)

To manually boot the armhf image, here is a working command line:
```
qemu-system-arm \
-machine virt -m 3G -smp 1 -nographic \
-object rng-random,filename=/dev/urandom,id=rng0 \
-device virtio-rng-pci,rng=rng0,id=rng-device0 \
-drive if=pflash,format=raw,unit=0,read-only=on,file=/usr/share/AAVMF/AAVMF32_CODE.fd \
-device virtio-net-pci,netdev=eth0 \
-netdev user,id=eth0,hostfwd=tcp::8022-:22 \
-drive file=autopkgtest-noble-armhf.img,if=virtio,format=qcow2
```
A firmware bug indeed prevents using more than 3G of memory, giving the stack trace I saw.

[Paride Legovini (paride)]

I'll try to do a recap here to make sure we're on the same page.

1. There was an issue with the armhf images preventing the serial console to work, and thus preventing autopkgtest(-buildvm-ubuntu-cloud) to communicate with VMs based on those images. This is now fixed and the Noble images are working fine.

2. There is a bug in edk2 or qemu (LP: #2043471) preventing qemu-system-arm to boot armhf images with more than 3GB or RAM or so. This is not a bug in autopkgtest (and does not affect the autopkgtest-virt-qemu default RAM size, which is of 1GB). The right place to work on this is LP: #2043471.

3. We have an annoyance: there is currently no mechanism for `autopkgtest --architecture` to pass the requested architecture to the virt server. Ideally we should implement this somehow, e.g. by having autopkgtest pass the requested architecture to the virt server via an environment variable. I believe this mechanism should be virt-server-agnostic (as the env var would be). An approach like [1] entangles two components that should not be too tightly coupled (autopkgtest and the virt-server). We can think of good ways to implement this, but this will make upstreaming the cross-arch testing diff even more difficult (bigger diff, more design decisions to discuss). Provided that my understanding here is correct, I propose considering this a documentation issue, and adding a section to [2] explaining how to properly do cross-arch testing when using autopkgtest-virt-qemu. What do you think?

[1] https://code.launchpad.net/~hyask/ubuntu/+source/autopkgtest/+git/autopkgtest/+merge/455429
[2] https://wiki.ubuntu.com/ProposedMigration#autopkgtests

[Andreas Hasenack (ahasenack)]

@Paride, could you please review https://code.launchpad.net/~andersson123/ubuntu/+source/autopkgtest/+git/autopkgtest/+merge/455001 which is associated with this bug?

[Tim Andersson (andersson123)]

@ahasenack that MP I believe is superseded by work from @hyask

[Skia (hyask)]

Following @Paride's suggestion of making this a documentation issue, I added a section to the wiki page explaining how to run cross-architecture autopkgtest:
https://wiki.ubuntu.com/ProposedMigration#How_to_run_cross-architecture_autopkgtest_locally_using_QEMU

~~Closing with "Won't fix" as we don't have a proper path to a clean solution yet.~~ Feel free to change or re-open if needed.

EDIT: "Won't fix" cannot be selected for whatever reason in LP.

[Paride Legovini (paride)]

@Andreas that MP is now gone, but I suppose it was the one making autopkgtest(1) call autopkgtest-virt-qemu with --qemu-architecture (or --dpkg-architecture) if needed. While that may have worked in practice, it was a hacky solution with downsides, see point 3 of comment 31 above. We went for better documenting how to do cross-arch testing for now; using the autopkgtest-virt-qemu will require specifying the architecture explicitly.

Let us know if I'm mistaken and the MP was about something else.

[Steve Langasek (vorlon)]

An upload of livecd-rootfs to jammy-proposed has been rejected from the upload queue for the following reason: "reuploading with proper -v".

[Steve Langasek (vorlon)]

An upload of livecd-rootfs to jammy-proposed has been rejected from the upload queue for the following reason: "reuploading with correct -v".

[Andreas Hasenack (ahasenack)]

Hello Heinrich, or anyone else affected,

Accepted livecd-rootfs into lunar-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.828.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-lunar to verification-done-lunar. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-lunar. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Andreas Hasenack (ahasenack)]

Hello Heinrich, or anyone else affected,

Accepted livecd-rootfs into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.765.32 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

My test case in the description is actually incorrect as an armhf cloud image would need to be built with the version of livecd-rootfs from -proposed. Phil Roche did build those images for me though and I downloaded them from https://private-fileshare.canonical.com/~philroche/LP-2036730/. After which I booted the image with the following command (also seen in a previous comment of mine).

 $ qemu-system-aarch64 \
-machine virt -m 4G -smp 4 -nographic \
-object rng-random,filename=/dev/urandom,id=rng0 \
-device virtio-rng-pci,rng=rng0,id=rng-device0 \
-drive if=pflash,format=raw,unit=0,read-only=on,file=/usr/share/AAVMF/AAVMF32_CODE.fd \
-device virtio-net-pci,netdev=eth0 \
-netdev user,id=eth0,hostfwd=tcp::8022-:22 \
-drive file=lunar-server-cloudimg-armhf.img,if=virtio,format=qcow2

With the current daily lunar-server-cloudimg-armhf.img file I never saw any output when booting the image other than:

EFI stub: Booting Linux Kernel...
EFI stub: Entering in SVC mode with MMU enabled
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services...

With the image build with the version of livecd-rootfs from -proposed I saw the full boot sequence and was presented with a log-in prompt.

Ubuntu 23.04 ubuntu ttyAMA0
....
[ 253.999776] cloud-init[1368]: 2023-12-12 17:58:11,049 - cc_final_message.py[WARNING]: Used fallback datasource
[ OK ] Finished cloud-final.servi… Execute cloud user/final scripts.
[ OK ] Reached target cloud-init.target - Cloud-init target.

ubuntu login: ubuntu

So I'm setting this to verification-done for Lunar.

[Brian Murray (brian-murray)]

I went through the same process identified in comment #40 and also received tty output with a Jammy armhf image from Phil.

[ OK ] Started OpenBSD Secure Shell server.

Ubuntu 22.04.3 LTS ubuntu ttyAMA0

ubuntu login:

[Andreas Hasenack (ahasenack)]

Hello Heinrich, or anyone else affected,

Accepted livecd-rootfs into mantic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/23.10.57 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-mantic to verification-done-mantic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-mantic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

I went through the same process identified in comment #40 and also received tty output with a Mantic armhf image from Phil.

[ OK ] Started polkit.service - Authorization Manager.
         Starting ModemManager.service - Modem Manager...

Ubuntu 23.10 ubuntu ttyAMA0

ubuntu login:

[Launchpad Janitor (janitor)]

This bug was fixed in the package livecd-rootfs - 23.10.57

---------------
livecd-rootfs (23.10.57) mantic; urgency=medium

  [ Heinrich Schuchardt ]
  * arm: fix console parameter for ARM cloud-images (LP: #2036730)

 -- Brian Murray <email address hidden> Wed, 06 Dec 2023 11:30:35 -0800

[Andreas Hasenack (ahasenack)]

The verification of the Stable Release Update for livecd-rootfs has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package livecd-rootfs - 2.828.3

---------------
livecd-rootfs (2.828.3) lunar; urgency=medium

  [ Heinrich Schuchardt ]
  * arm: fix console parameter for ARM cloud-images (LP: #2036730)

livecd-rootfs (2.828.2) lunar; urgency=medium

  [ Steve Langasek ]
  * The chroot tmpfs mount should only be /var/lib/apt/lists, not
    /var/lib/apt; the latter breaks changes to /var/lib/apt/extended_states.
    (LP: #2036195).

livecd-rootfs (2.828.1) lunar; urgency=medium

  * Fix unminimize to correctly list packages. (LP: #1996489)
  * Install LXD snap from stable/ubuntu-<version> channel. (LP: #2036725)

 -- Steve Langasek <email address hidden> Thu, 07 Dec 2023 08:51:51 -0800

[Łukasz Zemczak (sil2100)]

Hello Heinrich, or anyone else affected,

Accepted livecd-rootfs into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.765.33 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello Heinrich, or anyone else affected,

Accepted livecd-rootfs into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.765.34 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Andrew Cloke (andrew-cloke)]

Adding verification-done-jammy after a +1 from Brian Murray in MM.

[Launchpad Janitor (janitor)]

This bug was fixed in the package livecd-rootfs - 2.765.34

---------------
livecd-rootfs (2.765.34) jammy; urgency=medium

  * Remove comments in the debian/control dependencies as it was causing the
    package build to fail.

livecd-rootfs (2.765.33) jammy; urgency=medium

  [ Steve Langasek ]
  * Use losetup instead of kpartx to resolve race conditions in riscv64
    image builds. LP: #2045797.

  [ Utkarsh Gupta ]
  * unminimize: Use lxd-installer to install LXD itself (LP: #2036725)

livecd-rootfs (2.765.32) jammy; urgency=medium

  [ Heinrich Schuchardt ]
  * arm: fix console parameter for ARM cloud-images (LP: #2036730)

livecd-rootfs (2.765.31) jammy; urgency=medium

  [ John Chittum ]
  * fix: add 6.5 kernel apparmor features to livecd-rootfs based on
    features of 6.5 in ubuntu/mantic. This will roll as HWE. (LP: #2037567)

livecd-rootfs (2.765.30) jammy; urgency=medium

  * Enable snap preseeding with ppc64el images where /boot/vmlinux is used
    instead of /boot/vmlinuz. (LP: #2038957)

livecd-rootfs (2.765.29) jammy; urgency=medium

  [ Steve Langasek ]
  * The chroot tmpfs mount should only be /var/lib/apt/lists, not
    /var/lib/apt; the latter breaks changes to /var/lib/apt/extended_states.
    (LP: #2036195).

livecd-rootfs (2.765.28) jammy; urgency=medium

  * Fix unminimize to correctly list packages. (LP: #1996489)
  * Install LXD snap from stable/ubuntu-<version> channel. (LP: #2036725)

livecd-rootfs (2.765.27) jammy; urgency=medium

  [ Thomas Bechtold ]
  * Do not modify /etc/ssh/sshd_config for ubuntu-cpc
    project builds (LP: #1968873)

 -- Łukasz 'sil2100' Zemczak <email address hidden> Mon, 15 Jan 2024 16:41:18 +0100

[Thomas Bechtold (toabctl)]

The fix created a regression on AWS (jammy image) where the serial port is always (also for arm) /dev/ttyS0 . See https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/connect-to-serial-console.html and an internal SF case.

The bug for tracking the regression is https://bugs.launchpad.net/cloud-images/+bug/2050873 .

[Paride Legovini (paride)]

On the autopkgtest side we triaged this as a documentation issue, see comment 31. Things now improved:

1. autopkgtest(1) now better documents what --architecture does.

2. Skia (hyask) greatly improved the documentation on the Ubuntu wiki [1] on how to run cross architecture tests, on when --architecture is needed, all correlated by working examples.

I think we can mark the relevant tasks as Fix Released.

[1] https://wiki.ubuntu.com/ProposedMigration#How_to_run_cross-architecture_autopkgtest_locally_using_QEMU