ubuntu_lttng_smoke_test failed with module build timeout on AWS J-5.15 ARM64k instance a1.medium

Bug #1978438 reported by Po-Hsu Lin
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
ubuntu-kernel-tests
Confirmed
Undecided
Unassigned

Bug Description

Issue found on Jammy 5.15.0-36.37 ARM64k with AWS instance a1.medium (passed with ARM64 a1.medium)

This is not a regression, as this issue can be found since 5.15.0-28.29

The reason is that the lttng module will failed to build on this instance:

$ time sudo apt install lttng-modules-dkms
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  dkms lttng-tools
Suggested packages:
  menu
The following NEW packages will be installed:
  dkms lttng-modules-dkms lttng-tools
0 upgraded, 3 newly installed, 0 to remove and 14 not upgraded.
Need to get 0 B/1471 kB of archives.
After this operation, 7338 kB of additional disk space will be used.
Do you want to continue? [Y/n]
Selecting previously unselected package dkms.
(Reading database ... 113883 files and directories currently installed.)
Preparing to unpack .../dkms_2.8.7-2ubuntu2_all.deb ...
Unpacking dkms (2.8.7-2ubuntu2) ...
Selecting previously unselected package lttng-tools.
Preparing to unpack .../lttng-tools_2.13.4-1build1_arm64.deb ...
Unpacking lttng-tools (2.13.4-1build1) ...
Selecting previously unselected package lttng-modules-dkms.
Preparing to unpack .../lttng-modules-dkms_2.13.1-1_all.deb ...
Unpacking lttng-modules-dkms (2.13.1-1) ...
Setting up dkms (2.8.7-2ubuntu2) ...
Setting up lttng-modules-dkms (2.13.1-1) ...
Loading new lttng-modules-2.13.1 DKMS files...
Building for 5.15.0-37-generic-64k 5.15.0-1009-aws
Building initial module for 5.15.0-37-generic-64k
ERROR: Cannot create report: [Errno 17] File exists: '/var/crash/lttng-modules-dkms.0.crash'
Error! Bad return status for module build on kernel: 5.15.0-37-generic-64k (aarch64)
Consult /var/lib/dkms/lttng-modules/2.13.1/build/make.log for more information.
dpkg: error processing package lttng-modules-dkms (--configure):
 installed lttng-modules-dkms package post-installation script subprocess returned error exit status 10
Setting up lttng-tools (2.13.4-1build1) ...
Processing triggers for man-db (2.10.2-1) ...
Errors were encountered while processing:
 lttng-modules-dkms
needrestart is being skipped since dpkg has failed
E: Sub-process /usr/bin/dpkg returned an error code (1)

real 27m36.381s
user 0m0.004s
sys 0m0.018s

DKMS make.log for lttng-modules-2.13.1 for kernel 5.15.0-37-generic-64k (aarch64)
Mon Jun 13 03:32:05 UTC 2022
make: Entering directory '/usr/src/linux-headers-5.15.0-37-generic-64k'
/var/lib/dkms/lttng-modules/2.13.1/build/src/probes/Kbuild:166: Files ./fs/btrfs/*.h not found. Probe "btrfs" is disabled. Use full kernel source tree to enable it.
/var/lib/dkms/lttng-modules/2.13.1/build/src/probes/Kbuild:177: Files ./fs/ext4/*.h not found. Probe "ext4" is disabled. Use full kernel source tree to enable it.
/var/lib/dkms/lttng-modules/2.13.1/build/src/probes/Kbuild:208: File ./drivers/base/regmap/trace.h not found. Probe "regmap" is disabled. Need Linux 4.1+ kernel source tree to enable it.
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/lttng-probe-sched.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/lttng-probe-irq.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/lttng-probe-timer.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/lttng-probe-kmem.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/lttng-probe-module.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/lttng-probe-power.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/lttng-probe-statedump.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/lttng-probe-9p.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/lttng-probe-i2c.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/lttng-probe-kvm.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/lttng-probe-signal.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/lttng-probe-block.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/lttng-probe-napi.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/lttng-probe-skb.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/lttng-probe-net.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/lttng-probe-sock.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/lttng-probe-udp.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/lttng-probe-asoc.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/lttng-probe-gpio.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/lttng-probe-jbd2.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/lttng-probe-regulator.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/lttng-probe-scsi.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/lttng-probe-vmscan.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/lttng-probe-compaction.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/lttng-probe-printk.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/lttng-probe-random.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/lttng-probe-rcu.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/lttng-probe-sunrpc.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/lttng-probe-v4l2.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/lttng-probe-workqueue.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/lttng-probe-writeback.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/lttng-kprobes.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/lttng-uprobes.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/lttng-kretprobes.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lib/ringbuffer/ring_buffer_backend.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lib/ringbuffer/ring_buffer_frontend.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lib/ringbuffer/ring_buffer_iterator.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lib/ringbuffer/ring_buffer_vfs.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lib/ringbuffer/ring_buffer_splice.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lib/ringbuffer/ring_buffer_mmap.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lib/prio_heap/lttng_prio_heap.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lib/../wrapper/splice.o
  LD [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lib/lttng-lib-ring-buffer.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lib/counter/counter.o
  LD [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lib/lttng-counter.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/tests/probes/lttng-test.o
  LD [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/tests/lttng-test.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/tests/clock-plugin/lttng-clock-plugin-test.o
  LD [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/tests/lttng-clock-plugin-test.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-ring-buffer-client-discard.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-ring-buffer-client-overwrite.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-ring-buffer-metadata-client.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-ring-buffer-client-mmap-discard.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-ring-buffer-client-mmap-overwrite.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-ring-buffer-metadata-mmap-client.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-ring-buffer-event-notifier-client.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-counter-client-percpu-32-modular.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-counter-client-percpu-64-modular.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-clock.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lib/msgpack/msgpack.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-events.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-abi.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-string-utils.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-probes.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-context.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-context-pid.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-context-procname.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-context-prio.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-context-nice.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-context-vpid.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-context-tid.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-context-vtid.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-context-ppid.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-context-vppid.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-context-cpu-id.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-context-uid.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-context-euid.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-context-suid.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-context-gid.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-context-egid.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-context-sgid.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-context-vuid.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-context-veuid.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-context-vsuid.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-context-vgid.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-context-vegid.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-context-vsgid.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-context-interruptible.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-context-need-reschedule.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-calibrate.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-context-hostname.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-context-callstack.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/lttng.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-tracker-id.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-bytecode.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-bytecode-interpreter.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-bytecode-specialize.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-bytecode-validator.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/probes/lttng-probe-user.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-tp-mempool.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-event-notifier-notification.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-syscalls.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-syscalls-entry-table.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-syscalls-entry-compat-table.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-syscalls-exit-table.o
  CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-syscalls-exit-compat-table.o
gcc: fatal error: Killed signal terminated program cc1
compilation terminated.
make[2]: *** [scripts/Makefile.build:285: /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-syscalls-exit-compat-table.o] Error 1
make[1]: *** [scripts/Makefile.build:548: /var/lib/dkms/lttng-modules/2.13.1/build/src] Error 2
make: *** [Makefile:1875: /var/lib/dkms/lttng-modules/2.13.1/build] Error 2
make: Leaving directory '/usr/src/linux-headers-5.15.0-37-generic-64k'

At the very beginning I thought it's because the ram is too small on this instance:
$ free -h
               total used free shared buff/cache available
Mem: 1.8Gi 425Mi 1.0Gi 15Mi 347Mi 1.1Gi
Swap: 0B 0B 0B

But if you install the generic arm64 kernel on this node and remove the generic-64k one, the module can be built in about 14 minutes (for the generic and aws kernel, with just the generic kernel it will be just 7 minutes)

Po-Hsu Lin (cypressyew)
tags: added: 5.15 arm64 aws jammy ubuntu-lttng-smoke-test
tags: added: sru-20220530
description: updated
Po-Hsu Lin (cypressyew)
tags: added: sru-20220509
description: updated
Changed in ubuntu-kernel-tests:
status: New → Confirmed
description: updated
Po-Hsu Lin (cypressyew)
description: updated
Po-Hsu Lin (cypressyew)
description: updated
Po-Hsu Lin (cypressyew)
description: updated
Revision history for this message
Po-Hsu Lin (cypressyew) wrote :

I tried to bump the timeout from 20 minutes to 40 minutes, but it's still failing.

Looks like it's been killed by the system:
04:41:13 DEBUG| [stdout] Setting up lttng-modules-dkms (2.13.1-1) ...
04:41:13 DEBUG| [stdout] Loading new lttng-modules-2.13.1 DKMS files...
04:41:15 DEBUG| [stdout] Building for 5.15.0-40-generic-64k 5.15.0-1013-aws
04:41:15 DEBUG| [stdout] Building initial module for 5.15.0-40-generic-64k
05:07:27 DEBUG| [stdout] ERROR (dkms apport): kernel package linux-headers-5.15.0-40-generic-64k is not supported
05:07:27 DEBUG| [stdout] Error! Bad return status for module build on kernel: 5.15.0-40-generic-64k (aarch64)
05:07:27 DEBUG| [stdout] Consult /var/lib/dkms/lttng-modules/2.13.1/build/make.log for more information.

In the build log:
05:07:32 DEBUG| [stdout] CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-syscalls-entry-compat-table.o
05:07:32 DEBUG| [stdout] CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-syscalls-exit-table.o
05:07:32 DEBUG| [stdout] CC [M] /var/lib/dkms/lttng-modules/2.13.1/build/src/lttng-syscalls-exit-compat-table.o
05:07:32 DEBUG| [stdout] gcc: fatal error: Killed signal terminated program cc1
05:07:32 DEBUG| [stdout] compilation terminated.

Revision history for this message
Po-Hsu Lin (cypressyew) wrote :

This is because this a1.medium instance is too small, because of this, the test won't even try to use swapon to workaround this issue.

$ free -m
               total used free shared buff/cache available
Mem: 1838 441 792 15 605 1115
Swap: 0 0 0

$ df -h
Filesystem Size Used Avail Use% Mounted on
tmpfs 184M 16M 169M 9% /run
/dev/nvme0n1p1 7.6G 4.0G 3.6G 53% /
tmpfs 920M 64K 920M 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/nvme0n1p15 98M 5.1M 93M 6% /boot/efi
tmpfs 184M 64K 184M 1% /run/user/1000

The criteria to create a swap is:
  if mem_gb < 3.0 and df_gb > 4.0 and swapon == '':

Po-Hsu Lin (cypressyew)
Changed in ubuntu-kernel-tests:
assignee: nobody → Po-Hsu Lin (cypressyew)
status: Confirmed → In Progress
Revision history for this message
Po-Hsu Lin (cypressyew) wrote :

We will take care of this with test rework in Avocado.
Keep this open.

Changed in ubuntu-kernel-tests:
assignee: Po-Hsu Lin (cypressyew) → nobody
Po-Hsu Lin (cypressyew)
Changed in ubuntu-kernel-tests:
status: In Progress → Confirmed
Po-Hsu Lin (cypressyew)
summary: - ubuntu_lttng_smoke_test failed with module build timeout on AWS J-5.15
+ ubuntu_lttng_smoke_test failed with module build timeout on AWS K-5.19
ARM64k instance a1.medium
summary: - ubuntu_lttng_smoke_test failed with module build timeout on AWS K-5.19
+ ubuntu_lttng_smoke_test failed with module build timeout on AWS J-5.15
ARM64k instance a1.medium
Revision history for this message
Po-Hsu Lin (cypressyew) wrote :

This affects Kinetic as well, and note that we got this test disabled in CKCT for Jammy.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.