Failed to compile lttng module on small Azure nodes

[Po-Hsu Lin]

Node:
  * BasicA1
  * StandardA0
  * StandardB1s

Running 'cat /var/lib/dkms/lttng-modules/*/build/make.log'
 DKMS make.log for lttng-modules-2.10.5 for kernel 4.15.0-1023-azure (x86_64)
 Thu Aug 30 14:27:53 UTC 2018
 make: Entering directory '/usr/src/linux-headers-4.15.0-1023-azure'
 Makefile:976: "Cannot use CONFIG_STACK_VALIDATION=y, please install libelf-dev, libelf-devel or elfutils-libelf-devel"
 CC [M] /var/lib/dkms/lttng-modules/2.10.5/build/lttng-ring-buffer-client-discard.o
 CC [M] /var/lib/dkms/lttng-modules/2.10.5/build/lttng-ring-buffer-client-overwrite.o
 CC [M] /var/lib/dkms/lttng-modules/2.10.5/build/lttng-ring-buffer-metadata-client.o
 CC [M] /var/lib/dkms/lttng-modules/2.10.5/build/lttng-ring-buffer-client-mmap-discard.o
 CC [M] /var/lib/dkms/lttng-modules/2.10.5/build/lttng-ring-buffer-client-mmap-overwrite.o
 CC [M] /var/lib/dkms/lttng-modules/2.10.5/build/lttng-ring-buffer-metadata-mmap-client.o
 CC [M] /var/lib/dkms/lttng-modules/2.10.5/build/lttng-clock.o
 CC [M] /var/lib/dkms/lttng-modules/2.10.5/build/lttng-events.o
 CC [M] /var/lib/dkms/lttng-modules/2.10.5/build/lttng-abi.o
 CC [M] /var/lib/dkms/lttng-modules/2.10.5/build/lttng-string-utils.o
 CC [M] /var/lib/dkms/lttng-modules/2.10.5/build/lttng-probes.o
 CC [M] /var/lib/dkms/lttng-modules/2.10.5/build/lttng-context.o
 CC [M] /var/lib/dkms/lttng-modules/2.10.5/build/lttng-context-pid.o
 CC [M] /var/lib/dkms/lttng-modules/2.10.5/build/lttng-context-procname.o
 CC [M] /var/lib/dkms/lttng-modules/2.10.5/build/lttng-context-prio.o
 CC [M] /var/lib/dkms/lttng-modules/2.10.5/build/lttng-context-nice.o
 CC [M] /var/lib/dkms/lttng-modules/2.10.5/build/lttng-context-vpid.o
 CC [M] /var/lib/dkms/lttng-modules/2.10.5/build/lttng-context-tid.o
 CC [M] /var/lib/dkms/lttng-modules/2.10.5/build/lttng-context-vtid.o
 CC [M] /var/lib/dkms/lttng-modules/2.10.5/build/lttng-context-ppid.o
 CC [M] /var/lib/dkms/lttng-modules/2.10.5/build/lttng-context-vppid.o
 CC [M] /var/lib/dkms/lttng-modules/2.10.5/build/lttng-context-cpu-id.o
 CC [M] /var/lib/dkms/lttng-modules/2.10.5/build/lttng-context-interruptible.o
 CC [M] /var/lib/dkms/lttng-modules/2.10.5/build/lttng-context-need-reschedule.o
 CC [M] /var/lib/dkms/lttng-modules/2.10.5/build/lttng-calibrate.o
 CC [M] /var/lib/dkms/lttng-modules/2.10.5/build/lttng-context-hostname.o
 CC [M] /var/lib/dkms/lttng-modules/2.10.5/build/wrapper/random.o
 CC [M] /var/lib/dkms/lttng-modules/2.10.5/build/probes/lttng.o
 CC [M] /var/lib/dkms/lttng-modules/2.10.5/build/wrapper/trace-clock.o
 CC [M] /var/lib/dkms/lttng-modules/2.10.5/build/wrapper/page_alloc.o
 CC [M] /var/lib/dkms/lttng-modules/2.10.5/build/lttng-tracker-pid.o
 CC [M] /var/lib/dkms/lttng-modules/2.10.5/build/lttng-filter.o
 CC [M] /var/lib/dkms/lttng-modules/2.10.5/build/lttng-filter-interpreter.o
 CC [M] /var/lib/dkms/lttng-modules/2.10.5/build/lttng-filter-specialize.o
 CC [M] /var/lib/dkms/lttng-modules/2.10.5/build/lttng-filter-validator.o
 CC [M] /var/lib/dkms/lttng-modules/2.10.5/build/probes/lttng-probe-user.o
 CC [M] /var/lib/dkms/lttng-modules/2.10.5/build/lttng-syscalls.o
 virtual memory exhausted: Cannot allocate memory
 scripts/Makefile.build:332: recipe for target '/var/lib/dkms/lttng-modules/2.10.5/build/lttng-syscalls.o' failed
 make[1]: *** [/var/lib/dkms/lttng-modules/2.10.5/build/lttng-syscalls.o] Error 1
 Makefile:1552: recipe for target '_module_/var/lib/dkms/lttng-modules/2.10.5/build' failed

[Po-Hsu Lin (cypressyew)]

Memory usage on Basic A1:
azure@b-la-azure-4-15-0-Basic-A1-aio-dio-bugs:~$ free -m
              total used free shared buff/cache available
Mem: 1678 230 1279 0 168 1252
Swap: 0 0 0

Adding a 1G swap as suggested by Colin can solve this issue.

[Po-Hsu Lin (cypressyew)]

Memory / Disk usage on Standard_B1s:

azure@t-la-azure-4-15-0-Standard-B1s-iosched-bugs:~$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 454M 12K 454M 1% /dev
tmpfs 93M 332K 92M 1% /run
/dev/sda1 30G 1.4G 27G 5% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
none 5.0M 0 5.0M 0% /run/lock
none 462M 0 462M 0% /run/shm
none 100M 0 100M 0% /run/user
/dev/sdb1 3.9G 8.0M 3.7G 1% /mnt
azure@t-la-azure-4-15-0-Standard-B1s-iosched-bugs:~$ free -m
             total used free shared buffers cached
Mem: 922 653 268 0 11 419
-/+ buffers/cache: 222 699
Swap: 0 0 0

[Po-Hsu Lin (cypressyew)]

Memory / Disk usage on Standard_A0:

azure@t-la-azure-4-15-0-Standard-A0-iosched-bugs:~$ free -m
             total used free shared buffers cached
Mem: 670 599 70 0 23 354
-/+ buffers/cache: 221 448
Swap: 0 0 0
azure@t-la-azure-4-15-0-Standard-A0-iosched-bugs:~$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 328M 12K 328M 1% /dev
tmpfs 68M 332K 67M 1% /run
/dev/sda1 30G 1.5G 27G 6% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
none 5.0M 0 5.0M 0% /run/lock
none 336M 0 336M 0% /run/shm
none 100M 0 100M 0% /run/user
/dev/sdb1 20G 44M 19G 1% /mnt

[Po-Hsu Lin (cypressyew)]

Error message on Standard_A0:
  CC [M] /var/lib/dkms/lttng-modules/2.10.5/build/lttng-filter.o
  CC [M] /var/lib/dkms/lttng-modules/2.10.5/build/lttng-filter-interpreter.o
  CC [M] /var/lib/dkms/lttng-modules/2.10.5/build/lttng-filter-specialize.o
  CC [M] /var/lib/dkms/lttng-modules/2.10.5/build/lttng-filter-validator.o
  CC [M] /var/lib/dkms/lttng-modules/2.10.5/build/probes/lttng-probe-user.o
  CC [M] /var/lib/dkms/lttng-modules/2.10.5/build/lttng-syscalls.o
gcc: internal compiler error: Killed (program cc1)
Please submit a full bug report,

[Colin Ian King (colin-king)]

Looks like out of memory killing gcc again.

[Po-Hsu Lin (cypressyew)]

Patch sent, it will create a swapfile for a node that:
  * Running with Azure kernel
  * No swap
  * < 1.8 GB ram
  * > 4 GB free disk space

[Po-Hsu Lin (cypressyew)]

http://kernel.ubuntu.com/git/ubuntu/autotest-client-tests.git/commit/?id=2e48b876459358e2f1eba8f999c513ddd4fbdc9d

[Arnis Jaundzeikars (sangsatori)]

Still affecting Standard B1ls nodes.

Ubuntu 18.04 with Linux 4.18; system info --

uname -r
4.18.0-1013-azure

uname -v
#13~18.04.1-Ubuntu SMP Thu Feb 28 23:48:47 UTC 2019

Encountered with DKMS build failing with --

...
gcc: internal compiler error: Killed (program cc1)

Available instance memory --

free -m
              total used free shared buff/cache available
Mem: 415 220 18 0 177 184
Swap: 2047 44 2003

Adding a 2G swap file solved the issue.