Comment 2 for bug 1973628

Revision history for this message
Chad Smith (chad.smith) wrote : Re: autopkgtest-buildvm-ubuntu-cloud fails on ppc64el in cloud-init

This does not seem to affect x86_64.

In reviewing the #cloud-config user-data provided by autopkgtest-buildvm-ubuntu-cloud it is valid YAML and valid #cloud-config user-data. So, invalid YAML user-data format isn't the problem here. it is something uring reading and processing and that is leading to something trampling on the Reader instance. I have seen both stacktraces failing at line reader.py:105 int' object has no attribute 'index' and line 111 'int' object has no attribute 'column'.

I'm able to reproduce on openstack PPC64LE Jammy VMs and qemu kvm locally emulating ppc64le with the following:

sudo apt install autopkgtest qemu-utils genisoimage qemu-system-ppc
autopkgtest-buildvm-ubuntu-cloud -v --release jammy

It seems to be only an intermittent issue on first boot only for me in qemu kvm launches via virt-manager. Some boots experience it, some don't. I ran cloudinit's safeyaml.load to ensure it's not doing something terrible w.r.t. memory management as it reads YAML and things seem ok there.

I also see this failure occurring at different cloud-init boot stages. It doesn't seem confined to just the 'init' boot stage. Sometimes it happens in init-local, init or modules:config stages as al cloud-config yaml files are loaded.

Invoking the same logical path after initial boot succeeds without error 100% of the time. I can occasionally see OOMs on test ppc64le VMs in question during early boot which could lead to some symptoms due to resource contention.

An example of the OOM on ppc64le is below in cloud-init's modules:config boot stage related to locale generation. Just an indicator that the system under load with general memory contention. Not certain this is the cause of the problem.

This seems intermittent on my test envs.[ 364.196460] cloud-init[1305]: Cloud-init v. 22.2-0ubuntu1~22.04.2 running 'modules:config' at Tue, 28 Jun 2022 16:14:20 +0000. Up 362.41 seconds.
[ 365.296377] cloud-init[1305]: Generating locales (this might take a while)...
[ 380.015993] Out of memory: Killed process 1324 (localedef) total-vm:133120kB, anon-rss:118784kB, file-rss:576kB, shmem-rss:0kB, UID:0 pgtables:45kB oom_score_adj:0
[ 380.436996] cloud-init[1305]: en_US.UTF-8...
Cloud-init 22.2-0ubuntu1~22.04.2 received SIGTERM, exiting...
  Filename: /usr/lib/python3.10/subprocess.py
  Function: _try_wait
  Line number: 1896
    Filename: /usr/lib/python3.10/subprocess.py
    Function: _wait
    Line number: 1938
      Filename: /usr/lib/python3.10/subprocess.py
      Function: wait
      Line number: 1204