Cloud-init received SIGTERM and is thereby unable to install packages or running runcmd completely

[Carbrich]

Fore some reason cloud-init failed to install package "SL". It receives a sigterm before trying to install package "SL" This only happens on CentOS 8. For all other packages I have the same issue. If I try to install packages within user-scrips "runcmd" I have the same issue. If I do a sleep or a ping within runcmd, I have the same issue.

If I remove /var/lib/cloud after a failed deployment and reboot the client, everything works fine. If i run the module with python the same way as cloud-init does, it works without exit 1.

It almost looks like some kind of race condition.

I packaged our base image with a daily build of cloud-init, to be sure it is not resolved in a later version of cloud-init. It has been tested with cloud-init 20.2+137.gc0450c02-1 and the behaviour is the same on cloud-init 18.5/19.4.

Our workflow; Image is getting build with packer and installed with all necessary items to run cloud-init.; Image is added to vCenter as a template; Foreman clones VM on vCenter; Machine boots, Vmware does the network configuration as described in cloud-init docs(Cloning on vCenter). Cloud-init installs puppet and registers client to puppet master.; In the background the image is imported in vCloud and more stuff happens for our monitoring etc..; For our CentOS 7 deployment street this workflow is working without any issues.

For more reading follow the link below, sorry for the mixed up data on the forum. I tried a lot! https://community.theforeman.org/t/cloud-init-does-not-install-puppet-but-runs-the-final-module-and-succeeds-according-cloud-init/20155/27

I also added cloud-init-1.tar.gz with the logs. The data is anonymized.

Your guys help will be welcome!

PS. Sometimes really sometimes it just works 1 out of 100 or so.

### Cloud-config ###
--------------------

#cloud-config
hostname: test05
fqdn: test05
manage_etc_hosts: true
users: {}
packages:
- sl
runcmd:
- touch /tmp/test

#cloud.cfg
cloud_init_modules:
 - bootcmd
 - growpart

cloud_config_modules:
 - runcmd
 - timezone
 - package-update-upgrade-install

cloud_final_modules:
 - scripts-per-once
 - scripts-per-boot
 - scripts-per-instance
 - scripts-user
 - phone-home
 - power_state_change
 - final-message

system_info:
  distro: rhel
  paths:
    cloud_dir: /var/lib/cloud
    templates_dir: /etc/cloud/templates
  ssh_svcname: sshd

# vim:syntax=yaml

### /etc/cloud/cloud.cfg.d/01_network.cfg ###
---------------------------------------------

network:
  config: disabled

### 10_foreman.cfg
cat /etc/cloud/cloud.cfg.d/10_foreman.cfg
datasource_list: [NoCloud]
datasource:
  NoCloud:
    seedfrom: http://foreman/userdata/

#
- &log_syslog |
   [handler_cloudLogHandler]
   class=handlers.SysLogHandler
   level=DEBUG
   formatter=simpleFormatter
   args=("/dev/log", handlers.SysLogHandler.LOG_USER)

### /etc/cloud/cloud.cfg.d/05_logging.cfg ###
---------------------------------------------

## This yaml formated config file handles setting
## logger information. The values that are necessary to be set
## are seen at the bottom. The top '_log' are only used to remove
## redundency in a syslog and fallback-to-file case.
##
## The 'log_cfgs' entry defines a list of logger configs
## Each entry in the list is tried, and the first one that
## works is used. If a log_cfg list entry is an array, it will
## be joined with '\n'.
_log:
 - &log_base |
   [loggers]
   keys=root,cloudinit

   [handlers]
   keys=consoleHandler,cloudLogHandler

   [formatters]
   keys=simpleFormatter,arg0Formatter

   [logger_root]
   level=DEBUG
   handlers=consoleHandler,cloudLogHandler

   [logger_cloudinit]
   level=DEBUG
   qualname=cloudinit
   handlers=
   propagate=1

   [handler_consoleHandler]
   class=StreamHandler
   level=WARNING
   formatter=arg0Formatter
   args=(sys.stderr,)

   [formatter_arg0Formatter]
   format=%(asctime)s - %(filename)s[%(levelname)s]: %(message)s

   [formatter_simpleFormatter]
   format=[CLOUDINIT] %(filename)s[%(levelname)s]: %(message)s
 - &log_file |
   [handler_cloudLogHandler]
   class=FileHandler
   level=DEBUG
   formatter=arg0Formatter
   args=('/var/log/cloud-init.log', 'a', 'UTF-8')
 - &log_syslog |
   [handler_cloudLogHandler]
   class=handlers.SysLogHandler
   level=DEBUG
   formatter=simpleFormatter
   args=("/dev/log", handlers.SysLogHandler.LOG_USER)

log_cfgs:
# Array entries in this list will be joined into a string
# that defines the configuration.
#
# If you want logs to go to syslog, uncomment the following line.
# - [ *log_base, *log_syslog ]
#
# The default behavior is to just log to a file.
# This mechanism that does not depend on a system service to operate.
 - [ *log_base, *log_file ]
# A file path can also be used.
# - /etc/log.conf

# This tells cloud-init to redirect its stdout and stderr to
# 'tee -a /var/log/cloud-init-output.log' so the user can see output
# there without needing to look on the console.
output: {all: '| tee -a /var/log/cloud-init-output.log'}

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/3775