cloud-init

cloud-init reporting does not report boot stage start;finish events for init-local, init-network, modules-config, modules-final

Bug #1992711 reported by Chad Smith on 2022-10-13

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	cloud-init	Expired	Undecided	Unassigned

Bug Description

cloud-init 22.3 reporting configuration doesn't report paired start/finish event_types for all of cloud-init's boot stages.

Expected behavior:
cloud-init boot stages (init-local, init-network, config-modules and config-final) to emit both event_type start and finish to represent entry and exit of each boot stage and result "SUCCESS" or "FAILURE".

Results seen:
Only intermittently see init-network finish events, otherwise strictly event children within the named scope of init-local/*, init-network/*, modules-config/* and modules-final/* publish both start and finish events.

# desired init-network event only seen sometimes.
{"name": "init-network", "description": "searching for network datasources", "event_type": "finish", "origin": "cloudinit", "timestamp": 1665613954.3995728, "result": "SUCCESS"}

Test Procedure:

# terminal 1 on LXC host setup simple http server to log POST name:event_type tuples

cat > report_server.py << EOF
#!/usr/bin/env python3

from http.server import BaseHTTPRequestHandler, HTTPServer
import logging
import json

class S(BaseHTTPRequestHandler):
    def _set_response(self):
        self.send_response(200)
        self.send_header('Content-type', 'text/plain')
        self.end_headers()

    def do_POST(self):
        content_length = int(self.headers['Content-Length'])
        post_data = self.rfile.read(content_length)
        json_data = json.loads(post_data.decode('utf-8'))
        msg = f"{json_data['name']}:{json_data['event_type']}"
        logging.info(msg)
        self._set_response()

def run(server_class=HTTPServer, handler_class=S, port=8080):
    logging.basicConfig(level=logging.INFO)
    server_address = ('', port)
    httpd = server_class(server_address, handler_class)
    logging.info('Starting httpd...\n')
    try:
        httpd.serve_forever()
    except KeyboardInterrupt:
        pass
    httpd.server_close()
    logging.info('Stopping httpd...\n')

if __name__ == '__main__':
from sys import argv

    if len(argv) == 2:
        run(port=int(argv[1]))
    else:
        run()

EOF

python3 report_server.py 2> /dev/null

#### Terminal 2
# Setup LXC container with a reporting configuration which will talk to the LXC host
cat > 99-reporting.cfg <<EOF
reporting:
  webserver:
    type: webhook
    endpoint: "http://_gateway:8080/"
    timeout: 5
    retries: 3
EOF

lxc launch ubuntu-daily:kinetic test-reporting
lxc exec test-reporting -- cloud-init status --wait
lxc file push 99-reporting.cfg test-reporting/etc/cloud/cloud.cfg.d
lxc exec test-reporting -- cloud-init clean --logs --reboot

# on the simple http server terminal we end up seeing only event children, we only occasionally see init-network:finish events. Typically the parent event start and finish is not logged and it is only child events within the boot stage init-local, init-network, modules-config and modules-final this gets published in event handler

init-local/check-cache:start
init-local/check-cache:finish
init-local/search-LXD:start
init-local/search-LXD:finish
init-network/check-cache:start
init-network/check-cache:finish
init-network/setup-datasource:start
init-network/setup-datasource:finish
init-network/consume-user-data:start
init-network/consume-user-data:finish
init-network/consume-vendor-data:start
init-network/consume-vendor-data:finish
init-network/consume-vendor-data2:start
init-network/consume-vendor-data2:finish
init-network/activate-datasource:start
init-network/activate-datasource:finish
init-network/config-migrator:start
init-network/config-migrator:finish
init-network/config-seed_random:start
init-network/config-seed_random:finish
init-network/config-growpart:start
init-network/config-growpart:finish
init-network/config-resizefs:start
init-network/config-resizefs:finish
init-network/config-mounts:start
init-network/config-mounts:finish
init-network/config-set_hostname:start
init-network/config-set_hostname:finish
init-network/config-update_hostname:start
init-network/config-update_hostname:finish
init-network/config-users-groups:start
init-network/config-users-groups:finish
init-network/config-ssh:start
init-network/config-ssh:finish
modules-config/config-ssh-import-id:start
modules-config/config-ssh-import-id:finish
modules-config/config-locale:start
modules-config/config-locale:finish
modules-config/config-set-passwords:start
modules-config/config-set-passwords:finish
modules-config/config-grub-dpkg:start
modules-config/config-grub-dpkg:finish
modules-config/config-apt-configure:start
modules-config/config-apt-configure:finish
modules-config/config-ubuntu-advantage:start
modules-config/config-ubuntu-advantage:finish
modules-config/config-byobu:start
modules-config/config-byobu:finish
modules-final/config-reset_rmc:start
modules-final/config-reset_rmc:finish
modules-final/config-refresh_rmc_and_interface:start
modules-final/config-refresh_rmc_and_interface:finish
modules-final/config-rightscale_userdata:start
modules-final/config-rightscale_userdata:finish
modules-final/config-scripts-vendor:start
modules-final/config-scripts-vendor:finish
modules-final/config-scripts-per-once:start
modules-final/config-scripts-per-once:finish
modules-final/config-scripts-per-boot:start
modules-final/config-scripts-per-boot:finish
modules-final/config-scripts-per-instance:start
modules-final/config-scripts-per-instance:finish
modules-final/config-scripts-user:start
modules-final/config-scripts-user:finish
modules-final/config-ssh-authkey-fingerprints:start
modules-final/config-ssh-authkey-fingerprints:finish
modules-final/config-keys-to-console:start
modules-final/config-keys-to-console:finish
modules-final/config-install-hotplug:start
modules-final/config-install-hotplug:finish
modules-final/config-final-message:start
modules-final/config-final-message:finish