logs mangled

[Peter Surda]

The fix of #1978422 mangles logs as `sed` modifies them while they are still open (at least that's my deduction). Applies to both cloud-init.log and cloud-init-output.log.

[Brett Holman (holmanb)]

Hi Peter. Thanks for reporting.

Can you please add more information?

I don't think we can do anything to help until we have a better understanding of both:

 1. what happened
 2. what you expected to happen.

Could you please include examples of these mangled logs?

[Peter Surda (surda)]

The log files /var/log/cloud-init.log and /var/log/cloud-init-output.log end prematurely, and the the last line of /var/log/cloud-init-output.log says

"Redacting sensitive logs due to invalid cloud-config user-data from /var/log/cloud-init.log"

I matched the message to a file `/var/lib/dpkg/info/cloud-init.postinst` on the system. This file contains a function `fix_1978422_redact_sensitive_logs_on_invalid_userdata_schema`. This function uses `sed -i ...` to modify both aforementioned log files. However, as I deduce, if the files are still open, this will result in them being truncated, which matches the symptoms I observe.

I matched the change source here: https://git.launchpad.net/cloud-init/commit/debian/cloud-init.postinst?h=ubuntu/focal-22.2-hotfix&id=7bb959fec6544e619117ad0883819c0598449f41

[Peter Surda (surda)]

After doing some thinking, maybe this only happens if cloud-init is installed/upgraded while it's already running.

[Peter Surda (surda)]

Upon further tests, there are more conditions:

So to trigger this, both of the following is needed:
1. use an include file as user-data
2. upgrade cloud init while cloud-init is running (e.g. when using `package_upgrade: true`)
3. there must be some specific combination of old and new version of cloud-init to trigger this

This will result in truncated logs.

[Brett Holman (holmanb)]

Thanks for the additional information.

[Peter Surda (surda)]

I did more tests, and it's even more complicated than I originally thought. If I just have an "#include", the logs don't get mangled. But if I also have a jinja2 template error, THEN the logs get mangled. Even though "cloud-init schema --system" when run manually complains about "#include", that alone doesn't appear to cause mangled logs. That only happens after there is also a jinja2 template error, even while still the "cloud-init schema --system" only keeps complaining about "#include", however the cloud-init-output.log complains about a jinja2 problem. If there is no jinja2 problem, then there is no error in cloud-init-output-log even though when run manually in a terminal, "cloud-init schema --system" still complains about "#include"

[James Falcon (falcojr)]

Given that this only happens if there is a Jinja error present, and most systems will have already upgraded at this point, I don't see much value in fixing this.

[Peter Surda (surda)]

I tend to agree, this requires a combination of multiple conditions and even then can be prevented by fixing a Jinja2 template error. I just didn't know what's happening and how to fix it. How about just amending the message, instead of:

Redacting sensitive logs due to invalid cloud-config user-data from /var/log/cloud-init.log

write:

Redacting sensitive logs due to invalid cloud-config user-data from /var/log/cloud-init.log. Note: if this is the last message you see in the log, try fixing your cloud-init Jinja2 template errors.

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/4003