Charm lxd-profile race-condition with download from charmhub

[Adam Dyess]

juju 3.3 / juju 3.4

I observed situations where a locally deployed charm would have the correct lxd-profile from the charm applied (specifically security.privileged: true) but the same charm from charmhub would be deployed with the same profile -- but the security.privileged: true though applied to the profile wasn't actively available on the machine until it was restarted.

A juju team individual hypothesized that the controller starts the machine for the charm while simultaneously downloading the charm from charmhub. On a sufficiently quick machine/connection the download would finish in time to apply the profile before the container started... but there are potential situations where the machine would be started before the profile could be applied leaving the machine with the appearance of having the profile correct -- but it might still require a restart to engage the correct profile.

Using juju download first to pull the charm locally, then deploying the local charm changes the situation. The controller might not start the machine until after the client completes the upload to the controller. Because the upload is complete, juju can start the machine with the correct lxd profiles.

Restarting the machine did in-fact set the machine into security:privileged

We also observed starting new units in a model would start with the correct profile installed before the machine begin.

POTENTIAL WORKAROUNDS:

1) Deploy the charm into the model with a fake temporary application name. This downloads the charm to the controller, and creates a unit that is in error, then remove that application. Once primed, the charm can be deployed to the correct application name
2) after the model is created, but before the charms are deployed -- edit the model's lxd-profile. This opens a security vulnerability because every machine in the model will have the same kernel permissions where some units may not necessarily need those profile adjustments.

[Joseph Phillips (manadart)]

This isn't the only issue manifest by this behaviour.
See https://bugs.launchpad.net/juju/+bug/2059990.