Support for configuring lxd-profile for applications in bundles

[James Hebden]

As discussed with thumper -
Juju as of 2.5.0 allows charms to configure a LXD profile, using the lxd-profile.yaml embedded in a charm, to provide sensible configuration to be used when an application is deployed to a LXD controller.

It would be super useful if this could be extended to allow a lxd-profile statement to be configured in the application sections of a bundle, regardless of whether or not a charm has an existing lxd-profile.yaml, to customise the profile used when deploying LXD units of the specific application.

Example uses which come to mind are configuring uidmapping, nesting, or device passthrough in a clean way when deploying complex platforms such as OpenStack, where traditionally device passthrough for devices such as SR-IOV VNFs has had to be done manually, with scripts, or other tools, prior to deploying applications to LXD containers.

Another use case is configuring the uidmapping for volumes which are required to access data on the host machine when deploying applications to unprivileged containers - this would allow specific UIDs and GIDs to be specified as well as volumes to pass through from the host for services deployed to LXD controllers, where mounting a networked or local filesystem might be possible on the host, but not possible in an unprivileged container due to kernel drivers or permissions restrictions in the LXD container.

I think the most sensible configuration would be to have lxd-profile sections specified in the bundle merge with any existing charm-specified values, where the charm specifies a profile, prior to upload to the controller, so a single lxd-profile map could be sent to the controller and then applied by the LXD controller to the existing or new LXD profile used by deployed/deploying services.

Happy to discuss further if any of this isn't clear or if there are considerations I haven't though of - however I feel this change would allow Juju to be much more self sufficient as a tool to deploy workloads atop LXD, and allow users to better leverage LXD's functionality from Juju without resorting to manual hacks to profiles. The intrinsic benefit also of doing this in the bundle is there is then an artefact detailing the LXD configuration required for a deployed model to function correctly.

[Dmitrii Shcherbakov (dmitriis)]

Encountered the same need for nested containers with Contrail charms as they deploy docker containers which either requires collocation on bare-metal or usage of kvm as standard lxd profiles do not allow nesting and we do not control the code base of the charm to modify its profile.

Docker containers they run do not have separate network namespaces so they had access to all host interfaces.

As a result, using a workaround is necessary where a dummy charm is created with a profile, deployed with as many units as you need for the target application and the units of that application are placed into containers via service collocation.

[John A Meinel (jameinel)]

Working around a charm issue by *not fixing the charm* is not a feature we
really want to support. Because it just leads to fragmentation and nobody
else gets the benefit of the improvements you could have made to the
ecosystem.

I wouldn't be opposed to doing this where we need it, but it shouldn't be a
"the proper place doesn't have the right information, so let me put the
information in the wrapper around the right place".

Especially with things like network passthrough, those are places where the
devices need to be modeled in Juju. Again, we might be ok with a stopgap,
but we want to be careful that where it gets used, means we have a bug.
End-users shouldn't need to carefully craft workarounds in their bundles,
to make up for deficiencies in the charm.

On Thu, Mar 28, 2019 at 11:15 AM Dmitrii Shcherbakov <
<email address hidden>> wrote:

> Encountered the same need for nested containers with Contrail charms as
> they deploy docker containers which either requires collocation on bare-
> metal or usage of kvm as standard lxd profiles do not allow nesting and
> we do not control the code base of the charm to modify its profile.
>
> Docker containers they run do not have separate network namespaces so
> they had access to all host interfaces.
>
> As a result, using a workaround is necessary where a dummy charm is
> created with a profile, deployed with as many units as you need for the
> target application and the units of that application are placed into
> containers via service collocation.
>
> --
> You received this bug notification because you are subscribed to juju.
> Matching subscriptions: juju bugs
> https://bugs.launchpad.net/bugs/1822016
>
> Title:
> Support for configuring lxd-profile for applications in bundles
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1822016/+subscriptions
>

[Richard Harding (rharding)]

I just want to back John up on the principle side of it.

I'm sure there are one off use cases but I almost would prefer to see us advance the feature such that the charm could perhaps adapt/update the lxd-profile in use at runtime based on charm config vs seeing the bundles grow overriding ability. If the charm is responding to config and generating a profile than the person with the most knowledge/experience of running that software is able to take that more into account vs a situation where operators are running the charm in ways never imagined by the charm author exposing potentially harmful interactions that are very difficult to debug down the road.

This also is hurt in that bundles aren't directly tracked as entities in Juju so storing where these lxd profile changes come from, and how they'd adapt over time (through charm upgrades/etc) is a bit of a minefield.

I'd be much more in favor of finding ways of doing this via charm vs bundle.

[James Hebden (ec0)]

I am not married to any specific implementation. My main goal with raising this bug, is to have Juju grow a means for LXD configuration to be applied to a deployed application in a way that suits environmental and user requirements, rather than assuming any and all LXD customisations can be forecast in the form of a per-charm hard-coded lxd-profile.yaml file, which is the current supported method (and is really useful, already, btw).

In many cases, I agree with Richard, that growing the ability for charms to modify the LXD configuration or ideally profile for a deployed application would address this use case, by allowing charm authors to implement more user-friendly charm options which would update LXD configuration in a way that was friendly to the overall goals of a given charm. An example of this would be what Dmitrii is describing - we could add logic to selectively enable nesting to the charm hook code, or a more direct example would be adding an option to specify which GPUs or VNFs get passed through to Kubernetes worker or OpenStack nova applications, which the charm could then configure LXD to pass through, so hacking LXD profiles and charm config and restarting LXD units manually would not be required. This does seem like a fairly rigid and heavy-duty approach however, given at this point of the application lifecycle, the LXD unit would already be deployed, meaning changes would need to be applied and units restarted based on activity performed in a charm hook.

Ideally though, my goal is to have a generic mechanism for users to codify environmental requirements both in and outside a given charm's area of concern to be used when deploying LXD units. Part of this, is wanting to leverage the security and deployment benefits of LXD, without having to reach outside of Juju to configure and describe deployed models. I think adding a way to influence (overriding is maybe stronger language than necessary here) LXD configuration at the bundle level, leveraging existing controls to adjust LXD profiles prior to the container being launched, not only avoids container restarts after charm code comes in and modified LXD config, it also generally increases Juju's value in combination with LXD when it comes to modeling infrastructure and application deployments.

When it comes to use cases and how they could be solved -
* VNFs, SR-IOV -> these could be solved if Juju grew the ability to have a hook change LXD configuration or profiles. Agree that's a clean way to do it, given we could expose user-friendly options with detailed descriptions which would make this functionality self-documenting.
* Nesting -> I agree also that mode charms which require this know they require this ahead of time, so for instance, the Docker charms should/would set up a static lxd-profile.yaml to enable nesting, which would cover this off.
* UID management -> When using non-privileged containers (which we should be doing everything in our power to enable!) one aspect that becomes worth managing is how UID/GID pairs from the host are mapped through to containers. Where containers rely on data volumes mounted/present on the host machine, mapping UIDs and GIDs through to u...