libvirt cannot kill dhcp in containers

[Serge Hallyn]

If I create a ubuntu container and start it as a libvirt-lxc container, it runs under the host dhcp profile. Then when I try virsh -c lxc:/// destroy c1, libvirtd tries to kill dhcp in the container but fails:

Jun 5 17:54:14 t1 kernel: [ 2563.620698] type=1400 audit(1401983654.375:28): apparmor="DENIED" operation="signal" profile="/sbin/dhclient" pid=4304 comm="libvirtd" requested_mask="receive" denied_mask="receive" signal=term peer="/usr/sbin/libvirtd"
Jun 5 17:54:14 t1 kernel: [ 2563.660491] type=1400 audit(1401983654.415:29): apparmor="DENIED" operation="signal" profile="/sbin/dhclient" pid=4293 comm="libvirtd" requested_mask="receive" denied_mask="receive" signal=term peer="/usr/sbin/libvirtd"
Jun 5 17:54:14 t1 kernel: [ 2563.660600] type=1400 audit(1401983654.415:30): apparmor="DENIED" operation="signal" profile="/sbin/dhclient" pid=4293 comm="libvirtd" requested_mask="receive" denied_mask="receive" signal=term peer="/usr/sbin/libvirtd"

I don't actually understand the mechanisms here (that a profile should be able to refuse receiving signals), and it seems like the proper fix is to have libvirt-lxc start containers confined in a container policy, but Jamie seemed to have another solution, which would be great.

[Jamie Strandboge (jdstrand)]

libvirt-lxc should start containers in a container policy, but I guess these needs probably significant upstream work since libvirt-lxc and lxc are not the same project.

The way signals work with AppArmor is that there are two checks-- can the sending process send a signal to another process and can the target process receive the signal from the sending process. The AppArmor base abstraction has the following:
  # Allow unconfined processes to send us signals by default
  signal (receive) peer=unconfined,
  # Allow us to signal ourselves
  signal peer=@{profile_name},
  # Checking for PID existence is quite common so add it by default for now
  signal (receive, send) set=("exists"),

In the case of libvirt-lxc and dhclient, libvirtd runs under confinement so the 'peer' is not 'unconfined' but rather the AppArmor label (in this case, /usr/sbin/libvirtd) which is why we have the denial.

In thinking about this more, we will have a similar issue with any confined process in the container (eg, mysql). Perhaps the best way forward is to not run libvirtd under confinement at all. Historically we did this because change_profile could not be done by a process that was unconfined-- I believe this requirement is lifted now. The current policy for libvirt is extremely permissive and offers very little protection anyway.

[John Johansen (jjohansen)]

It is a policy cross consistency check, and allows for writing policy that is selective in who it will communicate with, think rild which needs assurances it is talking to only a specific set of processes, instead of having to examine every profile to ensure the send rules are correct, only the receivers policy needs to be examined.

The fix is a small update to the dhclient profile.

  signal peer=/usr/sbin/libvirtd,

or even tighter
  signal (receive) set=(term) peer=/usr/sbin/libvirtd,

[John Johansen (jjohansen)]

Actually change_profile has always been allowed by confined processes. It just requires a rule in the profile.

My question here is who is running dclient and why? Is dhclient being started by libvirt as part of its network setup for the container? Or is it a dhclient being run in the container.

[Serge Hallyn (serge-hallyn)]

the container is running dhclient, not libvirt.

Which is why I don't think modifying dhcp's profile sounds like the right fix. Unfortunately it may be the easiest one.

[John Johansen (jjohansen)]

Its the temporary fix. The correct long term fix is stacking and having the container have a different profile namespace.

[Jamie Strandboge (jdstrand)]

John and I agree that probably the best thing is to no longer ship an apparmor profile for libvirtd. That would give it the "unconfined" label and it would therefore be able to kill processes in the libvirt-lxc container.

That said, using libvirt-lxc at all is likely pretty dangerous operation if you don't trust the container since there isn't any apparmor confinement for the container like with lxc. Even if you did trust the container, apparmor policy could be loaded from within the container and apply to the host system, which is obviously bad and can lead to unpredictable behavior. It might be a reasonable idea to either disable the feature or invest time in a libvirt-lxc apparmor driver (istr quite a bit of work on an selinux driver for libvirt-lxc and may remember others trying to get apparmor support too).

[Jamie Strandboge (jdstrand)]

Another idea would be to make sure that at least Ubuntu systems don't load apparmor policy in the container when using libvirt-lxc like we've done with lxc. This would at least make trusted Ubuntu containers to work reasonably well and not mess up the host policy.

[Seth Arnold (seth-arnold)]

Why does libvirtd run under a dhclient profile? Something about this seems wrong to me.

[Serge Hallyn (serge-hallyn)]

Summary:

libvirt runs under its own profile; it starts a container which starts
dhclient which runs under dhclient's profile; libvirt, bc it is confined,
cannot signal dhclient unless dhclient's policy allows that.

Ideally libvirt-lxc would start containers under a lxc-libvirt-container
policy, so that dhclient in the container stayed in that policy (or
entered a stacked policy). Until we have time to do that, having
libvirt run unconfined would bypass the restriction on dhclient accepting
signals.

[Serge Hallyn (serge-hallyn)]

Quoting Jamie Strandboge (<email address hidden>):
> Another idea would be to make sure that at least Ubuntu systems don't
> load apparmor policy in the container when using libvirt-lxc like we've
> done with lxc.

That actually is not happening with lxc. If you run a container
unconfined (lxc.aa_profile = unconfined) then dhclient does end up
running as /sbin/dhclient (enforce)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.