[Impact]
TL;DR
libvirt-bin stops working after a release upgrade from Trusty to Xenial. Other scenarios possible.
Workaround after it breaks:
sudo touch /etc/apparmor.d/{usr.lib.libvirt.virt-aa-helper,usr.sbin.libvirtd}
sudo apt install --reinstall libvirt-bin
The libvirt-bin package in Trusty and Xenial reloads the apparmor profile in postinst, but without taking into account possible apparmor caches. It uses this call:
apparmor_parser -r <profile> || true
instead of what dh_apparmor and every other package is using nowadays, and is also recommended in the apparmor_parser manpage:
apparmor_parser -r -T -W <profile> || true
Where:
-T: skip reading any existing cache
-W: write out the cache
The apparmor_parser(8) manpage has this to say about how the apparmor cache is considered:
"""
By default, if a profile's cache is found in the location specified by --cache-loc and the timestamp is newer than the profile, it will be loaded from the cache.
"""
That is reasonable behaviour. After all, the cache is generated from the profile file. If someone wants to change the profile, it will be edited and thus have a more recent timestamp than the cache.
Furthermore, since the libvirt-bin packages in Trusty and Xenial do not specify -W, that is, they do not write out a cache file, then using just "-r" to load a profile is consistent.
But if something outside the libvirt-bin package decides to generate apparmor caches, then we might have a problem.
One such scenario is an Ubuntu release upgrade from Trusty to Xenial. Here is what was observed during such an upgrade (here is a pastebin: http://pastebin.ubuntu.com/25222966/. It shows libvirt apparently restarting successfully at the end, but it didn't):
- new libvirt-bin is unpacked
- new apparmor is unpacked
- new apparmor is set up. This sets up new abstractions, and also generates an apparmor cache for all profiles. This is new behaviour: the trusty apparmor package does not generate caches. Crucial: at this time, the old libvirt-bin apparmor profiles are installed still.
- new libvirt-bin is setup. This installs the new apparmor profile for this version of libvirt-bin. Crucial: the profile timestamp is not $(now), but whatever timestamp the file has inside the debian package. Which is *older* than the cache generated above
- libvirt-bin reloads the apparmor profile with -r. apparmor picks the cached profile because its timestamp is newer than the profile
- libvirt-bin fails to start
The fix is to call apparmor_parser with -T and -W in postinst. That will always invalidate the apparmor cache and generate a new one based on the current contents of the profile file.
Another fix would be do use dh_apparmor to install the two profiles libvirt-bin uses. In fact, debian/rules even have those calls, but they are commented. We believe that doing that is a more invasive fix, and that just adding the -T and -W options to the existing apparmor_parser call has the same effect and is less invasive, being more in the spirit of an SRU to an LTS release.
In Yakkety and Zesty, dh_apparmor is used, but the call with just "-r" remains in postinst. That was only removed in artful, where we finally only rely on dh_apparmor for this.
[Test Case]
* install libvirt-bin
* check it's working. This command should work and return an empty list of virtual machines:
- sudo virsh list
* break the apparmor profile /etc/apparmor.d/usr.sbin.libvirtd by editing it and commenting the line "network inet stream," like this:
#network inet stream,
* generate a cache file for it:
- sudo apparmor_parser -r -T -W /etc/apparmor.d/usr.sbin.libvirtd
* purge libvirt-bin:
- sudo apt purge libvirt-bin
* install libvirt-bin back. It will fail to start the service:
- sudo apt install libvirt-bin
* verify that virsh list fails to connect to libvirt:
- sudo virsh list
* verify that service status shows the servicce being down:
- sudo service libvirt-bin status
If you repeat that last step with the fixed package, either after having encountered the error, or by running all steps again and skipping installing the broken package, the service will start correctly.
[Regression Potential]
In a nutshell, this fix does introduce a change in behaviour. But it's a change that other packages have already adopted (just grep for apparmor_parser in /var/lib/dpkg/info/*.postinst), and shouldn't be user visible. I also believe it's a less surprising behaviour than what we currently have.
I took the option of fixing one specific apparmor_parser call instead of introducing dh_apparmor in d/rules, which would have been a much bigger change, even though that's what we have in Yakkety and later.
Cache corruption seems to be dealt with by the tool correctly, although I haven't experimented with it. The documentation says:
"""
The default behaviour of the parser is to check if a cached version of a profile exists and if it does it attempt (sic) to load it into the kernel. If that load is rejected, then the parser will attempt to rebuild the cache file, and load again.
"""
[Other Info]
This bug affects Trusty too, but we haven't had a report about it yet. The only case so far is this release upgrade to Xenial.
Only administrators using Trusty who for one reason or another decide to use apparmor caches for libvirt *could* be affected, depending on the sequence of events. The test case shows such a possible scenario.
Since the change is simple, I included it for Trusty as well and will leave it up to the SRU team to decide if it's worth fixing there or not. I would perfectly understand if it is deemed not worthy for Trusty at this time.
I see apparmor DENIED messages:
[132257.005761] audit: type=1400 audit(150133353
And errors from libvirt trying to connect which match the refusals from apparmor:
Jul 29 09:05:14 hostname libvirtd[32055]: cannot connect to netlink socket with protocol 0: Permission denied
I tried reproducing this issue on a fresh xenial VM, but libvirt with apparmor worked just fine. I tried the original version from xenial, and the update. The profile is in enforcing mode, and all I get are STATUS messages in syslog.
Have you modified any apparmor file in /etc/apparmor.d by any chance?
One last thing to try is to upgrade from trusty to xenial, and then install libvirt-bin.