AppArmor profile causes QEMU/KVM - Not Connected

[Mike Salvatore]

[Impact]

 * libvirt in Focal in some cases e.g. with non local users
   needs to resolve those users. When trying to do so it fails
   due to apparmor isolation and breaks badly.

 * In later and former releases this issue isn't triggered,
   but it is unknown which (potentially complex) set of changes
   did that. A simple apparmor rule would help to allow libvirt
   to better function in environments with non known user IDs.

[Test Plan]

 * Following these steps in an unfixed release triggers the issue

sudo apt update; sudo apt dist-upgrade -y
sudo apt install -y sssd sssd-ldap slapd ldap-utils openssl expect lsb-release libvirt-clients libvirt-daemon-system ubuntu-dev-tools
pull-lp-source sssd
cd sssd-2.4.1
echo "*;*;*;Al0000-2400;libvirt" | sudo tee -a /etc/security/group.conf
head -n -5 debian/tests/ldap-user-group-ldap-auth > debian/tests/lp1890858-test
chmod +x debian/tests/lp1890858-test
sudo ./debian/tests/lp1890858-test
sudo systemctl restart libvirtd
# ensure it works in a normal login
virsh list
journalctl -u libvirtd
# try the sssd login
sudo login
# use testuser1 / testuser1secret to log in
virsh list

If affected this will not work reporting an error like:
$ virsh list
error: failed to connect to the hypervisor
error: End of file while reading data: Input/output error

And in dmesg/journal an apparmor denial like:

Jun 14 11:25:26 ldap.example.com audit[48330]: AVC apparmor="DENIED" operation="bind" profile="libvirtd" pid=48330 comm="rpc-worker" family="unix" sock_type="dgram" protocol=0 requested_mask="bind" denied_mask="bind" addr="@userdb-f283d575d74df972f9e10bd14d0befe3"

[Where problems could occur]

 * Allowing a little bit more to a daemon that already is rather powerful
   and open in regard to it's profile usually isn't changing behavior.
   If anything it would be considered a potential risk, but this rule
   should be ok to be added and ubuntu-security confirmed this.

[Other Info]

 * Comment 38 confirms that this should be ok - from the security Teams
   POV. https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1890858/comments/38

---

On some focal 20.04 systems, users are seeing "QEMU/KVM - Not Connected" when they attempt to use virt-manager to manage virtual machines. AppArmor denials like the following are seen in the logs:

sudo grep libvirt /var/log/syslog | grep -i apparmor | grep -i denied
Jun 28 14:53:27 koromicha kernel: [ 334.660844] audit: type=1400 audit(1593345207.778:951): apparmor="DENIED" operation="bind" profile="libvirtd" pid=12254 comm="libvirtd" family="unix" sock_type="dgram" protocol=0 requested_mask="bind" denied_mask="bind" addr="@userdb-6228daaaf66b14dfd14d93ef46d962c3"
Jun 28 14:54:19 koromicha kernel: [ 386.034970] audit: type=1400 audit(1593345259.145:952): apparmor="DENIED" operation="bind" profile="libvirtd" pid=14311 comm="libvirtd" family="unix" sock_type="dgram" protocol=0 requested_mask="bind" denied_mask="bind" addr="@userdb-c861507740da1fa0c3356ad3b78bffe9"
Jun 28 15:02:30 koromicha kernel: [ 877.339057] audit: type=1400 audit(1593345750.437:968): apparmor="DENIED" operation="bind" profile="libvirtd" pid=16175 comm="libvirtd" family="unix" sock_type="dgram" protocol=0 requested_mask="bind" denied_mask="bind" addr="@userdb-7d70643a9f8da0342f6359907817b664"

Users have reported that the "solution" is to disable the AppArmor profile. More details, screenshots, etc. can be found here: https://kifarunix.com/how-to-fix-qemu-kvm-not-connected-error-on-ubuntu-20-04/

[Christian Ehrhardt  (paelzer)]

Hi Mike,
I'm more than happy to write a patch for this, but "Not Connected when they attempt to use virt-manager" isn't enough as that works fine for me and several other.

I'll need to be able to reproduce or at least consciously explain why the denial is happening to be able to extend the rules what is allowed.

Therefore I wanted to ask if you can reproduce that yourself, if you happened to find what makes a difference e.g. connect to a remote system and/or other configurations on that system?

Note - we already have these rules for unix sockets which cover the known cases

 50 # for --p2p migrations
 51 unix (send, receive) type=stream addr=none peer=(label=unconfined addr=none),
...
 64 # For communication/control to qemu-bridge-helper
 65 unix (send, receive) type=stream addr=none peer=(label=libvirtd//qemu_bridge_helper),
 66 signal (send) set=("term") peer=libvirtd//qemu_bridge_helper,
 67
 68 # allow connect with openGraphicsFD, direction reversed in newer versions
 69 unix (send, receive) type=stream addr=none peer=(label=libvirt-[0-9a-f]*-[0-9a-f]*-[0-9a-f]*-[0-9a-f]*-[0-9a-f]*),
 70 # unconfined also required if guests run without security module
 71 unix (send, receive) type=stream addr=none peer=(label=unconfined),
...
126 unix (send, receive) type=stream addr=none peer=(label=libvirtd),

Therefore the question now is what is the use-case/setup detail we need to trigger this?

[Mike Salvatore (mikesalvatore)]

I haven't yet tried to reproduce it. It also works fine for me. My only guess at this point is there's some difference between a machine that has been upgrade to focal and a machine that has a freshly installed focal.

[Christian Ehrhardt  (paelzer)]

I have it working on upgraded (from 16.04 and from 18.04) as well as on fresh installs.
Really to be able to do anything about it we would need to find what makes this issue appear :-/

I'm unsure - could you try to get in touch with the people that wrote the blog and/or said "yes I'm affected" in any of those places?

[Floyd (zer0uid)]

I had the issue on an install of:

DISTRIB_DESCRIPTION="Ubuntu 20.04.1 LTS"

I followed the "fix" in the URL previously posted and symlinked the libvirt profile into the apparmor disable directory, and everything worked. In an attempt to recreate, I removed the symlink and rebooted, verified apparmor is still enforcing the libvirtd profile, but I no longer have the issue.

[Christian Ehrhardt  (paelzer)]

Thanks for your feedback Floyd - glad it is resolved for you by now and you can run with proper isolation again. But OTOH that means still no insight what/how triggers it :-/

[Tommy Nevtelen (dal)]

I'm having the same issue on an upgraded 18.04 -> 20.04.
This is bugging me immensely.

Ready to help out in any way. I have also confirmed that if I disable apparmor it works. But that is not a solution.

Here is an strace showing what it's up to starting at the socket access.

$ VIRSH_DEBUG=0 strace virsh list
...
access("/var/run/libvirt/virtqemud-sock", F_OK) = -1 ENOENT (No such file or directory)
access("/var/run/libvirt/libvirt-sock", F_OK) = 0
socket(AF_UNIX, SOCK_STREAM, 0) = 6
connect(6, {sa_family=AF_UNIX, sun_path="/var/run/libvirt/libvirt-sock"}, 110) = 0
getsockname(6, {sa_family=AF_UNIX}, [128->2]) = 0
futex(0x7f22454da060, FUTEX_WAKE_PRIVATE, 2147483647) = 0
fcntl(6, F_GETFD) = 0
fcntl(6, F_SETFD, FD_CLOEXEC) = 0
fcntl(6, F_GETFL) = 0x2 (flags O_RDWR)
fcntl(6, F_SETFL, O_RDWR|O_NONBLOCK) = 0
futex(0x7f22454da180, FUTEX_WAKE_PRIVATE, 2147483647) = 0
pipe2([7, 8], O_CLOEXEC) = 0
write(5, "\0", 1) = 1
futex(0x7f22454d9320, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x7f22454da078, FUTEX_WAKE_PRIVATE, 2147483647) = 0
futex(0x7f22454da150, FUTEX_WAKE_PRIVATE, 2147483647) = 0
write(5, "\0", 1) = 1
futex(0x7f22454d9320, FUTEX_WAKE_PRIVATE, 1) = 1
rt_sigprocmask(SIG_BLOCK, [PIPE CHLD WINCH], [], 8) = 0
poll([{fd=6, events=POLLOUT}, {fd=7, events=POLLIN}], 2, -1) = 1 ([{fd=6, revents=POLLOUT}])
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
write(6, "\0\0\0\34 \0\200\206\0\0\0\1\0\0\0B\0\0\0\0\0\0\0\0\0\0\0\0", 28) = 28
rt_sigprocmask(SIG_BLOCK, [PIPE CHLD WINCH], [], 8) = 0
poll([{fd=6, events=POLLIN}, {fd=7, events=POLLIN}], 2, -1) = 1 ([{fd=6, revents=POLLIN|POLLHUP}])
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
read(6, "", 4) = 0
openat(AT_FDCWD, "/usr/share/locale/locale.alias", O_RDONLY|O_CLOEXEC) = 9
fstat(9, {st_mode=S_IFREG|0644, st_size=2996, ...}) = 0
read(9, "# Locale name alias data base.\n#"..., 4096) = 2996
read(9, "", 4096) = 0
close(9) = 0
openat(AT_FDCWD, "/usr/share/locale/en_US/LC_MESSAGES/libvirt.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en/LC_MESSAGES/libvirt.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale-langpack/en_US/LC_MESSAGES/libvirt.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale-langpack/en/LC_MESSAGES/libvirt.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en_US/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale-langpack/en_US/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale-langpack/en/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/charset.alias", O_RDONLY) = -1 ENOENT (No such file or directory)
write(5, "\0", 1) = 1
futex(0x7f2...

[Tommy Nevtelen (dal)]

Okay so I did aa-complain on the libvirt profile and sudo aa-logprof when running "virsh list".

It generated a new profile which I'll be attaching here

When running with that enabled I think it works.. I'm able to do virsh list as a normal user. Unless I'm doing something wrong and misunderstanding..
Have not compared the original to this yet because the diff was long. But it would be interesting to hear if it works for others.

[Christian Ehrhardt  (paelzer)]

Hi Tommy, thank you already for your effort.

I have realized that my answer was rather long, so let me split it into:
- what does happen
- why does it happen
- how to fix

in the following three answers.
Keeping the replies to those separate as well might help to get things right without confusing everyone :-)

[Christian Ehrhardt  (paelzer)]

# Clarify what does happen #

Tracing and Profiling virsh should (tm) not change anything.
The Denial that was reported looks like:
  apparmor="DENIED" operation="bind" profile="libvirtd" comm="libvirtd" ...

So virsh might connect to libvirtd daemon, but then the daemon does something that is denied.
If the issue is reproducible in your setup, then a good next step would be.

Assumption: let's say "virsh list" reproducibly causes the apparmor denials in dmesg
1. attach strace to the libvirt daemon process (check `systemctl status libvirtd` for the pid)
2. run "virsh list"

Report:
- the apparmor denial that was caused
- the strace output of the daemon

I'd hope we can find the call of libvirtd in that strace that causes the denial.

[Christian Ehrhardt  (paelzer)]

# Clarify why it happens on your setup #

Binding unix sockets is what the denial is about and there isn't too much in libvirt doing that.

In Focal all these should be systemd socket files that are taken over by the service once started. Maybe in the setup of these is a difference.

If you could attach the full output of the following (if it changes then prior and post triggering the issue).
$ systemctl status $(basename -a $(dpkg -L libvirt-daemon-system | grep -e .socket -e .service | xargs) | xargs)

I'm stabbing in the dark here still, but we need to start somewhere.

[Christian Ehrhardt  (paelzer)]

# Clarify how to fix it #

Your rules out of logprof are interesting.

I have compared them with the one in the package in regard to unix/dgram rules which is what the denial is about. The only entry your example has on top is the following:

  network unix dgram,

Could you please try to:
1. revert /etc/apparmor.d/usr.sbin.libvirtd to the content delivered by the package in 20.04
2. retry and verify the issue triggers
3. restart libvirtd (systemctl restart libvirtd)
4. retry and verify the issue triggers still
5. add this line above to /etc/apparmor.d/usr.sbin.libvirtd where the other network rules are
6. sudo apparmor_parser -r /etc/apparmor.d/usr.sbin.libvirtd
7. restart libvirtd (systemctl restart libvirtd)
8. retry and verify the issue triggers still (or is it fixed now?)

#3 ensures that restarting the service without a change does not by accident resolve the issue
#6+#7 ensures that the apparmor profile with the change is reloaded and the rule is good

Please report how that test worked for you.
Hopefully that is the one entry we need, otherwise we need to continue looking for differences,

[Bertrand Rétif (bretif)]

I have the same issue.
And the modification of /etc/apparmor.d/usr.sbin.libvirtd as described in previous message fix the issue.
Thanks
Bertrand

[Christian Ehrhardt  (paelzer)]

Hi Bertrand and thanks for joining in,
with your report that the isolated "network unix dgram," rule helps is great to confirm the "how to fix".
But as outlined before - since this doesn't reproduce for everyone - we'd need to find what the subset of people that hit this do to trigger it. Only then we can really add the rule.

I was (once more) trying to trigger this, but still it just doesn't happen.
# clean new Focal system after install from a cloudimg
$ sudo apt install libvirt-clients libvirt-daemon-system
$ virsh list --all
# works, no denial

I compared the strace that tommy provided in comment #6 with one on this clean system:
Tommy:
socket(AF_UNIX, SOCK_STREAM, 0) = 6
connect(6, {sa_family=AF_UNIX, sun_path="/var/run/libvirt/libvirt-sock"}, 110) = 0
getsockname(6, {sa_family=AF_UNIX}, [128->2]) = 0
write(6, "\0\0\0\34 \0\200\206\0\0\0\1\0\0\0B\0\0\0\0\0\0\0\0\0\0\0\0", 28) = 28
poll([{fd=6, events=POLLIN}, {fd=7, events=POLLIN}], 2, -1) = 1 ([{fd=6, revents=POLLIN|POLLHUP}])
read(6, "", 4) = 0

Comparison:
socket(AF_UNIX, SOCK_STREAM, 0) = 5
connect(5, {sa_family=AF_UNIX, sun_path="/var/run/libvirt/libvirt-sock"}, 110) = 0
getsockname(5, {sa_family=AF_UNIX}, [128->2]) = 0
write(5, "\0\0\0\34 \0\200\206\0\0\0\1\0\0\0B\0\0\0\0\0\0\0\0\0\0\0\0", 28) = 28
poll([{fd=5, events=POLLIN}, {fd=6, events=POLLIN}], 2, -1) = 1 ([{fd=5, revents=POLLIN}])
read(5, "\0\0\0$", 4) = 4

Here the "good case" got something back from the socket, and the "bad one" didn't.
The denial is on libvirt daemons side, so virsh (or virt-manager) tried to talk to it, but communication was blocked it seems. The strace is on the client and only sees that the receiving end doesn't get anything. From there the "bad case" is on an exit path (but that is virsh code and not libvirtd where the denial happened).

None of this yet explains why it happens to some users, but not others.
I also tried a B->F upgrade as some reported to see it then to no repro-luck :-/

All that still is just dancing around the symptom, as I outlined in comment #9 and #10 we'd need to have status of the services/sockets as well as a strace of the daemon (not the client) in the bad case to get towards some next steps.

In addition anything from your config might help, e.g. a tarball of /etc/libvirt/ maybe that will allow me to trigger and debug it. If you (or others) need help what exactly to debug, which commands to use please get in touch here and I can help.

For the sake of recognizing a pattern - are all of the affected seeing this on a 18.04->20.04 upgrade or are new installs affected as well? Also are all affected systems Desktop installs or are there Server/Cloud-images as well?

@Bertrand - Tommy didn't yet get to answer to these questions. I think you can now switch back to the bad state by removing that apparmor line that you added and restarting libvirtd.
If that is true could you help to answer these questions?

[Bertrand Rétif (bretif)]

Hi Christian, Sorry for late reply.

I still have the issue.
In fact the issue is back every time I reboot.
I fix it by restart apparmor:

$ virsh list --all
error: failed to connect to the hypervisor
error: End of file while reading data: Erreur d'entrée/sortie
$ sudo apparmor_parser -r /etc/apparmor.d/usr.sbin.libvirtd
$ virsh list --all

 Id Name State
---------------------------
 1 minikube running

As you see I do not change any files, just restart apparmor.

If I remove "network unix dgram," from /etc/apparmor.d/usr.sbin.libvirtd I still have the issue.

I attach the tar of my /etc/libvirt

[Bertrand Rétif (bretif)]

Tar of my /etc/libvirt

[Kim Covil (vendor-ubuntu)]

I am seeing this too on a machine upgraded from 18.04 to 20.04:

# Clarify what does happen #

virsh list as non-root user produces the following audit apparmor DENIED in dmesg:
[452295.854406] audit: type=1400 audit(1609252406.821:273932): apparmor="DENIED" operation="bind" profile="libvirtd" pid=2117888 comm="libvirtd" family="unix" sock_type="dgram" protocol=0 requested_mask="bind" denied_mask="bind" addr="@userdb-200402f6469c0bb726e901462b8cedb4"

strace of libvirtd process while running virsh list attached

[Kim Covil (vendor-ubuntu)]

# Clarify why it happens on your setup #

Attached output of running:
systemctl status $(basename -a $(dpkg -L libvirt-daemon-system | grep -e .socket -e .service | xargs) | xargs)

[Kim Covil (vendor-ubuntu)]

# Clarify how to fix it #

Adding the network unix dgram, line works here:

1. revert /etc/apparmor.d/usr.sbin.libvirtd to the content delivered by the package in 20.04
   $ dpkg-query -W -f '${Conffiles}\n' libvirt-daemon-system | awk -vOFS=" " '/apparmor/{print $2,$1}' | LANG=C sudo md5sum -c 2>/dev/null
   /etc/apparmor.d/abstractions/libvirt-lxc: OK
   /etc/apparmor.d/abstractions/libvirt-qemu: OK
   /etc/apparmor.d/libvirt/TEMPLATE.lxc: OK
   /etc/apparmor.d/libvirt/TEMPLATE.qemu: OK
   /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper: OK
   /etc/apparmor.d/usr.sbin.libvirtd: OK

2. retry and verify the issue triggers
   $ virsh list
   error: failed to connect to the hypervisor
   error: End of file while reading data: Input/output error

3. restart libvirtd (systemctl restart libvirtd)
   $ sudo systemctl restart libvirtd

4. retry and verify the issue triggers still
   $ virsh list
   error: failed to connect to the hypervisor
   error: End of file while reading data: Input/output error

5. add this line above to /etc/apparmor.d/usr.sbin.libvirtd where the other network rules are
   $ cp /etc/apparmor.d/usr.sbin.libvirtd /tmp/usr.sbin.libvirtd.bak
   $ sudo sed -i -e '/^ network inet stream/i \ \ network unix dgram,' /etc/apparmor.d/usr.sbin.libvirtd
   $ diff -p /tmp/usr.sbin.libvirtd.bak /etc/apparmor.d/usr.sbin.libvirtd
   *** /tmp/usr.sbin.libvirtd.bak 2020-12-29 14:46:26.716346230 +0000
   --- /etc/apparmor.d/usr.sbin.libvirtd 2020-12-29 14:48:58.816884722 +0000
   *************** profile libvirtd /usr/sbin/libvirtd flag
   *** 39,44 ****
   --- 39,45 ----
       mount options=(rw, move) /{,var/}run/libvirt/qemu/*.dev/ -> /dev/,
       mount options=(rw, move) /{,var/}run/libvirt/qemu/*{,/} -> /dev/**,

   + network unix dgram,
       network inet stream,
       network inet dgram,
       network inet6 stream,

6. sudo apparmor_parser -r /etc/apparmor.d/usr.sbin.libvirtd
   $ sudo apparmor_parser -r /etc/apparmor.d/usr.sbin.libvirtd

7. restart libvirtd (systemctl restart libvirtd)
   $ sudo systemctl restart libvirtd

8. retry and verify the issue triggers still (or is it fixed now?)
   $ virsh list
    Id Name State
   --------------------

[Christian Ehrhardt  (paelzer)]

Thanks for your Feedback Kim!

This confirms (as we've assumed before) that it is a valid case and the discussed rule will make it work for you. But far we didn't know how others could trigger it. After all we seem to have a few affected but the majority isn't.
But to generally allow "network unix dgram" we'll have to make a case what use case it is needed for.

So the question is (again), what is different on your libvirt (and the rest of) installation that makes you trigger this?
I've tried this - with/without upgrades and several config - it never triggered over here.
So I beg your pardon and want to ask:
- what else than "apt install libvirt-daemon-system" + "virsh ..." commands to spawn your guest did you do?
- Is it affecting all guest, if not what makes the affected ones special?
- Can you recreate the same on a new fresh system - if so what is the sequence of commands to get there?

[Robert Euhus (euhus-liste1)]

Hi Christian,

we noticed this problem on several of our machines, so I would like to give this bug report another try. ;)

I have a (relatively) clean install of Ubuntu 20.04 (no upgrade), which is joined to a Windows AD-domain via sssd, but currently used off site with cached credentials.

My domain user experiences the problem, but any local user doesn't.

I have checked with three different local users: the admin created during install, a new ordinary user, and another new one with the nonstandard HOME directory /home/TEST/tester1. All users were members of the libvirt group, the domain user is also in adm sudo users lpadmin.

The 'network unix dgram' entry "fixes" the problem for my domain user.

The problem was also reproduceable on a fresh install after domain join with a domain user.

Here is what I did:

1) normal startup with default profile

2) get sockets status:
 - systemctl status $(basename -a $(dpkg -L libvirt-daemon-system | grep -e .socket -e .service | xargs) | xargs) > systemctl_status_before

3) run strace in the pid of libvirtd:
 - systemctl status libvirtd
 - strace -p 1246 2>&1 | tee -a strace_local_user_success

4) try to connect al local user -> success:
 - virsh list
 Id Name State
--------------------

5) get sockets status:
 - systemctl status $(basename -a $(dpkg -L libvirt-daemon-system | grep -e .socket -e .service | xargs) | xargs) > systemctl_status_after_success

6) stop strace and restart for next try:
 - strace -p 1246 2>&1 | tee -a strace_domain_user_fail

7) try to connect as domain user -> failure:
 - virsh list
error: failed to connect to the hypervisor
error: End of file while reading data: Eingabe-/Ausgabefehler

8) get sockets status:
 - systemctl status $(basename -a $(dpkg -L libvirt-daemon-system | grep -e .socket -e .service | xargs) | xargs) > systemctl_status_after_failure

9) add 'network unix dgram,' to /etc/apparmor.d/usr.sbin.libvirtd and apply changes:
 - vim /etc/apparmor.d/usr.sbin.libvirtd
 - diff /etc/apparmor.d/usr.sbin.libvirtd{,.orig}
42d4
< network unix dgram,

10) run strace on the new process:
 - systemctl status libvirtd
 - strace -p 11051 2>&1 | tee -a strace_domain_user_network_unix_dgram_success

11) try again as domain user -> success:
 - virsh list
 Id Name State
--------------------

12) get surrounding area from syslog:
 - grep 'Jun 8 14:5\(7\|8\|9\)' /var/log/syslog > syslog-error

I will upload all of the mentioned log files.

If You need enything else, please let me know.

I am really curious about the reason and the real fix for this issue.

[Christian Ehrhardt  (paelzer)]

Hi Robert,
thanks for chiming in - I have to admit that I'm dedicated to other tasks this week so I beg your pardon for taking a while to digest all that good data that you added.

[Robert Euhus (euhus-liste1)]

Hi Christian,
thanks for Your quick reply! This has been lingering for so long and a workaround is available, so no need to hurry. I am just very curious about the real cause for this issue and would prefer a proper solution. :)

For now we are using this little drop in /etc/apparmor.d/local/usr.sbin.libvirtd:
network unix dgram,

followed by 'apparmor_parser -r -W -T /etc/apparmor.d/usr.sbin.libvirtd'

All wrapped up in a nice little Deb-package:
https://ubuntu.repo.uni-hannover.de/ubuntu/pool/pub/l/luh-sssd-libvirtd-fix/luh-sssd-libvirtd-fix_0.4_all.deb

Just in case somebody else needs this workaround.

[Robert Euhus (euhus-liste1)]

Hi Christian,

thanks for Your quick reply! This has been lingering for so long and a workaround is available, so no need to hurry. I am just very curious about the real cause for this issue and would prefer a proper solution. :)

For now we are using this little drop in /etc/apparmor.d/local/usr.sbin.libvirtd:
network unix dgram,

followed by 'apparmor_parser -r -W -T /etc/apparmor.d/usr.sbin.libvirtd'.

All wrapped up in a little Deb-package:
https://ubuntu.repo.uni-hannover.de/ubuntu/pool/pub/l/luh-sssd-libvirtd-fix/

Just in case somebody else needs this workaround.

[Christian Ehrhardt  (paelzer)]

Yeah that is the mentioned workaround that we have identified before wrapped up nicely for consumption. Still - like many others - I'd want to understand the problem :-)

And BTW I wanted to mention a lot of thanks to everyone participating so far!
It makes me feel proud of our user community to work together constructively until we find it.

And another BTW we need to be careful as I think we might have at least one or
two other cases here on the bug that might look like the same symptom but
have different reasons. So if/once we apply that change some (with other root
causes) might say "still not working for me" but those we'd then need to fork
out into other bugs to track down their individual root causes.

I have no sssd setup around but will ask someone for help on that.
Thanks for this new opportunity to this.

And even if the sssd approach fails, I think despite still not being able to recreate it there
are by now enough people hit and fixed by adding the dgram rule that we have together
come up. The libvirtd profile itself is after all meant to be very lenient (not
the guests, but the host service). So by now it might be fair to at least propose that even IF I can't reproduce still.

[Christian Ehrhardt  (paelzer)]

The puzzling bit here is that libvirtd is the service that is ran as root by systemd.
I'm not seeing why the kind of user that calls virt-manager/virsh would make any difference.
As the problem so far seemed to be on the service side.

And it still is for your log
Jun 8 14:59:20 notebook kernel: [ 467.626033] audit: type=1400 audit(1623157160.575:108): apparmor="DENIED" operation="bind" profile="libvirtd" pid=1246 comm="libvirtd" family="unix" sock_type="dgram" protocol=0 requested_mask="bind" denied_mask="bind" addr="@userdb-faa385c89baf9fdc10b8cd36f899a797"

That still is "libvirtd" so why would the user calling virsh matter for the service?

---

I tried to connect to libvirt in various cases
1. use Bionic -> works
2. upgrade to Focal -> works
3. have sssd to log in a domain user ...
   I had no AD config, so I used a ldap based sssd auth that the autopkgtests of sssd use.
   That allowed me to check on the usual steps needed here.

#3.1 log in a trivial user - permission error lacking "libvirtd" group

At first I got:
testuser1@ldap:~$ virsh list
error: failed to connect to the hypervisor
error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Permission denied

But that is expected as my user logged in via sssd has no group membership for "libvirt".
There is no apparmor denial as usually reported by others on this bug.

You (Robert) had not the "permission denied", but the apparmor related I/O error.

#3.2 libvirt checks on local users

After fixing my /etc/security/group.conf to apply "libvirtd" group
testuser1@ldap:~$ id
uid=10001(testuser1) gid=10001(testuser1) groups=10001(testuser1),117(libvirt),10100(ldapusers)

$ virsh list
error: failed to connect to the hypervisor
error: Failed to find user record for uid '10001'

Still not the reported apparmor issue, but due to the user not being revere mappable for libvirtd. That is because the daemon has not yet learned of the changed user setup.
Restarting libvirt is enough to have it pick that up.

$ sudo systemctl restart libvirtd

#3.3 Finally at the issue you all see

testuser1@ldap:~$ virsh list
error: failed to connect to the hypervisor
error: End of file while reading data: Input/output error

Jun 14 09:29:07 ldap.example.com kernel: audit: type=1400 audit(1623662947.357:174): apparmor="DENIED" operation="bind" profile="/usr/sbin/libvirtd" pid=3585 comm="libvirtd" family="unix" sock_type="dgram" protocol=0 requested_mask="bind" denied_mask="bind" addr="@userdb-5402976fa0b85f7da530b09e72db0f01"

Ok, let me try to break that down to reproducible minimal steps ...

[Christian Ehrhardt  (paelzer)]

Maybe the setup is a bit broken as I was rather rude and direct setting this up.
I've seen two things:

1. I was seeing that the PID on this denial was always changing
2. I found that this crashed libvirt like

Jun 14 09:36:44 ldap.example.com libvirtd[4585]: Illegal status in __nss_next.
Jun 14 09:36:44 ldap.example.com systemd[1]: libvirtd.service: Main process exited, code=killed, status=6/ABRT
Jun 14 09:36:44 ldap.example.com systemd[1]: libvirtd.service: Failed with result 'signal'.
Jun 14 09:36:45 ldap.example.com systemd[1]: libvirtd.service: Scheduled restart job, restart counter is at 9.

I have not seen this in any of the reports/logs added here so far, so I hope I find a setup that isn't half-broken but still reproduces the issue.

Also as one would expect in the above case the "network unix dgram," rule doesn't help.
Well consider this just a broken test for now.

[Christian Ehrhardt  (paelzer)]

Testcase:

# Setup Ldap/SSSD based on the tests
sudo apt install sssd sssd-ldap slapd ldap-utils openssl expect lsb-release virt-manager ubuntu-dev-tools
echo "*;*;*;Al0000-2400;libvirt" | sudo tee -a /etc/security/group.conf
sudo pull-lp-source sssd
cd sssd-2.4.1
head -n -5 debian/tests/ldap-user-group-ldap-auth > debian/tests/lp1890858-test
chmod +x debian/tests/lp1890858-test
sudo ./debian/tests/lp1890858-test
sudo systemctl restart libvirtd
sudo login
# use testuser1 / testuser1secret to log in
virth list

I tried the above in:
#1 Focal
- crash as reported in the former comment
- this crash is around the apparmor DENIED message, so it is somewhat related
#2 Impish
- works just fine (neither crash nor apparmor denial)

In both cases the dgram rules for libvirtd are the same.
$ grep -Hrn dgram /etc/apparmor.d/usr.sbin.libvirtd
/etc/apparmor.d/usr.sbin.libvirtd:43: network inet dgram,
/etc/apparmor.d/usr.sbin.libvirtd:45: network inet6 dgram,
/etc/apparmor.d/usr.sbin.libvirtd:47: network packet dgram,

Now that I have my clean steps to reproduce I'll throw away and recreate both environments to be sure there is nothing left from my try to get these steps created.

[Christian Ehrhardt  (paelzer)]

Update Repro steps:
 - add a pre-upgrade
 - virt-manager isn't needed, install a few less deps
 - use non interactive install

# Repro
sudo apt update; sudo apt dist-upgrade -y
sudo apt install -y sssd sssd-ldap slapd ldap-utils openssl expect lsb-release libvirt-clients libvirt-daemon-system ubuntu-dev-tools
pull-lp-source sssd
cd sssd-2.4.1
echo "*;*;*;Al0000-2400;libvirt" | sudo tee -a /etc/security/group.conf
head -n -5 debian/tests/ldap-user-group-ldap-auth > debian/tests/lp1890858-test
chmod +x debian/tests/lp1890858-test
sudo ./debian/tests/lp1890858-test
sudo systemctl restart libvirtd
# ensure it works in a normal login
virsh list
journalctl -u libvirtd
# try the sssd login
sudo login
# use testuser1 / testuser1secret to log in
virsh list

[Christian Ehrhardt  (paelzer)]

Tried in some setups:
- Focal
- Groovy
- Hirsute
- Focal + the virt stack of Hirsute (via https://launchpad.net/~canonical-server/+archive/ubuntu/server-backports)

Results:
- Focal - AA-Denial+Crash
- Groovy - Ok
- Hirsute - Ok
- Focal+Vnew - AA-Denial+Crash

So the results in my test-set are somewhat indicating a fix more in the rest of the system or maybe sssd - at least for the crash that I'm seeing :-/

The crazy part comes now and somewhat matches the mixed feedback I was getting from users on this bug. For example I was able to avoid the issue by adding the rule:
  echo "network unix dgram," | sudo tee -a /etc/apparmor.d/local/usr.sbin.libvirtd
  sudo apparmor_parser -r -W -T /etc/apparmor.d/usr.sbin.libvirtd
  sudo systemctl restart libvirtd # not strictly required, but to be sure
But then removing the issue does not bring it back (Neither the denial nor the crash).

I wondered if libvirt might cache something and only reach out (denied and broken) to nss/sssd if not present. In that case the apparmor rule would be needed to allow that.

Trying to clear anything old to get back into the error state:
  echo "" | sudo tee /etc/apparmor.d/local/usr.sbin.libvirtd
  sudo apparmor_parser -r -W -T /etc/apparmor.d/usr.sbin.libvirtd
  sudo apt remove --purge libvirt-daemon
  sudo apt install libvirt-daemon-system

And that worked to get it back to broken state.
I don't see/know yet what fixed it in >=Groovy, but it isn't the same apparmor rule that we use here.
Therefore it makes no sense applying the rule forward upstream or in new releases, but it might be a great avoidance for anyone affected to apply it in Focal's libvirt.

We now have:
- a repro case
- a clear scope (just Focal, fixed later and not a problem before)
- the libvirt profile being lenient before
- libvirt is meant to be able to bind unix sockets

I'll ask security if that is concerned to be a problem.

[Christian Ehrhardt  (paelzer)]

Theory/Summary:
- the sssd login makes libvirt need to resolve a uid/gid as it isn't known locally
- once that worked it isn't re-done later on
- To resolve it it needs to bind a unix socket
  Jun 14 11:25:24 ldap.example.com kernel: audit: type=1400 audit(1623669923.999:144): apparmor="DENIED" operation="bind" profile="libvirtd" pid=47723 comm="libvirtd" family="unix" sock_type= "dgram" protocol=0 requested_mask="bind" denied_mask="bind" addr="@userdb-e87bdc7da8b5fabb4fbc28e32c4d783d"
- in Focal (not later and not before) that triggers an apparmor denial followed sometimes (depending on the setup) by a crash
- Allowing "network unix dgram," for libvirtd avoids the issue
- Many users have hit this various ways, but most likely all caused libvirt to look for UID/GID resolution that then was denied.

@Security - the question is if it would be ok to add the not further restricted "network unix dgram," rule to /etc/apparmor.d/local/usr.sbin.libvirtd in Focal.

As Jamie usually said it is very lenient anyway and therefore such things should not be a problem.
But better err on the side of caution, so I wanted to hear your input please.

P.S. If you happen to realize "oh yeah sssd login contexts in focal kernels are special because foo" let me know, but I fail to see why

[Marc Deslauriers (mdeslaur)]

While I'm not really up-to-speed on how libvirt is confined, I can't really think of any alternative to handling this properly than adding the new rule. +1 from me.

[Christian Ehrhardt  (paelzer)]

I prepared the SRU Template in the description of this bug and furthermore
a PPA: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4582

@Robert - While it is just a single apparmor line added it would still be great if you could give this PPA an A/B comparison before we kick off the real SRU.

[Seth Arnold (seth-arnold)]

Hello Christian, a few thoughts:

- it sounds like we're going to modify the ../local/.. version of the file in a package update. I think this is a mistake. If we're going to modify policy, we should modify the 'real' policy and leave the ../local/.. version for the system administrator.

- it sounds like the rule we're adding a very broad rule that allows far more access than needed for the problem at hand. Reducing this to just allowing necessary privileges on the '@userdb-*' sockets would be much tighter.

- based on the fact that these interfaces look like they're part of systemd, it probably solves a lot more problems to add these rules to abstractions/nameservice near the existing systemd rules, or perhaps move those and add this to a new abstraction, and add it to the abstractions/nameservice file. That would address this same service when used via other applications, not just libvirt.

Funny thing though, I only see this userdb- string in our packaging: userdb_thread_sockaddr in:
systemd_245.4-4ubuntu3.4/src/shared/userdb.c
systemd_245.4-4ubuntu3.3/src/shared/userdb.c
systemd_245.2-1ubuntu1/src/shared/userdb.c
systemd_245.5-3ubuntu1/src/shared/userdb.c
systemd_245.4-4ubuntu3/src/shared/userdb.c
systemd_245.4-4ubuntu3.2/src/shared/userdb.c
systemd_245.4-2ubuntu1/src/shared/userdb.c
systemd_245.4-4ubuntu3.5/src/shared/userdb.c
systemd_245.5-2ubuntu2/src/shared/userdb.c
systemd_245.4-4ubuntu1/src/shared/userdb.c
systemd_245.4-4ubuntu3.6/src/shared/userdb.c
systemd_245.4-4ubuntu3.1/src/shared/userdb.c
systemd_245.6-1ubuntu1/src/shared/userdb.c

I can't find this code via Debian Code Search, eg:

http://codesearch.debian.net/search?q=userdb-%25016
http://codesearch.debian.net/search?q=ret_salen%20%3D%20offsetof
http://codesearch.debian.net/search?q=NSS%20emulation

I've also tried looking in userdb.c manually:
https://sources.debian.org/src/systemd/247.3-3/src/shared/userdb.c/

'sock' only shows up in a header file, "socket-util.h"

It sure looks like it still exists but it's much harder to find in newer packages.

Thanks

[Seth Arnold (seth-arnold)]

Ah, sorry, I missed the 'merge' link; you're not using the ../local/.. file :) Thanks

[Christian Ehrhardt  (paelzer)]

:-) NP Seth - Yes the "local" was only for manual workarounds in this bug.
And the proposed fix is in the right place for the package.

The abstractions, or generally other places for that rule are interesting.
Because as I stated above while I now finally can recreate it in Focal it is gone in later versions. I was unable to find a clear sssd/libvirt change that fixed this - but chances are one of those abstractions already got a rule that now allows it.
  #include <abstractions/base>
  #include <abstractions/dbus>
Neither of them leads to such a rule in >=Groovy.

It really is systemd that changed.
The code was indeed present in 245 (Focal) but not later.
That is the code on v245 (Focal):
https://github.com/systemd/systemd/blob/ea500ac513cf51bcb79a5666f1519499d029428f/src/shared/userdb.c#L1237
The whole functionality was added in v245 via
https://github.com/systemd/systemd/commit/ec8e4a0ef12ff2fd393e58c335602d605d94f846
and removed in v246 via
https://github.com/systemd/systemd/commit/037b0a47b0d7df09d720dda6703135117e7e0472

That explains why we only see this in Focal - it is the only version containing that mechanism.
And I think it is fair to say that the switch of the underlying tech in systemd isn't backportable for an SRU (compared to the rule we propose).

It now also makes sense why e.g. the non local sssd user trigger this. When calling the service through the socket of libvirt it will try to check who has called and that is exactly when the nss services will all be probed. With system 245 this also implies this generated socket to be bound.

I'll have a look at further restricting the rule ...

[Christian Ehrhardt  (paelzer)]

Limiting the rules is always good, so I went back to the repro setup.
I was trying various limitations for userdb- like:

@Seth - Thanks for the hint to check the addr element, now that I've found the root cause in systemd we can be sure what the pattern will look like.

I have found that the following rule works just as much and is much more fine grained:

  unix (bind) type=dgram addr=@userdb-*,

So whatever way we go, this should be the rule to use.

[Christian Ehrhardt  (paelzer)]

We could consider adding that to libvirt's profile directly.
Or to add it to `abstractions/nameservice` as it is nss means that trigger this.
Is that really the right place?

In the latter case I'd still need a libvirt SRU as it didn't need and include abstractions/nameservice up to now.

@Seth what would you think of this proposal:
A)
1. "unix (bind) type=dgram addr=@userdb-*," to /etc/apparmor.d/abstractions/nameservice (package: apparmor)
2. add abstractions/nameservice to /etc/apparmor.d/usr.sbin.libvirtd (package: libvirt)

or

B)
1. "unix (bind) type=dgram addr=@userdb-*," to /etc/apparmor.d/usr.sbin.libvirtd (package: libvirt)

[Seth Arnold (seth-arnold)]

Wow, thanks so much for the archaeology, Christian. That removal commit explains so much.

I prefer (b) for this case:

- fewer moving pieces in the change is more likely to be reliable
- adding abstractions/nameservice to a profile that didn't already have it is a fairly large increase in permissions
- if the code in question is going to be gone in ten years, 'planning for the future' is much less important
- apparently this functionality isn't widely used in Focal, otherwise we would have seen more than this.

It's a bit strange to come to a different conclusion the next day; I could probably be talked into (a) if someone else feels strongly about it. But the simpler approach is my preference now.

Thanks

[Robert Euhus (euhus-liste1)]

Thanks to both of You for figuring out the core of the problem.

While I understand the reasoning for preferring solution (b) I would like to examine a strange somehow similar behaviour in cupsd with sssd. (Which - shame on me - I haven't reported yet):

We had similar access problems with domain users trying to connect to the CUPS web interface (localhost:631) which was denied by apparmor. I added an exception to allow this, which worked in Bionic, but now in Focal cupsd crashes while trying to enumerate groups, which is disabled in SSSD for our Domain. I hadn't bothered reporting this, since nobody actually uses the cups web interface on a desktop and it's going away in the future anyway (CUPS 2.4 or 3.0 if I remember correctly).

I will now examine this, probably open a different bug and report back here shortly.

@Christian: Sorry for not replying earlier (I had typed in my reply, but got distracted, so forgot to actually post it). I had tested the package from the PPA and it worked like expected. But that doesn't really matter any more, since we have a better solution now.

[Christian Ehrhardt  (paelzer)]

Thanks Seth and Robert,
I'll follows Seth advise and for now propose (B) being the fix for libvirt.
But I'll add a task for apparmor (which carries abstraction/namespace) as that still might be an avenue worthwhile to follow independent to this libvirt fix.
If more people speak up or we find more cases affected then adding it there eventually seems good.

[Christian Ehrhardt  (paelzer)]

FYI MP https://code.launchpad.net/~paelzer/ubuntu/+source/libvirt/+git/libvirt/+merge/404124 updated with the more restricted rule.
I also updated the PPA with the new rule.

[Christian Ehrhardt  (paelzer)]

FYI - Uploaded to Focal-unapproved

[Robie Basak (racb)]

Hello Mike, or anyone else affected,

Accepted libvirt into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libvirt/6.0.0-0ubuntu8.10 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Robert Euhus (euhus-liste1)]

I have successfully tested the libvirt/6.0.0-0ubuntu8.10 packages from focal-proposed as requested.

The following packages where upgraded to 6.0.0-0ubuntu8.10:
libvirt-clients libvirt-daemon libvirt-daemon-driver-qemu libvirt-daemon-driver-storage-rbd libvirt-daemon-system libvirt-daemon-system-systemd libvirt0

Afterwards I was able to connect to libvirtd with my domain user as expected.

FYI: I have opened a separate bug report for cups, since the crashing issue is fixed by this as well:
https://bugs.launchpad.net/ubuntu/+source/cups/+bug/1932537

Thanks to everyone involved for the quick resolution!

[Christian Ehrhardt  (paelzer)]

Thanks for the quick test, works for me as well.
The global verification-done tag might be needed as well for the tools to work, adding that back.

[Christian Ehrhardt  (paelzer)]

The apparmor section of this is now properly covered in bug 1932537, therefore I'm dropping the apparmor task from this one.

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for libvirt has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package libvirt - 6.0.0-0ubuntu8.10

---------------
libvirt (6.0.0-0ubuntu8.10) focal; urgency=medium

  * d/p/ubuntu-aa/lp-1890858-unix-socket.patch: avoid issues of some users
    to connect to libvirtd (LP: #1890858)

 -- Christian Ehrhardt <email address hidden> Mon, 14 Jun 2021 14:36:04 +0200

[Chuan Li (lccn)]

An Ubuntu pro customer has reproduced this issue on Focal that a domain user can not run 'virsh list'

domain-user@HOST:~$ virsh list --all
error: failed to connect to the hypervisor
error: End of file while reading data: Input/output error

syslog:

libvirtd[23069]: Failed to open file '/sys/kernel/security/apparmor/profiles': Permission denied
libvirtd[23069]: Failed to read AppArmor profiles list '/sys/kernel/security/apparmor/profiles': Permission denied
libvirtd[23069]: Failed to open file '/sys/kernel/security/apparmor/profiles': Permission denied
libvirtd[23069]: Failed to read AppArmor profiles list '/sys/kernel/security/apparmor/profiles': Permission denied

systemd[2928]: Started D-Bus User Message Bus.
dbus-daemon[23092]: [session uid=1000 pid=23092] AppArmor D-Bus mediation is enabled
dbus-daemon[23092]: apparmor="DENIED" operation="dbus_method_call" bus="session" path="/org/freedesktop/DBus" interface="org.freedesktop.DBus" member="Hello" mask="send" name="org.freedesktop.DBus" pid=23069 label="libvirtd" peer_label="unconfined"
libvirtd[23069]: internal error: Unable to get DBus session bus connection: An AppArmor policy prevents this sender from sending this message to this recipient; type="method_call", sender="(null)" (inactive) interface="org.freedesktop.DBus" member="Hello" error name="(unset)" requested_reply="0" destination="org.freedesktop.DBus" (bus)

kernel: [ 184.634412] audit: type=1400 audit(1692758554.332:50): apparmor="DENIED" operation="capable" profile="libvirtd" pid=2483 comm="libvirtd" capability=17 capname="sys_rawio"
kernel: [ 413.761114] audit: type=1400 audit(1692760194.668:421): apparmor="DENIED" operation="capable" profile="libvirtd" pid=2590 comm="libvirtd" capability=17 capname="sys_rawio"
kernel: [ 514.808354] audit: type=1400 audit(1692760295.712:518): apparmor="DENIED" operation="bind" profile="libvirtd" pid=4644 comm="rpc-libvirtd" family="unix" sock_type="dgram" protocol=0 requested_mask="bind" denied_mask="bind" addr="@userdb-a"
kernel: [ 522.743590] audit: type=1400 audit(1692760303.648:521): apparmor="DENIED" operation="bind" profile="libvirtd" pid=5166 comm="rpc-libvirtd" family="unix" sock_type="dgram" protocol=0 requested_mask="bind" denied_mask="bind" addr="@userdb-4"
kernel: [ 632.217763] audit: type=1400 audit(1692760413.122:523): apparmor="DENIED" operation="bind" profile="libvirtd" pid=5279 comm="rpc-libvirtd" family="unix" sock_type="dgram" protocol=0 requested_mask="bind" denied_mask="bind" addr="@userdb-e"
kernel: [ 772.196074] audit: type=1400 audit(1692760553.101:536): apparmor="DENIED" operation="bind" profile="libvirtd" pid=5395 comm="rpc-libvirtd" family="unix" sock_type="dgram" protocol=0 requested_mask="bind" denied_mask="bind" addr="@userdb-1"
kernel: [ 955.224553] audit: type=1400 audit(1692760736.123:710): apparmor="DENIED" operation="bind" profile="libvirtd" pid=5739 comm="rpc-libvirtd" family="unix" sock_type="dgram" protocol=0 requested_mask="bind" denied_mask="bind" addr="@userdb-7"

And also the customer confirmed that running below steps can fix the issue.

echo "network unix dgram," | sudo tee -a /etc/apparmor.d/local/usr.sbin.libvirtd
sudo apparmor_parser...

[Chuan Li (lccn)]

However, what's different is that the customer has enabled UCA[1] on Focal(sudo add-apt-repository cloud-archive:yoga), so the current installed version of libvirt-daemon-system is 8.0.0-1ubuntu7.6~cloud0, instead of the original 6.0.0-0ubuntu8.16 in Focal.

The customer confirmed that this issue cannot be reproduced on Focal machine without enabling UCA. I believe the fix starting from "6.0.0-0ubuntu8.10" shoud fix the issue.

I'm unclear whether this issue would also occur with the Jammy version, as the current version of libvirt-daemon-system on Jammy is 8.0.0-1ubuntu7.6.

$ rmadison libvirt-daemon-system -a amd64|egrep 'focal|jammy'
 libvirt-daemon-system | 6.0.0-0ubuntu8 | focal | amd64
 libvirt-daemon-system | 6.0.0-0ubuntu8.16 | focal-security | amd64
 libvirt-daemon-system | 6.0.0-0ubuntu8.16 | focal-updates | amd64
 libvirt-daemon-system | 6.0.0-0ubuntu8.17 | focal-proposed | amd64
 libvirt-daemon-system | 8.0.0-1ubuntu7 | jammy | amd64
 libvirt-daemon-system | 8.0.0-1ubuntu7.5 | jammy-security | amd64
 libvirt-daemon-system | 8.0.0-1ubuntu7.6 | jammy-updates | amd64
 libvirt-daemon-system | 8.0.0-1ubuntu7.7 | jammy-proposed | amd64

Or should it be said that this problem is not just an issue with libvirt-daemon-system itself, but also involves a compatibility issue with AppArmor?

Again, the following pkg versions combination on Focal 20.04.6(enabled UCA) can reproduce the issue.

ii apparmor 2.13.3-7ubuntu5.2 amd64 user-space parser utility for AppArmor
ii gir1.2-libvirt-glib-1.0:amd64 3.0.0-1 amd64 GObject introspection files for the libvirt-glib library
ii libapparmor1:amd64 2.13.3-7ubuntu5.2 amd64 changehat AppArmor library
ii libvirt-clients 8.0.0-1ubuntu7.6~cloud0 amd64 Programs for the libvirt library
ii libvirt-daemon 8.0.0-1ubuntu7.6~cloud0 amd64 Virtualization daemon
ii libvirt-daemon-config-network 8.0.0-1ubuntu7.6~cloud0 all Libvirt daemon configuration files (default network)
ii libvirt-daemon-config-nwfilter 8.0.0-1ubuntu7.6~cloud0 all Libvirt daemon configuration files (default network filters)
ii libvirt-daemon-driver-qemu 8.0.0-1ubuntu7.6~cloud0 amd64 Virtualization daemon QEMU connection driver
ii libvirt-daemon-driver-storage-rbd 8.0.0-1ubuntu7.6~cloud0 amd64 Virtualization daemon RBD storage driver
ii libvirt-daemon-system 8.0.0-1ubuntu7.6~cloud0 amd64 Libvirt daemon configuration files
ii libvirt-daemon-system-systemd 8.0.0-1ubuntu7.6~cloud0 all Libvirt daemon configuration files (systemd)
ii libvirt-glib-1.0-0:amd64 3.0.0-1 amd64 libvirt GLib and GObject mapping library
ii libvirt0:amd64 8.0.0-1ubuntu7.6~cloud0 amd64 library for interfacing with differen...

[Christian Ehrhardt  (paelzer)]

The fix is in Focal and Focal only as that is where the problem occurs.
As the old bug stated, this doesn't affect later Ubuntu releases.

You said that the customer reported that
  network unix dgram,
works, would you mind asking to try if the more restrictive
  unix (bind) type=dgram addr=@userdb-*,
would also work?

The Openstack team could then add that onto the backports they provide for Focal.
Adding a bug task for the cloud-archive.

[Chuan Li (lccn)]

@Christian, Thank you for adding a task for cloud-archive.

Our client claimed that he replaced the previous line("network unix dgram,") in /etc/apparmor.d/local/usr.sbin.libvirtd with unix (bind) type=dgram addr=@userdb-*, and the issue could not be reproduced.

[David Negreira (dnegreira)]

Backport to focal-yoga adapted with the patch from Christian

[Edward Hope-Morley (hopem)]

Comment #58 says "The fix is in Focal and Focal only as that is where the problem occurs.
As the old bug stated, this doesn't affect later Ubuntu releases." so there is no need to update the cloud archive (and fwiw to update the Yoga uca you would first update Jammy but since the Jammy version is not affected there is nothing to be done here).