libvirtd failed to connect to socket

[Ivan Sussanin]

grep "syslog" /var/log/syslog:
---start grep---
Apr 20 12:30:25 bling libvirtd[1129]: libvirt version: 6.0.0, package: 0ubuntu8.8 (Victor Manuel Tapia King <email address hidden> Fri, 19 Feb 2021 17:15:56 +0100)
Apr 20 12:30:25 bling libvirtd[1129]: hostname: bling
Apr 20 12:30:25 bling libvirtd[1129]: internal error: Failed to apply firewall rules /usr/sbin/ip6tables --table filter --list-rules: ip6tables v1.8.4 (legacy): can't initialize ip6tables table `filter': Address family not supported by protocol#012Perhaps ip6tables or your kernel needs to be upgraded.
Apr 20 12:30:26 bling dnsmasq[1405]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses
Apr 20 12:30:26 bling dnsmasq-dhcp[1405]: read /var/lib/libvirt/dnsmasq/default.hostsfile
Apr 20 12:52:32 bling libvirtd[1129]: Failed to connect socket to '/var/run/libvirt/virtquemud-sock': No such file or directory
Apr 20 12:52:32 bling libvirtd[1129]: End of file while reading data: Input/output error
Apr 20 12:53:09 bling systemd[1]: Stopping Suspend/Resume Running libvirt Guests...
Apr 20 12:53:34 bling kernel: [ 7.694197] audit: type=1400 audit(1618923214.067:2): apparmor="STATUS" operation="profile_load" profile="unconfined" name="libvirtd" pid=916 comm="apparmor_parser"
Apr 20 12:53:34 bling kernel: [ 7.694200] audit: type=1400 audit(1618923214.067:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="libvirtd//qemu_bridge_helper" pid=916 comm="apparmor_parser"
Apr 20 12:53:35 bling systemd[1]: Starting Suspend/Resume Running libvirt Guests...
Apr 20 12:53:35 bling systemd[1]: Finished Suspend/Resume Running libvirt Guests.
---/end grep---
This was one session log from restart.

root@bling:~# ls /var/run/libvirt
hostdevmgr interface libvirt-admin-sock libvirt-sock libvirt-sock-ro network nodedev nwfilter nwfilter-binding qemu secrets storage virtlockd-admin-sock virtlockd-sock virtlogd-admin-sock virtlogd-sock

Steps made from zero to probable bug, logged in as root after initial install:
1. clean install ubuntu 20.04.2, reboot, log in, configure root, reboot, log in as root
2. apt-get upgrade, apt-get update, apt-get dist-upgrade, reboot
3. sudo apt install qemu qemu-kvm libvirt-clients libvirt-daemon-system virtinst bridge-utils, reboot
4. sudo service libvirtd start
5. root@bling:~/kvm# sudo update-rc.d libvirtd enable
   update-rc.d: error: cannot find a LSB script for libvirtd
6. virt-install [follows virtual machine conf description which works very well in 3 other computers]
   ERROR Failed to connect socket to '/var/run/libvirt/virtquemud-sock': No such file or directory

To me, more enduser than Ubuntu specialist, seems like "virtquemud-sock" has excess "u" after "q", must be "virtqemud-sock".

ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: libvirt-daemon-system 6.0.0-0ubuntu8.8
ProcVersionSignature: Ubuntu 5.4.0-72.80-generic 5.4.101
Uname: Linux 5.4.0-72-generic x86_64
ApportVersion: 2.20.11-0ubuntu27.16
Architecture: amd64
CasperMD5CheckResult: pass
Date: Tue Apr 20 14:55:10 2021
InstallationDate: Installed on 2021-04-17 (3 days ago)
InstallationMedia: Ubuntu-Server 20.04.1 LTS "Focal Fossa" - Release amd64 (20200731)
ProcEnviron:
 SHELL=/bin/bash
 LANG=en_US.UTF-8
 TERM=xterm
 XDG_RUNTIME_DIR=<set>
 PATH=(custom, no user)
SourcePackage: libvirt
UpgradeStatus: No upgrade log present (probably fresh install)
modified.conffile..etc.libvirt.nwfilter.allow-arp.xml: [modified]
modified.conffile..etc.libvirt.nwfilter.allow-dhcp-server.xml: [modified]
modified.conffile..etc.libvirt.nwfilter.allow-dhcp.xml: [modified]
modified.conffile..etc.libvirt.nwfilter.allow-incoming-ipv4.xml: [modified]
modified.conffile..etc.libvirt.nwfilter.allow-ipv4.xml: [modified]
modified.conffile..etc.libvirt.nwfilter.clean-traffic-gateway.xml: [modified]
modified.conffile..etc.libvirt.nwfilter.clean-traffic.xml: [modified]
modified.conffile..etc.libvirt.nwfilter.no-arp-ip-spoofing.xml: [modified]
modified.conffile..etc.libvirt.nwfilter.no-arp-mac-spoofing.xml: [modified]
modified.conffile..etc.libvirt.nwfilter.no-arp-spoofing.xml: [modified]
modified.conffile..etc.libvirt.nwfilter.no-ip-multicast.xml: [modified]
modified.conffile..etc.libvirt.nwfilter.no-ip-spoofing.xml: [modified]
modified.conffile..etc.libvirt.nwfilter.no-mac-broadcast.xml: [modified]
modified.conffile..etc.libvirt.nwfilter.no-mac-spoofing.xml: [modified]
modified.conffile..etc.libvirt.nwfilter.no-other-l2-traffic.xml: [modified]
modified.conffile..etc.libvirt.nwfilter.no-other-rarp-traffic.xml: [modified]
modified.conffile..etc.libvirt.nwfilter.qemu-announce-self-rarp.xml: [modified]
modified.conffile..etc.libvirt.nwfilter.qemu-announce-self.xml: [modified]
modified.conffile..etc.libvirt.qemu.networks.default.xml: [deleted]
mtime.conffile..etc.libvirt.nwfilter.allow-arp.xml: 2021-04-20T12:26:43.197729
mtime.conffile..etc.libvirt.nwfilter.allow-dhcp-server.xml: 2021-04-20T12:26:43.201729
mtime.conffile..etc.libvirt.nwfilter.allow-dhcp.xml: 2021-04-20T12:26:43.157729
mtime.conffile..etc.libvirt.nwfilter.allow-incoming-ipv4.xml: 2021-04-20T12:26:43.189729
mtime.conffile..etc.libvirt.nwfilter.allow-ipv4.xml: 2021-04-20T12:26:43.225729
mtime.conffile..etc.libvirt.nwfilter.clean-traffic-gateway.xml: 2021-04-20T12:26:43.177729
mtime.conffile..etc.libvirt.nwfilter.clean-traffic.xml: 2021-04-20T12:26:43.121729
mtime.conffile..etc.libvirt.nwfilter.no-arp-ip-spoofing.xml: 2021-04-20T12:26:43.129729
mtime.conffile..etc.libvirt.nwfilter.no-arp-mac-spoofing.xml: 2021-04-20T12:26:43.141729
mtime.conffile..etc.libvirt.nwfilter.no-arp-spoofing.xml: 2021-04-20T12:26:43.209729
mtime.conffile..etc.libvirt.nwfilter.no-ip-multicast.xml: 2021-04-20T12:26:43.145729
mtime.conffile..etc.libvirt.nwfilter.no-ip-spoofing.xml: 2021-04-20T12:26:43.165729
mtime.conffile..etc.libvirt.nwfilter.no-mac-broadcast.xml: 2021-04-20T12:26:43.153729
mtime.conffile..etc.libvirt.nwfilter.no-mac-spoofing.xml: 2021-04-20T12:26:43.181729
mtime.conffile..etc.libvirt.nwfilter.no-other-l2-traffic.xml: 2021-04-20T12:26:43.213729
mtime.conffile..etc.libvirt.nwfilter.no-other-rarp-traffic.xml: 2021-04-20T12:26:43.169729
mtime.conffile..etc.libvirt.nwfilter.qemu-announce-self-rarp.xml: 2021-04-20T12:26:43.133729
mtime.conffile..etc.libvirt.nwfilter.qemu-announce-self.xml: 2021-04-20T12:26:43.221729

[Ivan Sussanin (ivansu)]

Fixing posting text error:
first line must be 'grep "libvirt" /var/log/syslog' not 'grep "syslog" /var/log/syslog'.
-
Sry, Ivan

[Christian Ehrhardt  (paelzer)]

Hi Ivan,
thanks for the report and your help to make Ubuntu better.
I must admit I haven't seen this in all the time that 20.04 is out already.

I was checking logs of default installs of various Ubuntu versions but that problem doesn't seem to happen by default.

> To me, more enduser than Ubuntu specialist, seems like "virtquemud-sock" has excess "u" after
> "q", must be "virtqemud-sock".

Indeed - if you look at the source you'll find plenty of "virtqemud" but no "virtquemud".

TBH the /var/run/libvirt/virtqemud-sock (even if not mistyped) would be part of the so called "split daemon" approach which Ubuntu didn't switch to yet (not even in 21.04).

I haven't found a mention of the right (nor the wrong) name in virt-manager (which is the code base of virt-install) either.

Due to monolithic and split daemon being allowed in parallel what happens is probing for split and then fallback to the monolithic daemon then. That I can see in my logs as well:
  Apr 15 11:11:00 Keschdeichel libvirtd[160109]: Failed to connect socket to '/var/run/libvirt/virtqemud-sock': No such file or directory

But that isn't a real problem - it is probing one and then using the present, working former approach. Also the above (my example) only happened with a virsh that was part of a git-build and therefore was looking for split daemons - nothing that really is in the archive of Ubuntu 20.04.

My guess atm is that somewhere on your system must be a custom config or a custom build of some of the tools here that refers to "virtquemud-sock".
Does
$ grep -Hrn quemud /etc/
show anything on your affected system?

Thanks for providing steps to reproduce, I was trying your steps and also again simplifying them:
Note (FYI): step 4 and 5 of your steps are not needed as that is default after install of libvirt-daemon-system package.

$ apt install libvirt-daemon-system
$ virt-install --name u --ram 1024 --disk path=/var/lib/libvirt/images/ubuntu1804.qcow2,size=8 --vcpus 1 --os-type linux --os-variant generic --graphics none --location 'http://archive.ubuntu.com/ubuntu/dists/bionic/main/installer-amd64/'

The above installs and does not trigger the error you described.

Since I knew that I had it in my logs from when I was experimenting with other builds I was looking back at the tweaks I've done and one was debug-logs.
So I set /etc/libvirt/libvirtd.conf to
log_level = 2
log_outputs="2:syslog:libvirtd"

But even with that I can't trigger the issue as reported.

So let me try to summarize:
1. split daemons is a config option in recent libvirt >5.7, but not used in Ubuntu builds
2. the missed socket of /var/run/libvirt/virtqemud-sock would be part of the split daemon virtqemud.service which we don't ship
3. the typo to "/var/run/libvirt/virtquemud-sock" would be in config files for virtqemud.service which we don't ship

I recommend checking your local system for:
- custom config that might have triggered this
- custom builds that expect or set up split daemons

Extra Question:
I know that some behaviro of libvirt and virsh and such tools do depend on environment variables, do you get the same issue when you do not log in as root but as normal user (use virt-install with ...

[Ivan Sussanin (ivansu)]

Thank you Christian for one of best explanation I have seen!
IT WAS MY TYPO not bug, and I got everything work.
This problem was so strange that I made second clean install system. It worked.

I MADE TYPO to /etc/network/interfaces conf file, bridge definition:
...
auto kvmbr0
iface kvmbr0 inet static
 bridge ports none #must be "bridge_ports none", with _ (underscore) not space
...
No errors in syslog and dmesg, just ... no bridge.
-
How "/var/run/libvirt/virtquemud-sock" (with additional "u") catches up I don't understand, "grep -rnw '/etc/' -e 'virtqemud-sock'" shows nothing, also /var, /root, /usr and other searchable folders under /. So this spelling may be binary, somewhere in code.

Now commented out entire bridge definition in /etc/network/interfaces , so no bridge:
virt-install [virtual machine definition] shows again "/var/run/libvirt/virtquemud-sock", yes with additional "u".
but virsh handles missing bridge very well:
virsh start win2016 #virtual is defined previously
error: Failed to start domain win2016
error: Cannot get interface MTU on 'kvmbr0': No such device

Seems like that wrong spelling "virtquemud-sock" and the whole misleading errormessage comes from virt-install or it's dependencies. Probably "no bridge" error-handling goes unhandled and wild.
Sorry for false bugalarm.

[Christian Ehrhardt  (paelzer)]

NP, glad I could help.

Marking the bug as invalid then and best of luck with your VM!