Libvirt kvm windows guest has no network after upgrade to 18.10

[JR]

Today I upgraded to 18.10 beta and now my windows guest doesn't have network. The network is configured to NAT. The network card is of course shown in the guest, but it doesn't get an IP via dhcp. Even configuring a static IP doesn't help - the same network IP of the host is not reachable from the guest.

[Christian Ehrhardt  (paelzer)]

Hi Jürgen,
well lets take a look at your network that is associated to the guest.

Your guest XML representation will have a network definition with a source.
Like this:
  <source network='default' bridge='virbr0'/>
Check `virsh dumpxml <guestname>` to get that.

Once you know the name check the network - in my case "default".
 $ virsh net-info default
Name: default
UUID: 3a360c07-d792-422e-818a-2a61b6ab64d0
Active: yes
Persistent: yes
Autostart: yes
Bridge: virbr0

Is your network active?

If not start checking there what might be different.

If it is active check the definition and share it here.
$ virsh net-dumpxml default
<network connections='2'>
  <name>default</name>
  <uuid>3a360c07-d792-422e-818a-2a61b6ab64d0</uuid>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='virbr0' stp='on' delay='0'/>
  <mac address='52:54:00:34:06:2f'/>
  <ip address='192.168.122.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.122.2' end='192.168.122.254'/>
    </dhcp>
  </ip>
</network>

What you see above is the "most default" network as installed by the package.
You should see a dnsmasq process running for a config file of that name, like
$ ps axlf | grep dnsmasq
5 111 5607 1 20 0 49964 2424 - S ? 0:00 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper

The backend storage for all DHCP handling is in /var/lib/libvirt/dnsmasq/.
You'll find config files with the name of the network and lease/mac files with the name of the bridge they are atatched like virbr0.status

Please check the content of these files and if they change when your guest requests access.

Finally while the guest does DHCP the journal should get entries like this:
Oct 15 10:56:23 node-horsea dnsmasq-dhcp[5607]: DHCPDISCOVER(virbr0) 52:54:00:f4:4f:2d
Oct 15 10:56:23 node-horsea dnsmasq-dhcp[5607]: DHCPOFFER(virbr0) 192.168.122.59 52:54:00:f4:4f:2d
Oct 15 10:56:23 node-horsea dnsmasq-dhcp[5607]: DHCPREQUEST(virbr0) 192.168.122.59 52:54:00:f4:4f:2d
Oct 15 10:56:23 node-horsea dnsmasq-dhcp[5607]: DHCPACK(virbr0) 192.168.122.59 52:54:00:f4:4f:2d

Please check if you see those by using "journalctl -f" just before kicking a DHCP in the guest.

[Christian Ehrhardt  (paelzer)]

Thank you for taking the time to report this bug and helping to make Ubuntu better.
I hope the general hints above help you to resolve your situation, but as-is there isn't enough to help you or fix anything.

Since there isn't enough information in your report to differentiate between a local configuration problem and a bug in Ubuntu, I'm marking this bug as Incomplete.

If indeed this is a local configuration problem, you can find pointers to get help for this sort of problem here: http://www.ubuntu.com/support/community

Or if you believe that this is really a bug, then you may find it helpful to read "How to report bugs effectively" http://www.chiark.greenend.org.uk/~sgtatham/bugs.html. We'd be grateful if you would then provide a more complete description of the problem, explain why you believe this is a bug in Ubuntu rather than a problem specific to your system, and then change the bug status back to New.

[JR (juergen-richtsfeld)]

    <interface type='network'>
      <mac address='52:54:00:41:51:b4'/>
      <source network='default'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

$ virsh net-info default
Name: default
UUID: 5db32803-fd1e-4876-98d8-da194f535c17
Active: yes
Persistent: yes
Autostart: yes
Bridge: virbr0

$ virsh net-dumpxml default
<network>
  <name>default</name>
  <uuid>5db32803-fd1e-4876-98d8-da194f535c17</uuid>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='virbr0' stp='on' delay='0'/>
  <mac address='52:54:00:71:44:4c'/>
  <ip address='192.168.122.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.122.2' end='192.168.122.254'/>
    </dhcp>
  </ip>
</network>

from the journal:
Okt 15 17:18:06 juergen kernel: virbr0: port 2(vnet0) entered blocking state
Okt 15 17:18:06 juergen kernel: virbr0: port 2(vnet0) entered disabled state
Okt 15 17:18:06 juergen kernel: device vnet0 entered promiscuous mode
Okt 15 17:18:06 juergen kernel: virbr0: port 2(vnet0) entered blocking state
Okt 15 17:18:06 juergen kernel: virbr0: port 2(vnet0) entered listening state
Okt 15 17:18:06 juergen NetworkManager[1653]: <info> [1539616686.7302] manager: (vnet0): new Tun device (/org/freedesktop/NetworkManager/Devices/8)
Okt 15 17:18:06 juergen systemd-udevd[7327]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Okt 15 17:18:06 juergen NetworkManager[1653]: <info> [1539616686.7453] devices added (path: /sys/devices/virtual/net/vnet0, iface: vnet0)
Okt 15 17:18:06 juergen NetworkManager[1653]: <info> [1539616686.7454] device added (path: /sys/devices/virtual/net/vnet0, iface: vnet0): no ifupdown configuration found.
Okt 15 17:18:06 juergen NetworkManager[1653]: <info> [1539616686.7470] device (vnet0): state change: unmanaged -> unavailable (reason 'connection-assumed', sys-iface-state: 'external')
Okt 15 17:18:06 juergen NetworkManager[1653]: <info> [1539616686.7513] keyfile: add connection in-memory (697f6001-f8b1-4acc-bc69-b562c78a1372,"vnet0")
Okt 15 17:18:06 juergen NetworkManager[1653]: <info> [1539616686.7546] device (vnet0): state change: unavailable -> disconnected (reason 'connection-assumed', sys-iface-state: 'external')
Okt 15 17:18:06 juergen NetworkManager[1653]: <info> [1539616686.7570] device (vnet0): Activation: starting connection 'vnet0' (697f6001-f8b1-4acc-bc69-b562c78a1372)
Okt 15 17:18:06 juergen libvirtd[1706]: 2018-10-15 15:18:06.757+0000: 1776: error : virNetDevSendEthtoolIoctl:3071 : ethtool ioctl error: No such device
Okt 15 17:18:06 juergen libvirtd[1706]: 2018-10-15 15:18:06.758+0000: 1776: error : virNetDevSendEthtoolIoctl:3071 : ethtool ioctl error: No such device
Okt 15 17:18:06 juergen NetworkManager[1653]: <info> [1539616686.7576] device (vnet0): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'external')
Okt 15 17:18:06 juergen libvirtd[1706]: 2018-10-15 15:18:06.758+0000: 1776: error : virNetDevSendEthtoolIoctl:3071 : ethtool ioctl error: No such device
Okt 15 17:...

[Christian Ehrhardt  (paelzer)]

Odd, that seems as if the guest request for an IP never reach the host at all.

I'll take a look at the following error:
  virNetDevSendEthtoolIoctl:3071 : ethtool ioctl error: No such device

Could you:
- check if dnsmasq processes are running as I outlined?
- have any chance to try a Linux guest as well so we know if this is win-guest specific?
- you have a virtio setup, did you configure virtio drivers for windows on your own - if so how exactly?
- Could you try to switch to a non virtio card for a test to check if it is windows virtio support?
- if it affects only windows, could you describe the version of the guest OS that is affected?

[JR (juergen-richtsfeld)]

$ ps axlf | grep dnsmasq
5 135 2533 1 20 0 27668 368 - S ? 0:00 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper
1 0 2535 2533 20 0 27640 368 - S ? 0:00 \_ /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper
5 131 2820 1 20 0 33564 388 - S ? 0:00 dnsmasq -u lxc-dnsmasq --strict-order --bind-interfaces --pid-file=/run/lxc/dnsmasq.pid --listen-address 10.0.3.1 --dhcp-range 10.0.3.2,10.0.3.254 --dhcp-lease-max=253 --dhcp-no-override --except-interface=lo --interface=lxcbr0 --dhcp-leasefile=/var/lib/misc/dnsmasq.lxcbr0.leases --dhcp-authoritative

$ sudo cat /var/lib/libvirt/dnsmasq/default.conf
##WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
##OVERWRITTEN AND LOST. Changes to this configuration should be made using:
## virsh net-edit default
## or other application using the libvirt API.
##
## dnsmasq conf file created by libvirt
strict-order
user=libvirt-dnsmasq
pid-file=/var/run/libvirt/network/default.pid
except-interface=lo
bind-dynamic
interface=virbr0
dhcp-range=192.168.122.2,192.168.122.254
dhcp-no-override
dhcp-authoritative
dhcp-lease-max=253
dhcp-hostsfile=/var/lib/libvirt/dnsmasq/default.hostsfile
addn-hosts=/var/lib/libvirt/dnsmasq/default.addnhosts

I also started the arch linux live cd, but the installer didn't get a address via dhcp, so it doesn't appear to be a a windows guest issue.
I also checked the iptables rules, I even removed them but it didn't help.

[Christian Ehrhardt  (paelzer)]

Knowing that it affects Linux systems as well for you I upgraded on a few of my test systems but they all still worked just fine.

Also the configs and processes you have LGTM.

I have no great working theory right now :-/
The only thing left is the virNetDevSendEthtoolIoctl Error, I don't expect that this is it, but since it is the only lead we have left lets dive on this a bit.
It is used for two things:
- probing of ethdev features, but even failing it would just assume the feature isn't available and go on so it should mostly be fine without
- Manage Coalesce config of a device (sounds even less to be related)

@Jürgen, could you:
- Follow [1] to enable more verbose libvirt debugging, then start a guest (note the time) and wait a few minutes. Attach the created log here and mention the time the guest was started (the log is very noisy, so the time is helpful)
- If available use a fresh host to set up an Ubuntu the way you do it and check a Linux guest there. That would allow us to identify if we are looking for a special setting of your current vs a clean host or if it is a reproducible issue.
- If the latter reproduces on the second host, it would be great to get all steps you have done from install to failure to reproduce the issue

@Everyone - if there is anyone else with issues like this please let me know so we might use the different cases to isolate what actually triggers it.

[1]: https://libvirt.org/logging.html

[JR (juergen-richtsfeld)]

Hi Christian,
today I upgraded another Kubuntu to the current 18.10 - and everything worked as desired. The only thing I had to do on both systems was to change the machine type to 'pc-i144fx-cosmic' (on both systems it was a pc-i144fx-something' that went away)

[JR (juergen-richtsfeld)]

the vm was started around 21:00, the log starts a minute earlier

[Christian Ehrhardt  (paelzer)]

For the machine type you might read [1] to get some more background.
I'm glad that it works on the second machine, but that makes it even more puzzling what might curse your first one :-/

[1]: https://wiki.ubuntu.com/QemuKVMMigration

[JR (juergen-richtsfeld)]

I just created a 2nd network, and when looking at some numbers, I just found out, that the RX packets of virbr1 and vnet1 exactly match the number of TX packets inside the guest vm. The guest vm shows 0 RX packets, while the vnet1 device shows 207 packets. The virbr[01]-nic devices have no RX/TX packets at all.

not sure if this is of any help

[JR (juergen-richtsfeld)]

As I had too much network interfaces, I uninstalled the lxc package which I didn't use anyway. Then I removed the recently created network to have everything as simple as possible. As this didn't help at all I configured the IPv4 address statically (in the guest). I must have screwed something up the last time, because after I did this, I was able to ping the guest vm from my host (yay). I'm now also able to ping the host from the guest vm and everything else (even hosts like 8.8.8.8 - even tracert in windows works).

BUT: only ping seems to work. I cannot resolve names inside my guest, no matter which DNS server I configure. I also tried opening a webpage via IP in my local net - this doesn't work either. I have no idea what could cause everything but ICPM working...

$ iptables -L -nv
Chain INPUT (policy ACCEPT 315 packets, 63571 bytes)
 pkts bytes target prot opt in out source destination
    0 0 ACCEPT udp -- virbr0 * 0.0.0.0/0 0.0.0.0/0 udp dpt:53
    0 0 ACCEPT tcp -- virbr0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:53
    0 0 ACCEPT udp -- virbr0 * 0.0.0.0/0 0.0.0.0/0 udp dpt:67
    0 0 ACCEPT tcp -- virbr0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:67

Chain FORWARD (policy ACCEPT 39 packets, 2794 bytes)
 pkts bytes target prot opt in out source destination
    9 540 ACCEPT all -- * virbr0 0.0.0.0/0 192.168.122.0/24 ctstate RELATED,ESTABLISHED
  160 10888 ACCEPT all -- virbr0 * 192.168.122.0/24 0.0.0.0/0
    0 0 ACCEPT all -- virbr0 virbr0 0.0.0.0/0 0.0.0.0/0
    0 0 REJECT all -- * virbr0 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable
    0 0 REJECT all -- virbr0 * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable

Chain OUTPUT (policy ACCEPT 427 packets, 58983 bytes)
 pkts bytes target prot opt in out source destination
    0 0 ACCEPT udp -- * virbr0 0.0.0.0/0 0.0.0.0/0 udp dpt:68

[Christian Ehrhardt  (paelzer)]

Hi Jürgen,
I must admit that I have no great idea left to follow on for now.
I'd need a way to reproduce it to debug it further on my side, but it seems special to an unknown bit of your setup for now :-/

[JR (juergen-richtsfeld)]

Hi Christian,
I did some further testing with the network interfaces, and I can report:
* networking in bridged mode (macvtap) with my ethernet device works as desired
* networking in bridged mode (macvtap) with my wifi device does *not* work
* networking in nat mode does not work on any of my devices (this is what I always used)

still, as said earlier, ping in nat mode does work, but everything else I tested does not.

[Christian Ehrhardt  (paelzer)]

Well the two working modes do not do the nat forwarding.
So some of your host config in regard to that might be the reason here.
I wish I had a better idea for you Jürgen, but atm I have none :-/

[Launchpad Janitor (janitor)]

[Expired for libvirt (Ubuntu) because there has been no activity for 60 days.]