Ignoring the default NAT when using the virtio adapter

[The Darkness]

I am using libvirt with KVM on a UBUNTU 18.04.3 LTS

I have the following topology inside libvirt/KVM
The default virtual network with IP range 10.0.0.0/24 on virbr0 interface with IP 10.0.0.1 and DHCP enabled
Another virtual network named "confinada" with IP range 192.168.254.0/24 on virbr1 interface with no IP and DHCP disabled

I have one VM with pfSense that is connected to the two networks and is acting as a gateway to the others VMs.
I have one VM with UBUNTU 18.04 and one with Windows 7

The topology is as following:

           INTERNET
              |
              | 192.168.11.201
             HOST
              | 10.0.0.1
     (default)|
              | 10.0.0.138
           pfSense
              | 192.168.254.1
     +--------+--------+
     | (confinada) |
     | 192.168.254.2 | 192.168.254.103
  UBUNTU Windows 7

I have sucess accessing Internet on the Windows 7 VM but not on the UBUNTU machine.

During debug i found that the packets from the UBUNTU machine are not being NAT'ed correctly when leaving the host machine.
I compared the two VM and found that the UBUNTU VM is using device model "virtio" and the Windows VM is using "rtl8139".
When i changed the device model of the UBUNTU VM to "rtl8139" it start accessing the Internet.
The pfSense VM is using the device model "virtio" on both interfaces.

I tried to acess www.google.com (172.217.28.68) on the VM and used tcpdump on the host interface.
I have the following results:
1 - Using "virtio" device model:
# tcpdump -nN -i enp2s0 host 172.217.28.68
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp2s0, link-type EN10MB (Ethernet), capture size 262144 bytes
12:16:02.347830 IP 10.0.0.138.47764 > 172.217.28.68.80: Flags [S], seq 2073890688, win 29200, options [mss 1460,sackOK,TS val 3101189393 ecr 0,nop,wscale 7], length 0
12:16:03.359092 IP 10.0.0.138.47764 > 172.217.28.68.80: Flags [S], seq 2073890688, win 29200, options [mss 1460,sackOK,TS val 3101190405 ecr 0,nop,wscale 7], length 0
12:16:05.375124 IP 10.0.0.138.47764 > 172.217.28.68.80: Flags [S], seq 2073890688, win 29200, options [mss 1460,sackOK,TS val 3101192421 ecr 0,nop,wscale 7], length 0
12:16:09.631218 IP 10.0.0.138.47764 > 172.217.28.68.80: Flags [S], seq 2073890688, win 29200, options [mss 1460,sackOK,TS val 3101196677 ecr 0,nop,wscale 7], length 0

2 - Using "rtl8139" device model:
# tcpdump -nN -i enp2s0 host 172.217.28.68
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp2s0, link-type EN10MB (Ethernet), capture size 262144 bytes
12:17:15.206181 IP 192.168.11.201.6085 > 172.217.28.68.80: Flags [S], seq 733025182, win 29200, options [mss 1460,sackOK,TS val 550129346 ecr 0,nop,wscale 7], length 0
12:17:15.226156 IP 172.217.28.68.80 > 192.168.11.201.6085: Flags [S.], seq 1016175004, ack 733025183, win 60192, options [mss 1360,sackOK,TS val 1081017811 ecr 550129346,nop,wscale 8], length 0
12:17:15.227137 IP 192.168.11.201.6085 > 172.217.28.68.80: Flags [.], ack 1, win 229, options [nop,nop,TS val 550129367 ecr 1081017811], length 0
12:17:15.228442 IP 192.168.11.201.6085 > 172.217.28.68.80: Flags [P.], seq 1:142, ack 1, win 229, options [nop,nop,TS val 550129368 ecr 1081017811], length 141: HTTP: GET / HTTP/1.1

VIRTUAL NETWORK INFORMATION
virsh # net-dumpxml default
<network connections='1'>
  <name>default</name>
  <uuid>2d8b670b-a708-4914-9c4a-882a1958a931</uuid>
  <forward dev='enp2s0' mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
    <interface dev='enp2s0'/>
  </forward>
  <bridge name='virbr0' stp='on' delay='0'/>
  <mac address='52:54:00:73:6f:62'/>
  <domain name='default'/>
  <ip address='10.0.0.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='10.0.0.128' end='10.0.0.254'/>
    </dhcp>
  </ip>
</network>
virsh #

virsh # net-dumpxml confinada
<network connections='2'>
  <name>confinada</name>
  <uuid>3e9a02fa-5166-487a-a1ba-3d41e80686f9</uuid>
  <bridge name='virbr1' stp='off' delay='0'/>
  <mac address='52:54:00:9a:54:b6'/>
  <domain name='confinada'/>
</network>
virsh #

HOST SYSTEM INFORMATION:
root@jlbastos-desktop:~# lsb_release -rd
Description: Ubuntu 18.04.3 LTS
Release: 18.04
root@jlbastos-desktop:~#

root@jlbastos-desktop:~# apt-cache policy libvirt-bin libvirt-clients libvirt-daemon libvirt0 qemu-kvm
libvirt-bin:
  Instalado: 4.0.0-1ubuntu8.13
  Candidato: 4.0.0-1ubuntu8.13
  Tabela de versão:
 *** 4.0.0-1ubuntu8.13 500
        500 http://br.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     4.0.0-1ubuntu8.12 500
        500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages
     4.0.0-1ubuntu8 500
        500 http://br.archive.ubuntu.com/ubuntu bionic/main amd64 Packages
libvirt-clients:
  Instalado: 4.0.0-1ubuntu8.13
  Candidato: 4.0.0-1ubuntu8.13
  Tabela de versão:
 *** 4.0.0-1ubuntu8.13 500
        500 http://br.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     4.0.0-1ubuntu8.12 500
        500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages
     4.0.0-1ubuntu8 500
        500 http://br.archive.ubuntu.com/ubuntu bionic/main amd64 Packages
libvirt-daemon:
  Instalado: 4.0.0-1ubuntu8.13
  Candidato: 4.0.0-1ubuntu8.13
  Tabela de versão:
 *** 4.0.0-1ubuntu8.13 500
        500 http://br.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     4.0.0-1ubuntu8.12 500
        500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages
     4.0.0-1ubuntu8 500
        500 http://br.archive.ubuntu.com/ubuntu bionic/main amd64 Packages
libvirt0:
  Instalado: 4.0.0-1ubuntu8.13
  Candidato: 4.0.0-1ubuntu8.13
  Tabela de versão:
 *** 4.0.0-1ubuntu8.13 500
        500 http://br.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     4.0.0-1ubuntu8.12 500
        500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages
     4.0.0-1ubuntu8 500
        500 http://br.archive.ubuntu.com/ubuntu bionic/main amd64 Packages
qemu-kvm:
  Instalado: 1:2.11+dfsg-1ubuntu7.20
  Candidato: 1:2.11+dfsg-1ubuntu7.20
  Tabela de versão:
 *** 1:2.11+dfsg-1ubuntu7.20 500
        500 http://br.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages
        100 /var/lib/dpkg/status
     1:2.11+dfsg-1ubuntu7 500
        500 http://br.archive.ubuntu.com/ubuntu bionic/main amd64 Packages
root@jlbastos-desktop:~#

[Christian Ehrhardt  (paelzer)]

Hi,
interesting but nothing obvious comes directly to my mind.
It is great that you identified that the device driver virtio/rtl8139 makes a difference.
Lets start there to think about it.

Traffic for rtl8139 will always go to host-userspace for device emulation and flow on from there.
With virtio in comparison I think the default if available will be vhost-net which will handle things in the host kernel and should not (tm) but might behave differently.

While vhost-net is usually better for efficiency and performance lets check if that makes the difference.
Please set back the Ubuntu VM to virtio but then in addition to
   <model type="virtio"/>
also set
   <driver name="qemu"/>

The default is "try vhost but if failing (e.g. module not present) fall back to qemu"; but setting qemu explicitly there will force it to use the userspace backend.

Please let me know how virtio behaves in that case.
If we can move this from rtl8139-vs-virtio to virtio with/without vhost that already would be a good step to get closer to the root cause.

[The Darkness (jlbastos)]

I tried with and without the driver option set to qemu and the problem continues.

[The Darkness (jlbastos)]

I made some more tests and found that ICMP packets are NAT'ed OK independent of the device model used.

[The Darkness (jlbastos)]

I found that the problem only happens when the "virtio" device model is used in both pfSense internal and UBUNTU interfaces.
I tested all combinations and get the following results:
     pfSense
default confinada UBUNTU result
virtio virtio virtio ERR
virtio virtio rtl8139 OK

virtio rtl8139 virtio OK
virtio rtl8139 rtl8139 OK

rtl8139 virtio virtio ERR
rtl8139 virtio rtl8139 OK

rtl8139 rtl8139 virtio OK
rtl8139 rtl8139 rtl8139 OK

[Christian Ehrhardt  (paelzer)]

Maybe it fast-pathes around something when both are virtio.
But you said you set driver=qemu and it still was broken with virtio, I didn't expect that.

I have given it a few tries with a similar setup among a few VMs but haven't seen the issue myself.

You probably want to report that upstream to get more eyes/brains onto that case.
But with the case as-is atm I'm not really sure if that would be better started at libvirt [1] or qemu [2]. Maybe start with one and if redirected from there go to the other?

I'm subscribed on both, but would appreciate if you could add here a link to the thread you started on the ML to find it again later.

[1]: https://www.redhat.com/mailman/listinfo/libvir-list
[2]: https://lists.nongnu.org/mailman/listinfo/qemu-devel

[The Darkness (jlbastos)]

I will recreate this scenario on my home computer so i could have more flexibility on making tests.
I am sick and will be off for this week.
But i will return to this ASAP.

[Christian Ehrhardt  (paelzer)]

> I will recreate this scenario on my home computer so i could have more flexibility on making tests.

Sounds great, recreating it also means you might end up with a set of
configs/commands anyone can use to recreate the case on a system from
scratch.
Reproducibility will help on this already well filed bug.

> I am sick and will be off for this week.
> But i will return to this ASAP.

Get well soon, this isn't a rush - do it at the time you can and feel well.

[The Darkness (jlbastos)]

I was able to recreate the topology on my home machine and was able to reproduce the problem.
I will make more tests and report upstream soon.

[Launchpad Janitor (janitor)]

[Expired for libvirt (Ubuntu) because there has been no activity for 60 days.]