Virtio split ring reconnect case failed with qemu version>4.1.0

[xuan]

Issue discription：

Virtio split ring reconnect case failed with qemu version>4.1.0, no throughput after reconnect from vhost-user and vhost print log "VHOST_DATA : Failed to allocate memory for mbuf." when sending 1518 length packets.

Test Environment:
DPDK version: DPDK v20.05
OS: Linux 5.4
Compiler: gcc (Ubuntu 7.3.0-16ubuntu3) 8.4.0
Hardware platform: Purley.

The reproduce step is :
Test Case : pvp test with virtio 1.0 mergeable path
 ====================================================
1. Bind one port to vfio-pci, then launch testpmd by below command::

rm -rf vhost-net*
 ./testpmd -l 1-3 -n 4 --socket-mem 1024,1024 \
 --vdev 'eth_vhost0,iface=vhost-net,queues=2,client=1' -- -i --nb-cores=2 --rxq=2 --txq=2 --txd=1024 --rxd=1024
 testpmd>set fwd mac
 testpmd>start

2. launch VM with different qemu version separately, note: we need add "disable-modern=false" to enable virtio 1.0::
/home/qemu-install/qemu-4.2.0/bin/qemu-system-x86_64 -name vm0 -enable-kvm -pidfile /tmp/.vm0.pid -daemonize -monitor unix:/tmp/vm0_monitor.sock,server,nowait -device e1000,netdev=nttsip1 -netdev user,id=nttsip1,hostfwd=tcp:10.67.116.150:6001-:22 -chardev socket,id=char0,path=/root/dpdk/vhost-net,server -netdev type=vhost-user,id=netdev0,chardev=char0,vhostforce,queues=2 -device virtio-net-pci,netdev=netdev0,mac=52:54:00:00:00:01,disable-modern=false,mrg_rxbuf=on,rx_queue_size=1024,disable-modern=false,tx_queue_size=1024,mq=on,vectors=15 -cpu host -smp 14 -m 4096 -object memory-backend-file,id=mem,size=4096M,mem-path=/mnt/huge,share=on -numa node,memdev=mem -mem-prealloc -chardev socket,path=/tmp/vm0_qga0.sock,server,nowait,id=vm0_qga0 -device virtio-serial -device virtserialport,chardev=vm0_qga0,name=org.qemu.guest_agent.0 -vnc :4 -drive file=/home/osimg/ubuntu1910.img

3. On VM, bind virtio net to igb_uio and run testpmd without tx-offloads::

./testpmd -c 0x7 -n 4 – -i --enable-hw-vlan-strip\
 --nb-cores=2 --rxq=2 --txq=2 --txd=1024 --rxd=1024
 testpmd>start

4. Send 64B and 1518B packets by packet generator separately, show throughput with below command::

testpmd>show port stats all

5. Quit vhost-user, then re-launch, check if the reconnect can work and ensure the traffic can continue.

6. Kill VM, then re-launch VM, check if the reconnect can work and ensure the traffic can continue.

Analysis: this is a regression bug introduced by QEMU(bad commit:6ab79a20af). The change adds a flag to prevent repeated calls to vhost_net_stop(). However, it prevents any calls to vhost_net_cleanup() from net_vhost_user_event(). This caused failure to hot-remove a vhost user from guest because of mem leak.