QEMU

Bug #638955
Comment #27

Comment 27 for bug 638955

Revision history for this message

Stefan Hajnoczi (stefanha) wrote on 2011-03-05:

#27

I was able to reproduce this problem with qemu.git running OpenIndiana 148 with tap and bridge on the host. I did not see an issue with the userspace network stack - seems to manifest itself as a checksum error in transmitted packets.

Here is the host tcpdump during a TCP stall with mtu 1500:

19:47:54.601950 IP 192.168.122.33.22 > 192.168.122.1.40611: Flags [P.], seq 6949:7509, ack 3545, win 64436, options [nop,nop,TS val 24455 ecr 111832709], length 560
19:47:54.601966 IP 192.168.122.1.40611 > 192.168.122.33.22: Flags [.], ack 7509, win 163, options [nop,nop,TS val 111832710 ecr 24455], length 0
19:47:54.602312 IP 192.168.122.33.22 > 192.168.122.1.40611: Flags [P.], seq 7509:8069, ack 3545, win 64436, options [nop,nop,TS val 24455 ecr 111832709], length 560
19:47:54.602325 IP 192.168.122.1.40611 > 192.168.122.33.22: Flags [.], ack 8069, win 171, options [nop,nop,TS val 111832710 ecr 24455], length 0

Everything went fine up to here but now the stall shows up...

19:47:54.602594 IP 192.168.122.33.22 > 192.168.122.1.40611: Flags [P.], seq 8069:8629, ack 3545, win 64436, options [nop,nop,TS val 24455 ecr 111832709], length 560
19:47:54.602831 IP 192.168.122.33.22 > 192.168.122.1.40611: Flags [P.], seq 8629:9189, ack 3545, win 64436, options [nop,nop,TS val 24455 ecr 111832709], length 560
19:47:54.602847 IP 192.168.122.1.40611 > 192.168.122.33.22: Flags [.], ack 8069, win 171, options [nop,nop,TS val 111832710 ecr 24455,nop,nop,sack 1 {8629:9189}], length 0

Notice that only seq up to 8069 was acked by the host and this is a duplicate ack. I think it's prodding the guest to transmit from 8069 again.

19:47:54.603447 IP 192.168.122.33.22 > 192.168.122.1.40611: Flags [P.], seq 9189:9749, ack 3545, win 64436, options [nop,nop,TS val 24456 ecr 111832710], length 560
19:47:54.603459 IP 192.168.122.1.40611 > 192.168.122.33.22: Flags [.], ack 8069, win 171, options [nop,nop,TS val 111832710 ecr 24455,nop,nop,sack 1 {8629:9749}], length 0
19:47:54.603734 IP 192.168.122.33.22 > 192.168.122.1.40611: Flags [P.], seq 9749:10309, ack 3545, win 64436, options [nop,nop,TS val 24456 ecr 111832710], length 560
19:47:54.603751 IP 192.168.122.1.40611 > 192.168.122.33.22: Flags [.], ack 8069, win 171, options [nop,nop,TS val 111832710 ecr 24455,nop,nop,sack 1 {8629:10309}], length 0
19:47:54.603882 IP 192.168.122.33.22 > 192.168.122.1.40611: Flags [P.], seq 8069:8629, ack 3545, win 64436, options [nop,nop,TS val 24456 ecr 111832710], length 560
19:47:55.021608 IP 192.168.122.33.22 > 192.168.122.1.40611: Flags [.], seq 8069:9517, ack 3545, win 64436, options [nop,nop,TS val 24498 ecr 111832710], length 1448
19:47:55.578667 STP 802.1d, Config, Flags [none], bridge-id 8000.da:7b:46:27:8c:aa.8001, length 35
19:47:55.851350 IP 192.168.122.33.22 > 192.168.122.1.40611: Flags [.], seq 8069:9517, ack 3545, win 64436, options [nop,nop,TS val 24581 ecr 111832710], length 1448
19:47:57.577496 STP 802.1d, Config, Flags [none], bridge-id 8000.da:7b:46:27:8c:aa.8001, length 35
19:47:57.625504 IP 192.168.122.33.22 > 192.168.122.1.40611: Flags [.], seq 8069:9517, ack 3545, win 64436, options [nop,nop,TS val 24745 ecr 111832710], length 1448

Resends and more duplicate acks up to 8069. The host is not responding to the guest transmitted packets. Wireshark shows checksum errors for guest transmitted frames when the stall occurs.

I added instrumentation to hw/e1000.c and get the following information about transmitted frames:

tp 0x7fd6a8eef3a0 frames 0 size 626 vlan_needed 0 sum_needed 0x3 ip 0 tcp 0
tucso 0x32 tcp/udp checksum 0xdcf7
tp 0x7fd6a8eef3a0 frames 0 size 626 vlan_needed 0 sum_needed 0x3 ip 0 tcp 0
tucso 0x32 tcp/udp checksum 0xde66
tp 0x7fd6a8eef3a0 frames 0 size 626 vlan_needed 0 sum_needed 0 ip 0 tcp 0
tucso 0x32 tcp/udp checksum 0x77ca
tp 0x7fd6a8eef3a0 frames 0 size 626 vlan_needed 0 sum_needed 0x3 ip 0 tcp 0
tucso 0x32 tcp/udp checksum 0xf7a1
tp 0x7fd6a8eef3a0 frames 0 size 626 vlan_needed 0 sum_needed 0x3 ip 0 tcp 0
tucso 0x32 tcp/udp checksum 0xfe9d
tp 0x7fd6a8eef3a0 frames 0 size 626 vlan_needed 0 sum_needed 0x3 ip 0 tcp 0
tucso 0x32 tcp/udp checksum 0x50b9
tp 0x7fd6a8eef3a0 frames 0 size 626 vlan_needed 0 sum_needed 0 ip 0 tcp 0
tucso 0x32 tcp/udp checksum 0x77ca
tp 0x7fd6a8eef3a0 frames 0 size 1514 vlan_needed 0 sum_needed 0 ip 0 tcp 0
tucso 0x32 tcp/udp checksum 0x7b42
tp 0x7fd6a8eef3a0 frames 0 size 1514 vlan_needed 0 sum_needed 0 ip 0 tcp 0
tucso 0x32 tcp/udp checksum 0x7b42
tp 0x7fd6a8eef3a0 frames 0 size 1514 vlan_needed 0 sum_needed 0 ip 0 tcp 0
tucso 0x32 tcp/udp checksum 0x7b42
tp 0x7fd6a8eef3a0 frames 0 size 1514 vlan_needed 0 sum_needed 0 ip 0 tcp 0
tucso 0x32 tcp/udp checksum 0x7b42
tp 0x7fd6a8eef3a0 frames 0 size 1514 vlan_needed 0 sum_needed 0 ip 0 tcp 0
tucso 0x32 tcp/udp checksum 0x7b42

Perhaps there is a e1000 emulation bug here that causes us to miss the sum_needed bits and an invalid checksum ends up being transmitted. Need to investigate this more.

Here is the patch in case you want to confirm my results so far:
http://repo.or.cz/w/qemu/stefanha.git/commitdiff/fa963c73b254af2e43a9a45ff5cceb2c42519f55

I was able to reproduce this problem with qemu.git running OpenIndiana 148 with tap and bridge on the host.  I did not see an issue with the userspace network stack - seems to manifest itself as a checksum error in transmitted packets.

Here is the host tcpdump during a TCP stall with mtu 1500:

Everything went fine up to here but now the stall shows up...

Notice that only seq up to 8069 was acked by the host and this is a duplicate ack.  I think it's prodding the guest to transmit from 8069 again.

Resends and more duplicate acks up to 8069.  The host is not responding to the guest transmitted packets.  Wireshark shows checksum errors for guest transmitted frames when the stall occurs.

I added instrumentation to hw/e1000.c and get the following information about transmitted frames:

Perhaps there is a e1000 emulation bug here that causes us to miss the sum_needed bits and an invalid checksum ends up being transmitted.  Need to investigate this more.

Here is the patch in case you want to confirm my results so far:
http://repo.or.cz/w/qemu/stefanha.git/commitdiff/fa963c73b254af2e43a9a45ff5cceb2c42519f55