Packet loss while reaching public/mgmt VIP - fuel and openstack inside vsphere

[Igor Zinovik]

Packet loss while OpenStack nodes try to reach public/management VIP when Fuel and OpenStack nodes run inside vSphere VMs on ESXi hypervisors. OpenStack clients fail to create databases and tables in MySQL, because they use public URL by default which leads them to public VIP. Packet loss appears on br-ex for public network and br-mgmt on management network.

Steps to reproduce:

1. Create 4 VMs (1 master node and 3 target nodes) in vCenter 5.5 with
   NSX 6.2.
   Create portgroups for admin, public, management, storage and private
   networks.
   Each VMs has 5 NICs, each NIC is attached to separate portgroup.

   All MOS port groups on a single v5.5 vSphere distributed switch.

   Use vmxnet3 driver for all NICs of all VMs.

   Security settings for public and management portgroups (only non-default
   values for are shown):
   * Promiscouous mode - Accept
   * Forged transmits - Accept

   Security settings for Admin (PXE) portgroup:
   * Promiscouous mode - Accept

   Port group teaming and failover settings:
     Public and Management port groups:
     * Active uplinks: dvUplink1
     * Standby uplinks: dvUplink2
     All other port groups:
     * Active uplinks: dvUplink1, dvUplink2
     * Standby uplinks:

2. Install Fuel 8.0 and NSX plugin.
3. Configure cluster with vCenter and NSX plugin.
4. Deploy the cluster.

Actual result:

Deployment fails because of packet loss occurs when controller nodes try to
reach VIP.

Expected result:

Cluster gets successfully deployed.

[Artem Savinov (asavinov)]

We tried to change the linux native bridge to ovs, disable offloading on esxi host and on bridge(and all interfaces in bridge), reduce the number of rx/tx queues to 1 on vmxnet network card.

[Artem Savinov (asavinov)]

If delete "physical"(vmxnet3) nic from bridge- packet loss stops.

[Artem Savinov (asavinov)]

Packets lost regardless of the virtual nic adapter( vmxnet3 or e1000)

[Ruslan Khozinov (rkhozinov)]

Artem, I can’t understand: why a TCP session for openstack component can’t handle loss of some packets?
And could we reproduce that behaviour on our test labs?

[Artem Savinov (asavinov)]

change interrupt mode ro MSI not work
https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2058349

increase rx/tx buffer on nic not work (ethtool -G eth3 rx 4096 tx 4096)
https://access.redhat.com/sites/default/files/attachments/20150325_network_performance_tuning.pdf

[Artem Savinov (asavinov)]

Ruslan, i can not this reproduce on my test lab.
Packet loss can be more than a "few", deploy fails.

[Ruslan Khozinov (rkhozinov)]

What do you think if we’ll use one of our standard lab for vcenter (24 lcpu, 64gb ram) for hardware esxi and create a part of configuration that we’ve used on the performance lab?

[Ruslan Khozinov (rkhozinov)]

I think we must find a solution/workaround for that problem

[Artem Savinov (asavinov)]

update kernel to 3.19 (from 14.04.4) or install vmware-tools/open-vm-tools - not work.

[Artem Savinov (asavinov)]

reduction vcpu amount to 1 to reduce the number of rx / tx queues not work

[Igor Zinovik (izinovik)]

Root cause of packet loss is "Promiscuous mode: Accept" on portgroup that carries Public network traffic.
After we set it to "Reject" packet loss dissapears.

I attached two files with packet dump from ESXi host that was running controller node.

Correct settings for "Public" portgroup were:
"Promiscuous mode: Reject"
"MAC address change: Reject"
"Forged transmits: Accept" (Accept will allow traffic from linux network namespaces 'haproxy' and 'vrouter')

[Igor Zinovik (izinovik)]

Not a bug, infrastructure/configuration issue. Moving to Invalid.

[Artem Savinov (asavinov)]

If set "Promiscuous mode: Reject" on portgroup that carries Public network traffic, vip address available only from the machine on which is hosted. vSwitch does not perform MAC learning procedure, because of this it will not transmit frames for VIP MAC addresses of haproxy and vrouter namespace veth adapters, it knows only MAC addresses that were assigned to VM NICs by ESXi host during creation).
From packet dump placed above we can see, that if "promisc mode: accept" occur a duplication of garp packages, that does not occur in "promisc mode: reject". Probably it "breaks" linux native bridge forwarding table. After we set "brctl setageing br-ex 0"(bridge be work as hub) packet loss dissapears.

[Alexander Arzhanov (aarzhanov)]

Current state:
The configuration of the stand is made as in the "Bug Description".

We are faced with a duplication of garp packages after adding a second physical NIC to the dvSwitch uplink (see screenshot in link):
https://gyazo.com/4e1afc9ff793b1c93db27400c8965512
We see a problem with a duplication of garp packages, even if we put the second uplink in the "Unused uplinks" for all portgroup on dvSwitch (see screenshot in link):
https://gyazo.com/68cac17a0a4a288f301964e70c5d9f72
If we remove the second physical NIC from the dvSwitch uplink, then the problem is solved (see screenshot in link):
https://gyazo.com/4f139ac1b6eda9d9f203400c9f718693

Another way to solve the problem, if you use TWO physical NIC in dvSwitch uplink:
  set "brctl setageing br-ex 0" (bridge be work as hub) on all OpenStack controller nodes.

Link to a similar problem:
http://serverfault.com/questions/558162/arp-response-is-not-present-on-the-other-port-of-linux-bridge-based-on-ubuntu-13

[Alexander Arzhanov (aarzhanov)]

Some update:

1. I installed the vmware tools, but the problem not resolved.
2. I installed the vmware-esx-dvfilter-maclearn (http://www.virtuallyghetto.com/2014/08/new-vmware-fling-to-improve-networkcpu-performance-when-using-promiscuous-mode-for-nested-esxi.html) , but the problem not resolved.

[Alexander Arzhanov (aarzhanov)]

Update:

I created dvSwitch 4.1.0 and migrated from dvSwitch 5.5.0, check, and the problem not resolved.

[Alexander Arzhanov (aarzhanov)]

To solve the problem need enable the following options on all esxi and reboot esxi:
~ # esxcfg-advcfg -s 1 /Net/ReversePathFwdCheck
Value of ReversePathFwdCheck is 1
~ # esxcfg-advcfg -s 1 /Net/ReversePathFwdCheckPromisc
Value of ReversePathFwdCheckPromisc is 1

I mark this bug as an invalid, because the problem is on the VMware side.

What should be done:
1.Update documentation
2.Check how and when VMware will solve this error.