splinters fix creates very high load

[Andrew Woodward]

When enabling the splinters fix, Alex found that the load was very high

[root@node-1 ~]# top

top - 01:16:25 up 1:11, 2 users, load average: 12.85, 12.40, 13.22
Tasks: 161 total, 10 running, 151 sleeping, 0 stopped, 0 zombie
Cpu(s): 74.0%us, 26.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 2054680k total, 1505720k used, 548960k free, 34948k buffers
Swap: 4194296k total, 13340k used, 4180956k free, 122092k cached

and that an interface was created for every vlan in the range 1000 - 2999

ovs-vsctl show http://paste.openstack.org/show/54402/

I forgot to upload the ifconfig dump but it took 17 min to iterate

# ovs-vsctl show > log && time ifconfig -a 2>&1 >> log
real 17m40.369s
user 0m20.679s
sys 1m33.022s
[root@node-1 ~]# ls

[Andrew Woodward (xarses)]

This was not the case for the patch that I generated so I'm not sure how this is necessary.

Speaking with Alex S and Ryan M, we discussed that this isn't normal that the field in ovs-vsctl is usually just trunk and not a range like this, and cant explain why so many interfaces where generated.

[Andrew Woodward (xarses)]

The other patch was also tested at a customer deployment

[Mike Scherbakov (mihgen)]

Andrew - you talking about this patch https://github.com/Mirantis/fuel/pull/849/files ?

[Andrew Woodward (xarses)]

test node is accessable via 172.18.122.109 with default credentials

[Sergey Vasilenko (xenolog)]

Andrew, does open vSwitch can specify "trunks" field for port as range?
Could you explain syntax for it definition?

About highload:
I start top and seen, who eat CPU:
Cpu(s): 72.3%us, 27.7%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 2054680k total, 1801716k used, 252964k free, 200556k buffers
Swap: 4194296k total, 12512k used, 4181784k free, 232628k cached

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
 1822 swift 20 0 134m 18m 3596 R 12.8 0.9 86:10.96 swift-account-r
 3226 swift 20 0 127m 18m 3568 R 12.8 0.9 81:54.40 swift-object-re
26659 swift 20 0 134m 18m 3596 R 12.8 0.9 87:46.28 swift-container
 2789 root 10 -10 55428 22m 6968 S 2.2 1.1 10:52.49 ovs-vswitchd
11643 quantum -2 0 167m 29m 4320 S 0.6 1.5 0:27.42 python

[Sergey Vasilenko (xenolog)]

Andrew, why you testing vlan-splinters on virtual environment?
Are you shure, that real network cards behavior identical emulated?

[root@node-1 ~]# lspci
00:00.0 Host bridge: Intel Corporation 440BX/ZX/DX - 82443BX/ZX/DX Host bridge (rev 01)
00:01.0 PCI bridge: Intel Corporation 440BX/ZX/DX - 82443BX/ZX/DX AGP bridge (rev 01)
00:07.0 ISA bridge: Intel Corporation 82371AB/EB/MB PIIX4 ISA (rev 08)
00:07.1 IDE interface: Intel Corporation 82371AB/EB/MB PIIX4 IDE (rev 01)
00:07.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08)
00:07.7 System peripheral: VMware Virtual Machine Communication Interface (rev 10)
00:0f.0 VGA compatible controller: VMware SVGA II Adapter
00:11.0 PCI bridge: VMware PCI bridge (rev 02)
00:15.0 PCI bridge: VMware PCI Express Root Port (rev 01)
00:15.1 PCI bridge: VMware PCI Express Root Port (rev 01)
00:15.2 PCI bridge: VMware PCI Express Root Port (rev 01)
00:15.3 PCI bridge: VMware PCI Express Root Port (rev 01)
00:15.4 PCI bridge: VMware PCI Express Root Port (rev 01)
00:15.5 PCI bridge: VMware PCI Express Root Port (rev 01)

[Sergey Vasilenko (xenolog)]

As I researched, OVS Vlan Splinter -- very strange thing.
Its behavior also very strange. For example, if you add tags=1001,1002,1003 to port eth3 and enable vlan splinter to the interface eth3 -- OVS creates eth3.1001, eth3.1002, eth3.1003 interfaces and put its to promisc.mode.
But if you, AFTER THIS, enable vlan splinters to the eth2 interface -- OVS automatically creates eth2.1001, eth2.1002, eth2.1003, even without defining tags=... for eth2 interface.

As I observe, OVS vlan-splinters looks like ugly unfinished workaround. And operates accordingly...

[Andrew Woodward (xarses)]

Not fixing this is not a solution, If this is what happens when splinters is enabled, its better of not supporting the interface with buggy vlan support

do you know why we changed from using trunk [], to trunk [ (list of every vlan configured in cloud network settings) ]
I think it explains why all the eth.vlan interfaces are created but i don't know why it's needed when splinters on trunk [] works with https://github.com/Mirantis/fuel/pull/849/files

[Sergey Vasilenko (xenolog)]

Do you know another way for say to kernel what 802.1q tags this NIC must decode in hardware level?

If You  shure, that Vlan-splinters should work without trunks definition on real hardware (not in emulator), I can remove 1 line of code. But under your responsibility, because I can't understand how vlan-splinters can work without it.

[Andrew Woodward (xarses)]

This is how many interfaces are created with the default settings in fuel, this is 4x over the max documented in the man page of 1024

fuel@v-node4:~$ grep -r HWaddr log
br-eth1 Link encap:Ethernet HWaddr 00:0C:29:3E:D0:7F
br-eth2 Link encap:Ethernet HWaddr 00:0C:29:3E:D0:89
br-ex Link encap:Ethernet HWaddr EE:6D:A2:FE:96:43
br-ex:ka Link encap:Ethernet HWaddr EE:6D:A2:FE:96:43
br-int Link encap:Ethernet HWaddr 2A:59:57:2B:EC:4C
br-mgmt Link encap:Ethernet HWaddr 6A:95:99:A9:67:44
br-mgmt:ka Link encap:Ethernet HWaddr 6A:95:99:A9:67:44
br-prv Link encap:Ethernet HWaddr 96:1B:B6:64:6D:41
br-storage Link encap:Ethernet HWaddr B6:85:42:BF:55:4A
eth0 Link encap:Ethernet HWaddr 00:0C:29:3E:D0:75
eth1 Link encap:Ethernet HWaddr 00:0C:29:3E:D0:7F
eth1.1 Link encap:Ethernet HWaddr 00:0C:29:3E:D0:7F
eth1.100 Link encap:Ethernet HWaddr 00:0C:29:3E:D0:7F
eth1.101 Link encap:Ethernet HWaddr 00:0C:29:3E:D0:7F
eth1.102 Link encap:Ethernet HWaddr 00:0C:29:3E:D0:7F
eth1.1000 Link encap:Ethernet HWaddr 00:0C:29:3E:D0:7F
eth1.1001 Link encap:Ethernet HWaddr 00:0C:29:3E:D0:7F
eth1.1002 Link encap:Ethernet HWaddr 00:0C:29:3E:D0:7F
eth1.1003 Link encap:Ethernet HWaddr 00:0C:29:3E:D0:7F
eth1.1004 Link encap:Ethernet HWaddr 00:0C:29:3E:D0:7F
eth1.1005 Link encap:Ethernet HWaddr 00:0C:29:3E:D0:7F
eth1.1006 Link encap:Ethernet HWaddr 00:0C:29:3E:D0:7F
eth1.1007 Link encap:Ethernet HWaddr 00:0C:29:3E:D0:7F
eth1.1008 Link encap:Ethernet HWaddr 00:0C:29:3E:D0:7F
eth1.1009 Link encap:Ethernet HWaddr 00:0C:29:3E:D0:7F
eth1.1010 Link encap:Ethernet HWaddr 00:0C:29:3E:D0:7F
eth1.1011 Link encap:Ethernet HWaddr 00:0C:29:3E:D0:7F
eth1.1012 Link encap:Ethernet HWaddr 00:0C:29:3E:D0:7F
eth1.1013 Link encap:Ethernet HWaddr 00:0C:29:3E:D0:7F
eth1.1014 Link encap:Ethernet HWaddr 00:0C:29:3E:D0:7F
eth1.1015 Link encap:Ethernet HWaddr 00:0C:29:3E:D0:7F
eth1.1016 Link encap:Ethernet HWaddr 00:0C:29:3E:D0:7F
eth1.1017 Link encap:Ethernet HWaddr 00:0C:29:3E:D0:7F
eth1.1018 Link encap:Ethernet HWaddr 00:0C:29:3E:D0:7F
eth1.1019 Link encap:Ethernet HWaddr 00:0C:29:3E:D0:7F
eth1.1020 Link encap:Ethernet HWaddr 00:0C:29:3E:D0:7F
eth1.1021 Link encap:Ethernet HWaddr 00:0C:29:3E:D0:7F
eth1.1022 Link encap:Ethernet HWaddr 00:0C:29:3E:D0:7F
eth1.1023 Link encap:Ethernet HWaddr 00:0C:29:3E:D0:7F
eth1.1024 Link encap:Ethernet HWaddr 00:0C:29:3E:D0:7F
eth1.1025 Link encap:Ethernet HWaddr 00:0C:29:3E:D0:7F
eth1.1026 Link encap:Ethernet HWaddr 00:0C:29:3E:D0:7F
eth1.1027 Link encap:Ethernet HWaddr 00:0C:29:3E:D0:7F
eth1.1028 Link encap:Ethernet HWaddr 00:0C:29:3E:D0:7F
eth1.1029 Link encap:Ethernet HWaddr 00:0C:29:3E:D0:7F
eth1.1030 Link encap:Ethernet HWaddr 00:0C:29:3E:D0:7F
eth1.1031 Link encap:Ethernet HWaddr 00:0C:29:3E:D0:7F
eth1.1032 Link encap:Ethernet HWaddr 00:0C:29:3E:D0:7F
eth1.1033 Link encap:Ethernet HWaddr 00:0C:29:3E:D0:7F
eth1.1034 Link encap:Ethernet HWaddr 00:0C:29:3E:D0:7F
eth1.1035 Link encap:Ethernet HWaddr 00:0C:29:3E:D0:7F
eth1.1036 Link encap:Ethernet HWaddr 00:0C:29:3E:D0:7F
eth1.1037 L...

[Ryan Moe (rmoe)]

According to OVS documentation (http://manpages.ubuntu.com/manpages/precise/man5/ovs-vswitchd.conf.db.5.html) a VLAN is considered in-use in all of the following situations:

 1. The VLAN is the tag value in any Port record.

 2. The VLAN is listed within the trunks column of the Port
      record of an interface on which VLAN splinters are
      enabled. An empty trunks does not influence the in-use
      VLANs: creating 4,096 VLAN devices is impractical because
      it will exceed the current 1,024 port per datapath limit.

 3. An OpenFlow flow within any bridge matches the VLAN.

Currently we have implemented #2. This causes the side-effect seen in this bug. #1 would be a better way to go.
When 1) vlan-splinters is enabled on an interface and 2) a Neutron network is created, then the ethX.<vlan> interfaces are created as needed. e.g

[root@node-10 ~]# neutron net-list
+--------------------------------------+-----------+-------------------------------------------------------+
| id | name | subnets |
+--------------------------------------+-----------+-------------------------------------------------------+
| c0702c7d-f400-468c-92fa-647079abffa8 | net04 | be64324a-ec61-4ba9-9464-2886acb43fe1 192.168.111.0/24 |
| d1ee47c7-4801-486a-a02e-8dacb6f58489 | net04_ext | 520b4129-9859-4a4b-b432-9850b9e52069 10.108.6.0/24 |
+--------------------------------------+-----------+-------------------------------------------------------+

[root@node-10 ~]# ip a | grep eth1
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
15: br-eth1: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
29: eth1.1@eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
31: eth1.100@eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
32: eth1.101@eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
33: eth1.1000@eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP

[root@node-10 ~]# neutron net-create testnet1 --provider:network_type vlan --provider:physical_network physnet2 --provider:segmentation_id 1001
Created a new network:
+---------------------------+--------------------------------------+
| Field | Value |
+---------------------------+--------------------------------------+
| admin_state_up | True |
| id | 12ff1675-1bb9-4f70-8cfa-3eae703321a2 |
| name | testnet1 |
| provider:network_type | vlan |
| provider:physical_network | physnet2 |
| provider:segmentation_id | 1001 |
| shared | False |
| status | ACTIVE |
| subnets | |
| tenant_id | 288b2093b85f40019592e984ccd50f3e |
+---------------------------+-----------...

[Sergey Vasilenko (xenolog)]

Yes, Vlan-splinters, if enabled, can monitor existing vlan tags and trunks on ovs bridges and automaticaly create ethX.NNN interfaces.

[Sergey Vasilenko (xenolog)]

But in our topology required tag may not appear on br-ethN bridge. It appear only on 2-nd level bridge (br-ex, br-prv).
Because we must list all required tags in 'trunk' list.

[Sergey Vasilenko (xenolog)]

I suppose, we should add to "known limitation":

If customer want use hardware with old unsupported NICs -- he has following ways:
* Use Ubuntu
* Use Centos + vlan-splinters, but don't use VLAN segmentation in Neutron

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/61907

[Andrew Woodward (xarses)]

Works fine for me http://paste.openstack.org/show/54998/

applied over iso 139

tested
one vm on its own network
created three vms on another network
added both networks to the same router

deployed on two computes so

net1: vm1 compute1
net2: vm2 compute2
net2: vm3 compute1
net2: vm4 compute2

all machines could communicate fine
and get out to the internet

only vlans in use are created for splinters

load on systems was acceptable

[Andrew Woodward (xarses)]

snapshot from test env

[Andrew Woodward (xarses)]

and attaching a floating ip

andreww@andreww-lt:~/git/fuel$ ssh 10.108.11.100
The authenticity of host '10.108.11.100 (10.108.11.100)' can't be established.
RSA key fingerprint is 1e:61:b0:44:be:27:76:38:ee:98:dd:af:ca:f4:06:09.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.108.11.100' (RSA) to the list of known hosts.
^C
andreww@andreww-lt:~/git/fuel$ ssh 10.108.11.100 -l cirros
cirros@10.108.11.100's password:
$ Connection to 10.108.11.100 closed.
andreww@andreww-lt:~/git/fuel$

[Vladimir Kuklin (vkuklin)]

We confirmed to introduce 3 options:

1) install soft ovs splinter workaround without explicit tag setting
2) install hard ovs splinter workaround with explicit tag setting
3) install 3.10 kernel-ml if nothing helps

These should be implemented both in API, UI and manifests.

[Andrey Korolyov (xdeller)]

Can anyone please check performance w/o any kind of swift service enabled?

[Aleksandr Shaposhnikov (alashai8)]

I'll try to do but I'm able to do this only at today's evening.
I should use ceph instead of default ? Am I correct ?

[Aleksandr Shaposhnikov (alashai8)]

There is also one limitation we should add according datapaths limits in OVS. We should limit total number of used VLAN's with splinters enabled. I think limit of 50-100 VLAN's will be reasonable. Because usually it 2-4 interfaces and 100x4 splinter interfaces will be created.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/62513

[Sergey Vasilenko (xenolog)]

> We confirmed to introduce 3 options:

> 1) install soft ovs splinter workaround without explicit tag setting
> 2) install hard ovs splinter workaround with explicit tag setting
> 3) install 3.10 kernel-ml if nothing helps

> These should be implemented both in API, UI and manifests.

#2 we have already implemented.

for implement #1 we should simple modify nailgun code:
* current "use vlan splinters" checkbox convert to radiobox of 3 items:
   * do not use vlan splinters (default)
   * use vlan splinters in SOFT mode
   * use vlan splinters in HARD mode
* pass "trunks=[0]" instead trunks list from nailgun to astute if soft mode selected.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/62736

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/62736
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=4b0454dc6a3b4866f3ad6199f58b31f47f545c67
Submitter: Jenkins
Branch: master

commit 4b0454dc6a3b4866f3ad6199f58b31f47f545c67
Author: Andrey Danin <email address hidden>
Date: Tue Dec 17 22:29:53 2013 +0400

    Extend vlan_splinters control behaviour

    Change vlan_splinters control in openstack.json from checkbox to
    radio group with 3 values: 'disabled', 'soft', and 'hard'.
    'Soft' version always provides a 'trunk' field as a [0].
    'Hard' version puts all vlan numbers from this interface to a 'trunk'
    field.

    bug 1257604

    Change-Id: I9c1bbfc95eb777937bf0a9654d6f3cbc912055b9