On a VN with many VMs, updating VN's flood_unknown_unicast flag causes flow setup delay
Affects | Status | Importance | Assigned to | Milestone | ||
---|---|---|---|---|---|---|
Juniper Openstack | Status tracked in Trunk | |||||
R3.0 |
Fix Committed
|
High
|
Praveen | |||
Trunk |
Fix Committed
|
High
|
Praveen |
Bug Description
R3.0.2.0 Build 26 ubuntu 14.04 Kilo multinode setup
On a VN , there are about 100 vms spread across 9 compute nodes.
It is seen that a property such as flood_unknown_
Tasktrace :
SandeshTraceTex
traces
2016-04-20 10:41:43.545 TaskTrace: 32 0 TBB schedule time(in usec) 92 Function 1 controller/
2016-04-20 10:41:43.687 TaskTrace: 26 0 Schedule delay(in usec) 141319 Function 1 controller/
2016-04-20 10:41:43.687 TaskTrace: 26 0 TBB schedule time(in usec) 141340 Function 1 controller/
2016-04-20 10:41:43.691 TaskTrace: 1 0 Schedule delay(in usec) 143575 Netlink interface listener controller/
2016-04-20 10:41:43.691 TaskTrace: 1 0 TBB schedule time(in usec) 143604 Netlink interface listener controller/
2016-04-20 10:41:43.999 TaskTrace: 1 0 Run time(in usec) 46140 DBPartition QueueRunner controller/
2016-04-20 10:41:44.081 TaskTrace: 1 0 Run time(in usec) 23063 DBPartition QueueRunner controller/
2016-04-20 10:42:13.081 TaskTrace: 10 0 Run time(in usec) 83173 Vrouter stats collector controller/
2016-04-20 10:42:13.081 TaskTrace: 16 0 Schedule delay(in usec) 83371 VnUveTimer controller/
2016-04-20 10:42:13.082 TaskTrace: 16 0 TBB schedule time(in usec) 83394 VnUveTimer controller/
2016-04-20 10:42:13.082 TaskTrace: 16 0 Schedule delay(in usec) 83724 VmUveTimer controller/
2016-04-20 10:42:13.082 TaskTrace: 16 0 TBB schedule time(in usec) 83748 VmUveTimer controller/
2016-04-20 10:42:13.082 TaskTrace: 16 0 Schedule delay(in usec) 84143 UveDBWalkTimer controller/
2016-04-20 10:42:13.082 TaskTrace: 16 0 TBB schedule time(in usec) 84163 UveDBWalkTimer controller/
2016-04-20 10:42:13.082 TaskTrace: 16 0 Schedule delay(in usec) 84193 InterfaceUveTimer controller/
2016-04-20 10:42:13.082 TaskTrace: 16 0 TBB schedule time(in usec) 84205 InterfaceUveTimer controller/
2016-04-20 10:42:13.083 TaskTrace: 10 0 Schedule delay(in usec) 84642 Function 1 controller/
2016-04-20 10:42:13.083 TaskTrace: 10 0 TBB schedule time(in usec) 84677 Function 1 controller/
2016-04-20 10:44:16.006 TaskTrace: 1 0 Schedule delay(in usec) 64700 DBPartition QueueRunner controller/
2016-04-20 10:44:16.006 TaskTrace: 1 0 TBB schedule time(in usec) 64713 DBPartition QueueRunner controller/
2016-04-20 10:44:59.924 TaskTrace: 1 0 Run time(in usec) 20454 DBPartition QueueRunner controller/
2016-04-20 10:45:00.034 TaskTrace: 1 0 Schedule delay(in usec) 60524 DBPartition QueueRunner controller/
2016-04-20 10:45:00.034 TaskTrace: 1 0 TBB schedule time(in usec) 60538 DBPartition QueueRunner controller/
2016-04-20 10:45:10.732 TaskTrace: 1 0 Run time(in usec) 24163 DBPartition QueueRunner controller/
2016-04-20 10:45:10.793 TaskTrace: 1 0 Schedule delay(in usec) 60509 DBPartition QueueRunner controller/
2016-04-20 10:45:10.793 TaskTrace: 1 0 TBB schedule time(in usec) 60516 DBPartition QueueRunner controller/
2016-04-20 10:45:10.795 TaskTrace: 1 0 Schedule delay(in usec) 60617 Config Manager controller/
2016-04-20 10:45:10.795 TaskTrace: 1 0 TBB schedule time(in usec) 60631 Config Manager controller/
2016-04-20 10:45:33.615 TaskTrace: 1 0 Schedule delay(in usec) 82762 Config Manager controller/
2016-04-20 10:45:33.615 TaskTrace: 1 0 TBB schedule time(in usec) 82774 Config Manager controller/
2016-04-20 10:45:33.615 TaskTrace: 1 0 Schedule delay(in usec) 78213 DBPartition QueueRunner controller/
2016-04-20 10:45:33.615 TaskTrace: 1 0 TBB schedule time(in usec) 78221 DBPartition QueueRunner controller/
2016-04-20 10:47:06.039 TaskTrace: 1 0 Run time(in usec) 22312 DBPartition QueueRunner controller/
2016-04-20 10:47:33.689 TaskTrace: 1 0 Run time(in usec) 20788 DBPartition QueueRunner controller/
2016-04-20 10:47:33.773 TaskTrace: 1 0 Schedule delay(in usec) 68738 DBPartition QueueRunner controller/
2016-04-20 10:47:33.773 TaskTrace: 1 0 TBB schedule time(in usec) 68749 DBPartition QueueRunner controller/
2016-04-20 10:47:33.947 TaskTrace: 1 0 Schedule delay(in usec) 56250 DBPartition QueueRunner controller/
2016-04-20 10:47:33.947 TaskTrace: 1 0 TBB schedule time(in usec) 56261 DBPartition QueueRunner controller/
2016-04-20 10:47:54.859 TaskTrace: 1 0 Run time(in usec) 23958 DBPartition QueueRunner controller/
2016-04-20 10:48:08.054 TaskTrace: 1 0 Run time(in usec) 20469 DBPartition QueueRunner controller/
2016-04-20 10:48:08.089 TaskTrace: 1 0 Schedule delay(in usec) 55169 TaskTrigger:
2016-04-20 10:48:08.089 TaskTrace: 1 0 TBB schedule time(in usec) 55179 TaskTrigger:
2016-04-20 10:48:08.864 TaskTrace: 1 0 Schedule delay(in usec) 82987 DBPartition QueueRunner controller/
2016-04-20 10:48:08.864 TaskTrace: 1 0 TBB schedule time(in usec) 83011 DBPartition QueueRunner controller/
-------
ex hping3 output when the issue happens :
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=1702 win=14600 rtt=428.8 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=1703 win=14600 rtt=438.6 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=1704 win=14600 rtt=434.4 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=1705 win=14600 rtt=442.8 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=1706 win=14600 rtt=443.6 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=1707 win=14600 rtt=451.9 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=1708 win=14600 rtt=453.6 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=1709 win=14600 rtt=443.6 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=1710 win=14600 rtt=451.4 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=1711 win=14600 rtt=449.3 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=1712 win=14600 rtt=453.6 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=1713 win=14600 rtt=463.2 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=1714 win=14600 rtt=464.6 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=1715 win=14600 rtt=466.1 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=1717 win=14600 rtt=473.6 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=1718 win=14600 rtt=472.1 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=1719 win=14600 rtt=475.9 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=1720 win=14600 rtt=473.9 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=1721 win=14600 rtt=488.0 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=1722 win=14600 rtt=489.7 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=1723 win=14600 rtt=493.7 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=1724 win=14600 rtt=499.0 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=1725 win=14600 rtt=508.8 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=1726 win=14600 rtt=1504.4 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=1727 win=14600 rtt=510.9 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=1728 win=14600 rtt=509.4 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=1729 win=14600 rtt=519.8 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=1730 win=14600 rtt=517.3 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=1731 win=14600 rtt=528.1 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=1732 win=14600 rtt=538.0 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=1733 win=14600 rtt=535.0 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=1734 win=14600 rtt=545.9 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=1735 win=14600 rtt=543.0 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=1736 win=14600 rtt=554.0 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=1737 win=14600 rtt=557.7 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=1738 win=14600 rtt=570.1 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=1739 win=14600 rtt=568.1 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=1740 win=14600 rtt=571.9 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=1741 win=14600 rtt=568.0 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=1742 win=14600 rtt=579.1 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=1743 win=14600 rtt=578.2 ms
tags: | added: performance |
Review in progress for https:/ /review. opencontrail. org/19731
Submitter: Praveen K V (<email address hidden>)