Agent:flow got stuck in hold state after overnight traffic was run
Affects | Status | Importance | Assigned to | Milestone | ||
---|---|---|---|---|---|---|
Juniper Openstack | Status tracked in Trunk | |||||
R3.0 |
Fix Committed
|
High
|
Anand H. Krishnan | |||
R3.1 |
Fix Committed
|
High
|
Anand H. Krishnan | |||
Trunk |
Fix Committed
|
High
|
Anand H. Krishnan |
Bug Description
Build:R3.02 35 kilo
Had the below setup
Configuration
===============
1.NAT service chain with 8 node ecmp
2.TCP server running on a node outside this cluster.The server was listening on 1000 ports
3.Launched 20 vms with tcp client , which on bootup, tries to connect this tcp server on all those 1000 ports.And continuously connecting/
This setup ran overnight.
In the morning, deleted all those 20 vms.In nodel11 , one flow got stuck in hold state.
Logs saved at :http://
Setup:
------
host1 = 'root@10.
host2 = 'root@10.
host3 = 'root@10.
host4 = 'root@10.
host5 = 'root@10.
host6 = 'root@10.
host7 = 'root@10.
host8 = 'root@10.
host9 = 'root@10.
host10 = 'root@10.
host11 = 'root@10.
host12 = 'root@10.
host13 = 'root@10.
host14 = 'root@10.
host15 = 'root@10.
host16 = 'root@10.
host17 = 'root@10.
host18 = 'root@10.
host19 = 'root@10.
host20 = 'root@10.
host21 = 'root@10.
host22 = 'root@10.
host23 = 'root@10.
ext_routers = [('blr-mx2', '10.204.216.245')]
router_asn = 64512
public_vn_rtgt = 30001
#public_vn_subnet = "10.204.219.72/29"
host_build = 'vjoshi@
env.roledefs = {
'all': [host1, host2, host3, host4, host5, host6,host7, host8, host9, host10, host11, host12, host13, host14, host15,
'cfgm': [host1, host2, host3],
'openstack': [host4, host5, host6],
'webui': [host1, host2, host3],
'control': [host1, host2, host3],
'compute': [host7, host8, host9, host10, host11, host12, host13, host14, host15,
'collector': [host1, host2, host3],
'database': [host1, host2, host3],
'build': [host_build],
}
env.hostnames = {
'all': ['nodei27', 'nodei28', 'nodei35', 'nodei32', 'nodei35', 'nodei36', 'nodei37', 'nodei38', 'nodel4', 'nodel7', 'nodel8', 'nodel9', 'nodel10', 'nodel11', 'nodel12'
}
Changed in juniperopenstack: | |
assignee: | Hari Prasad Killi (haripk) → Anand H. Krishnan (anandhk) |
importance: | Undecided → High |
information type: | Proprietary → Public |
[5/12/16, 9:16:17 AM] Sandip Dey: Nodel11…
[5/12/16, 9:16:25 AM] Sandip Dey: this floe got stuck seems
[5/12/16, 9:16:26 AM] Sandip Dey: (Gen: 169, K(nh):158, Action:F, Flags:, S(nh):104, Stats:1/28, SPort 56550)
4644474 192.168.16.17:42233 6 (6)
10. 204.219. 157:1738
(Gen: 80, K(nh):134, Action:H, Flags:M, TCP:S, S(nh):0, Stats:23/1380, SPort 0)
[5/12/16, 9:53:39 AM] vedujoshi: somehow , the flow got added and deleted very quickly domain: sandipd: sandipd_ internal_ floating_ ip_net: service- b0f9dfe2- 43f0-4232- a3b8-2b1d2eb31d 33-default- domain_ sandipd_ pt_instance l3_flow = 1 smac = 00:00:00:00:00:00 dmac = 00:00:00:00:00:00 drop_reason = UNKNOWN table_id = 3 ] file = controller/ src/vnsw/ agent/pkt/ flow_mgmt. cc line = 484 domain: sandipd: sandipd_ internal_ floating_ ip_net: service- b0f9dfe2- 43f0-4232- a3b8-2b1d2eb31d 33-default- domain_ sandipd_ pt_instance l3_flow = 1 smac = 00:00:00:00:00:00 dmac = 00:00:00:00:00:00 drop_reason = UNKNOWN table_id = 3 ] file = controller/ src/vnsw/ agent/pkt/ flow_mgmt. cc line = 484
[5/12/16, 9:53:39 AM] vedujoshi: 2016-05-12 09:48:15.258 FlowTrace: operation = ADD info= [ gen_id = 80 flow_index = 4644474 nh_id = 134 source_ip = 192.168.16.17 source_port = 42233 destination_ip = 10.204.219.157 destination_port = 1738 protocol = 6 vrf = 4294967295 mirror_l= [ [ ] ] mirror_vrf = 65535 implicit_deny = 0 short_flow = 1 source_vn_list= [ [ (iter95) = UNKNOWN_, ] ] dest_vn_list= [ [ (iter96) = UNKNOWN_, ] ] source_vn_match = dest_vn_match = source_sg_id_l= [ [ ] ] dest_sg_id_l= [ [ ] ] vrf_assign = default-
2016-05-12 09:48:15.303 FlowTrace: operation = DEL info= [ gen_id = 80 flow_index = 4644474 nh_id = 134 source_ip = 192.168.16.17 source_port = 42233 destination_ip = 10.204.219.157 destination_port = 1738 protocol = 6 vrf = 4294967295 mirror_l= [ [ ] ] mirror_vrf = 65535 implicit_deny = 0 short_flow = 1 source_vn_list= [ [ (iter95) = UNKNOWN_, ] ] dest_vn_list= [ [ (iter96) = UNKNOWN_, ] ] source_vn_match = dest_vn_match = source_sg_id_l= [ [ ] ] dest_sg_id_l= [ [ ] ] vrf_assign = default-