agent crash at FlowEntry::InitRevFlow during sanity run

[Vedamurthy Joshi]

R2.20 Build 108 Ubuntu 12.04 Icehouse

Core is in /cs-shared/test_runs/nodea12/2015_11_10_20_22_13/nodei15_core.contrail-vroute.21994.nodei15.1447183956.gz

Build : 2.22-108
CoreLocation : /cs-shared/test_runs/nodea12/2015_11_10_20_22_13
cores : {'10.204.217.192': ['core.contrail-vroute.2772.nodei13.1447178803'], '10.204.217.127': ['core.contrail-vroute.21994.nodei15.1447183956', 'core.contrail-vroute.23177.nodei15.1447184180']}
LogsLocation : http://10.204.216.50/Docs/logs/2.22-108_2015_11_10_20_22_13/logs/
Report : http://10.204.216.50/Docs/logs/2.22-108_2015_11_10_20_22_13/junit-noframes.html
Topology :
Config Nodes : [u'nodea12', u'nodea15']
Control Nodes : [u'nodea15', u'nodea17']
Compute Nodes : [u'nodei13', u'nodei14', u'nodei15']
Openstack Node : nodea12
WebUI Node : nodea12
Analytics Nodes : [u'nodea12']

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/contrail-vrouter-agent'.
Program terminated with signal 6, Aborted.
#0 0x00007f876ca540d5 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#0 0x00007f876ca540d5 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f876ca5783b in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007f876ca9132e in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007f876ca9bb26 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x00007f876d38dff6 in std::string::assign(std::string const&) ()
   from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5 0x0000000000b31bff in FlowEntry::InitRevFlow(PktFlowInfo const*, PktInfo const*, PktControlInfo const*, PktControlInfo const*) ()
#6 0x0000000000b53b10 in PktFlowInfo::Add(PktInfo const*, PktControlInfo*, PktControlInfo*) ()
#7 0x0000000000b625f0 in FlowHandler::Run() ()
#8 0x0000000000b5f3c4 in Proto::ProcessProto(boost::shared_ptr<PktInfo>) ()
#9 0x0000000000b5ff62 in boost::detail::function::function_obj_invoker1<boost::_bi::bind_t<bool, boost::_mfi::mf1<bool, Proto, boost::shared_ptr<PktInfo> >, boost::_bi::list2<boost::_bi::value<Proto*>, boost::arg<1> > >, bool, boost::shared_ptr<PktInfo> >::invoke(boost::detail::function::function_buffer&, boost::shared_ptr<PktInfo>) ()
#10 0x0000000000b612d9 in QueueTaskRunner<boost::shared_ptr<PktInfo>, WorkQueue<boost::shared_ptr<PktInfo> > >::RunQueue() ()
#11 0x0000000001088ba5 in TaskImpl::execute() ()
#12 0x00007f876d60ce52 in ?? () from /usr/lib/libtbb.so.2
#13 0x00007f876d608c2d in ?? () from /usr/lib/libtbb.so.2
#14 0x00007f876d6080db in ?? () from /usr/lib/libtbb.so.2
#15 0x00007f876d605c1f in ?? () from /usr/lib/libtbb.so.2
#16 0x00007f876d605e59 in ?? () from /usr/lib/libtbb.so.2
#17 0x00007f876d823e9a in start_thread ()
   from /lib/x86_64-linux-gnu/libpthread.so.0
#18 0x00007f876cb1138d in clone () from /lib/x86_64-linux-gnu/libc.so.6
#19 0x0000000000000000 in ?? ()

[Hari Prasad Killi (haripk)]

Flow key shows invalid family.

(gdb) fr 6
#6 InitFlowCmn (rev_ctrl=0x7f87372fba10, ctrl=0x7f87372fba50, info=0x7f87372fbaa0, this=0x7f87580095d0) at controller/src/vnsw/agent/pkt/flow_table.cc:1432
1432 controller/src/vnsw/agent/pkt/flow_table.cc: No such file or directory.

(gdb) p /x *this
$6 = {
....
  key_ = {
    family = 0x58000078,
    nh = 0x7f87,
    src_addr = {
      type_ = 0x0,
      ipv4_address_ = {
        addr_ = {
          s_addr = 0x301020a
        }
      },
      ipv6_address_ = {
---Type <return> to continue, or q <return> to quit---
        addr_ = {
          __in6_u = {
            __u6_addr8 = {0x0 <repeats 16 times>},
            __u6_addr16 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
            __u6_addr32 = {0x0, 0x0, 0x0, 0x0}
          }
        },
        scope_id_ = 0x0
      }
    },
    dst_addr = {
      type_ = 0x0,
      ipv4_address_ = {
        addr_ = {
          s_addr = 0x301010a
        }
      },
      ipv6_address_ = {
        addr_ = {
          __in6_u = {
            __u6_addr8 = {0x0 <repeats 16 times>},
            __u6_addr16 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
            __u6_addr32 = {0x0, 0x0, 0x0, 0x0}
          }
        },
        scope_id_ = 0x0
      }
    },
    protocol = 0x11,
    src_port = 0x238c,
    dst_port = 0x13aa
  },
...

[Hari Prasad Killi (haripk)]

Not seen after other flow changes.