Virtual machine is present in vrouter-agent even after delete

[Prakash Bailkeri]

R2.0 Build 8
VM object is active in both oper and config in Vrouter-agent. Verified that interfaces are deleted and VM object is actually deleted in control-node.

UUID of the hanging VM is e72ca5ee-7628-4404-b6ff-1bb3773f61cf

VM is created for SNAT

2015-01-14 Wed 10:41:48:521.032 CET compute1-1 [Thread 46997046331136, Pid 7082]: NetNS run command: /usr/bin/opencontrail-vrouter-netns create source-nat e72ca5ee-7628-4404-b6ff-1bb3773f61cf 1b77de67-a5dd-4bb2-ae8b-c5d210a72d23 8ccad98c-490e-4f32-bb95-7fb373ee4fe8 --vmi-left-ip 100.64.0.5/29 --vmi-right-ip 10.88.125.130/28 --vmi-left-mac 02:1b:77:de:67:a5 --vmi-right-mac 02:8c:ca:d9:8c:49
2015-01-14 Wed 10:41:48:521.197 CET compute1-1 [Thread 46997046331136, Pid 7082]: NetNS run command queued: /usr/bin/opencontrail-vrouter-netns create source-nat e72ca5ee-7628-4404-b6ff-1bb3773f61cf 1b77de67-a5dd-4bb2-ae8b-c5d210a72d23 8ccad98c-490e-4f32-bb95-7fb373ee4fe8 --vmi-left-ip 100.64.0.5/29 --vmi-right-ip 10.88.125.130/28 --vmi-left-mac 02:1b:77:de:67:a5 --vmi-right-mac 02:8c:ca:d9:8c:49
2015-01-14 Wed 11:29:01:298.706 CET compute1-1 [Thread 46998032529152, Pid 7082]: NetNS run command: /usr/bin/opencontrail-vrouter-netns destroy source-nat e72ca5ee-7628-4404-b6ff-1bb3773f61cf 1b77de67-a5dd-4bb2-ae8b-c5d210a72d23 8ccad98c-490e-4f32-bb95-7fb373ee4fe8
2015-01-14 Wed 11:29:01:298.833 CET compute1-1 [Thread 46998032529152, Pid 7082]: NetNS run command queued: /usr/bin/opencontrail-vrouter-netns destroy source-nat e72ca5ee-7628-4404-b6ff-1bb3773f61cf 1b77de67-a5dd-4bb2-ae8b-c5d210a72d23 8ccad98c-490e-4f32-bb95-7fb373ee4fe8

Gcore was taken. and can be seen in http://mayamruga/Docs/bugs/<id>

[Ashish Ranjan (aranjan-n)]

Could you report as per Naveen. Can't make out anything from the gcore.

[Praveen (praveen-karadakal)]

The link from virtual-machine default-domain__admin__si_51fbf123-df71-4be5-8b73-2d383162dbe2__1 to service-instance default-domain:admin:si_51fbf123-df71-4be5-8b73-2d383162dbe2 is not deleted. Hence, the virtual-machine object is not deleted.

From XmppMessageTrace:

At time 2015-01-14 15:11:48.505160

            <link>
                <node type=\"virtual-machine\">
                    <name>default-domain__admin__si_51fbf123-df71-4be5-8b73-2d383162dbe2__1</name>
                </node>
                <node type=\"service-instance\">
                    <name>default-domain:admin:si_51fbf123-df71-4be5-8b73-2d383162dbe2</name>
                </node>
                <metadata type=\"virtual-machine-service-instance\" />
            </link>

There is no delete to the link between service-instance to virtual-machine. There is no delete for both service-instance and virtual-machine config nodes.

[Prakash Bailkeri (prakashmb)]

Setup had two control-nodes and 3 config nodes

I had verified in one control node that the Virtual-machine object was totally removed.

Agent gcore was taken and also, I had verified in the agent introspect that the VM was still present with neighbors. Agent gcore (traces) shows that VM add config message and missing DELETE of VM.

Possibilities are
1. Control-node (where introspect was not verified) had the VM and agent was connected to it. (IFMap servers are in inconsistent state)
2. config DELETE message was dropped/lost.

Since the VM was deleted in control-node, I don't suspect the issue to be in SVC monitor.

Since the gcore of the control-node(s) is missing, we can't make much progress.