Nova-compute has not sent the delete to agent in this case. The file for the orphan vif was still observed under /var/lib/contrail/ports


Hi Andriy,

Thanks for your analysis. Since the file for orphan vif tapbfe87886-e0 is still present in /var/lib/contrail/ports/, nova-compute has not sent delete to agent and that is the reason why delete has not been sent to vrouter. We need to check as to why nova-compute didn’t send the delete.

Regards,
Ashok




On 2/10/16, 2:46 PM, "Andriy Berestovskyy" <email address hidden> wrote:

Hi Ashok,
Thanks for the very detailed instructions. Answering your questions:

(1)
I run the script and got one orphan vif tapbfe87886-e0.
Sandesh Trace Request Config shows just add messages for this vif
while for other interfaces there are adds and deletes.


(2)
Sure, the file for the vif is present in /var/lib/contrail/ports. The
instance id is 1cd58568-a830-4f45-ac0a-a42e88229516

In the nova-compute.log for the instance id there are few strange logs
(I cut out the dates):

WARNING nova.compute.manager [-] [instance:
1cd58568-a830-4f45-ac0a-a42e88229516] Instance shutdown by itself.
Calling the stop API. Current vm_state: active, current task_state:
None, current DB power_state: 4, current VM power_state: 4

INFO nova.compute.manager [req-d0242ecf-6bf8-4cc1-9836-edb3ca8c0c29
None] [instance: 1cd58568-a830-4f45-ac0a-a42e88229516] Instance is
already powered off in the hypervisor when stop is called.

WARNING nova.virt.libvirt.driver [-] [instance:
1cd58568-a830-4f45-ac0a-a42e88229516] During wait destroy, instance
disappeared


(3)
I guess due to (2) there are no delete messages sent to the vRouter.

Andriy


On Tue, Feb 9, 2016 at 3:30 PM, Ashok Singh R <email address hidden> wrote:
Hi Andriy,

I didn’t get chance to run your script.

Would it be possible for you to check the following in your setup?

(1)
http://<compute-node-ip>:8085/Snh_SandeshTraceRequest?x=Config


This page should have logs which indicate whether the agent is receiving add/delete requests

(2) For Orphan vif, you can also check whether entry is present in directory.
/var/log/contrail/ports/
This directory should have one file for each port. In most cases this file is created/deleted by nova-compute on port-add/delete events. It should create/delete the file even when agent is dead

If the entry for the orphan vif is still present, please check the log file /var/log/nova/nova-compute.log to see if there are any errors/exceptions etc.

(3)
http://<compute-node-ip>:8085/Snh_SandeshTraceRequest?x=KSync%20Interface

This page should have information on whether agent sent the request to vrouter and whether vrouter acked it or not.

Regards,
Ashok




On 2/8/16, 9:19 PM, "Andriy Berestovskyy" <email address hidden> wrote:

Hi Ashok,
I just run my old script and got the "orphan" vifs again, so the issue
is still there. Please find the script attached.

How do we check the messages Agent receives/sends?

Andriy

On Sun, Feb 7, 2016 at 9:32 AM, Ashok Singh R <email address hidden> wrote:
Hi Andriy,

I am working on bug
https://bugs.launchpad.net/juniperopenstack/+bug/1495002. One of issues
mentioned in the bug is vif delete message gets lost. Do you have more
information on that? I have the following questions.

Did contrail-vrouter-agent receive the delete message?
If contrail-vrouter-agent received delete message, did it send delete
message to vrouter or not?


Regards,
Ashok