OVN Metadata set up very slow

[Nguyen Thanh Cong]

Hi everybody,

I reach a bug OVN Metadata set up very slow, that make the first VM can't get metadata.

Let's say I have network 1. In a compute that doesn't have any virtual machines with that network, there will be no namespace ovnmeta-<network-id> corresponding to network1.

Now I create a virtual machine with network1 in compute node, neutron-ovn-metadata will initialize namespace ovnmeta-<network-id>. But sometimes, I don't know why it's setup very slow, lead the VM can't get metadata.

- The namespace has been initialized successfully
- IP address already set (IP of subnet & IP 169.254.169.254)
- Haproxy has started and is listening to port 80

*** BUT I try to login to VM and ping to namespace but it fails. And I try ping gateway from namespace failed to.

After a while (30s or 1-2 minutes), when I can ping to gateway from namespace or can ping metadata from VM, VM will get the metadata on reboot.

This bug sometimes happens, not always. And usually happens on compute node with high load (60%) and many flows (100k flows). I think it relate to setting up slow flows.

[Lajos Katona (lajos-katona)]

Thanks for reporting this issue. Could you please provide more information which can be useful:
* Openstack, OVN and OVs versions
* have you tried perhaps to reproduce the issue and do you have some numbers for the load from which the issue can happen?

[Nguyen Thanh Cong (congnt95)]

1. I'm use OpenStack Victoria, OVN 20.03 and OVS 2.13.5
2. I tried reproduce the issue but can't sure how to exactly make it happen again. My cluster hit this bug very often. HVs has ~100k flows, and load ~60% cpu

[Nguyen Thanh Cong (congnt95)]

I found that after successfully setting up the namespace and the VM spawning, the VM can't ping the namespace and the namespace also can't ping the gateway. After adding below 6 flows, metadata works properly, but ovn add it so slow!

Do you know function added this flows? For me continue debugging

> cookie=0x1d192cc0,table=0,priority=100,in_port="tap477df9df-b0"actions=load:0x72->NXM_NX_REG13[],load:0x283->NXM_NX_REG11[],load:0x22e->NXM_NX_REG12[],load:0x55->OXM_OF_METADATA[],load:0x1->NXM_NX_REG14[],load:0x1->NXM_NX_REG10[10],resubmit(,8)
44886a44888
> cookie=0x1d192cc0,table=32,priority=150,reg14=0x1,metadata=0x55actions=resubmit(,33)
47114c47116,47117
< cookie=0xb5ff379f,table=33,priority=100,reg15=0x8000,metadata=0x55actions=load:0x47->NXM_NX_REG13[],load:0x3f->NXM_NX_REG15[],resubmit(,34),load:0x1a->NXM_NX_REG13[],load:0x3e->NXM_NX_REG15[],resubmit(,34),load:0x8000->NXM_NX_REG15[]
---
> cookie=0xb5ff379f,table=33,priority=100,reg15=0x8000,metadata=0x55actions=load:0x72->NXM_NX_REG13[],load:0x1->NXM_NX_REG15[],resubmit(,34),load:0x47->NXM_NX_REG13[],load:0x3f->NXM_NX_REG15[],resubmit(,34),load:0x1a->NXM_NX_REG13[],load:0x3e->NXM_NX_REG15[],resubmit(,34),load:0x8000->NXM_NX_REG15[]
> cookie=0x0,table=33,priority=100,reg15=0x1,metadata=0x55actions=load:0x72->NXM_NX_REG13[],load:0x283->NXM_NX_REG11[],load:0x22e->NXM_NX_REG12[],resubmit(,34)
47846a47850
> cookie=0x0,table=34,priority=100,reg10=0/0x1,reg14=0x1,reg15=0x1,metadata=0x55actions=drop
71093a71098
> cookie=0x0,table=64,priority=100,reg10=0x1/0x1,reg15=0x1,metadata=0x55actions=push:NXM_OF_IN_PORT[],load:0xffff->NXM_OF_IN_PORT[],resubmit(,65),pop:NXM_OF_IN_PORT[]
71820a71826
> cookie=0x1d192cc0,table=65,priority=100,reg15=0x1,metadata=0x55actions=output:"tap477df9df-b0"

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hello Nguyen:

How many subnets do you have in the network where the VM port is created?

Regards.

[Nguyen Thanh Cong (congnt95)]

Only one subnet for this network. I saw the bug with so many subnet in networks, but it is not my case. I saw my namespace is set up succesfully, listen on port 80, got ip, added port to compute,..

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

As commented in IRC, the problem is not in the metadata service. The creation of the namespace, assignation of the IP address and spawn of the proxy is done in 2 seconds.

The problem seems to be be in the addition of the OF rules in OVS. But this is not related to the metadata agent.

[Nguyen Thanh Cong (congnt95)]

So where should I create a bug? Should I make another bug in the neutron project?