The vnics are disappearing in the vm

[YG Kumar]

Hi,

We have a rocky OSA setup of branch 18.1.9.

When we create a vm from a particular image, the vm comes with two missing vnics inside of it , out of four, which we have provisioned to it from four dhcp tenant networks. The plugin is OVS and the firewall driver is conntrack. It was working earlier until we have noticed this issue.

So, if we reboot the VM, then, the vnics appear for a short time and again after a few seconds, they are disappearing. The disappearing vnics inside the vm have a floating ip associated with one of it and hence the vm is becoming unpingable. Again if we reboot the vm , it is comping up for a short time and again vanishing. At the moment when the pinging of the vm stops, we are not noticing any messages in the neutron server logs for that port, except when we try to manually do a vm reboot:

LOGS
-------------
2020-09-18 08:41:50.197 17808 INFO neutron.wsgi [req-ff4ad70c-568d-4625-9bea-2a472351d00a dae4d1b704943b11cb10287e984f9367070915c18f7cadd48f915af92d4b4d03 35391b98793b4c09bf87c91006d123c2 - f7834cb0083b4f8f81184b6595b46b34 f7834cb0083b4f8f81184b6595b46b34] 172.29.236.183,172.29.236.21 "GET /v2.0/floatingips?tenant_id=35391b98793b4c09bf87c91006d123c2&port_id=095edf83-2d8d-494d-b820-ef7540aefa7c&port_id=0da3066c-a0f5-49df-a10a-20919595a5b8&port_id=298916ee-91a2-428f-86aa-c5ed5f034563&port_id=3e6d29fe-7bee-47d4-bc98-00d934fc5764&port_id=6d456cbe-425a-4d17-86f6-2b77ab88a42f&port_id=853adf00-92ac-4444-8487-32a77e3efb66&port_id=86ec85de-609a-403d-b027-3097ac597e0c&port_id=86f1034d-837d-4e67-ad5e-63d9642a0b2a&port_id=947c5610-b0f4-439a-abf8-51ba3dc8d212&port_id=b00f8eae-18fa-44c4-92e6-9ee75c7c599c&port_id=b8ba1a0c-ebd9-4278-bc83-9b60e1036f63&port_id=c6ce1a3c-b8b3-493d-b052-2810efacbf5e&port_id=c7facefa-565f-46be-8048-a333505ee177&port_id=d5e1fc5f-1a84-4366-bf36-548f2bdc0366 HTTP/1.1" status: 200 len: 4363 time: 0.0851929
2020-09-18 08:41:50.585 17814 INFO neutron.wsgi [req-349c0d49-9671-43a4-a541-f04facff2ee7 c42abde21dee4c848dc653df8ec429aa e02428b1700247b98ad1d563133f6174 - default default] 172.29.236.57,172.29.236.21 "GET /v2.0/floatingips?fixed_ip_address=172.16.1.19&port_id=86f1034d-837d-4e67-ad5e-63d9642a0b2a HTTP/1.1" status: 200 len: 1042 time: 0.0781569
2020-09-18 08:41:59.951 17808 INFO neutron.wsgi [req-4bc06b83-cb44-469a-9ef1-0ce9a4fa0753 dae4d1b704943b11cb10287e984f9367070915c18f7cadd48f915af92d4b4d03 35391b98793b4c09bf87c91006d123c2 - f7834cb0083b4f8f81184b6595b46b34 f7834cb0083b4f8f81184b6595b46b34] 172.29.236.183,172.29.236.21 "GET /v2.0/floatingips?tenant_id=35391b98793b4c09bf87c91006d123c2&port_id=298916ee-91a2-428f-86aa-c5ed5f034563&port_id=6d456cbe-425a-4d17-86f6-2b77ab88a42f&port_id=86f1034d-837d-4e67-ad5e-63d9642a0b2a&port_id=947c5610-b0f4-439a-abf8-51ba3dc8d212 HTTP/1.1" status: 200 len: 1042 time: 0.0917962
2020-09-18 08:42:06.094 17817 DEBUG neutron.plugins.ml2.rpc [req-9cc5b8b2-a616-466d-9ed8-eae2b9b6056b - - - - -] Device 86f1034d-837d-4e67-ad5e-63d9642a0b2a up at agent ovs-agent-b7w update_device_up /openstack/venvs/neutron-18.1.9/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.py:256
2020-09-18 08:42:06.151 17817 DEBUG neutron.db.provisioning_blocks [req-9cc5b8b2-a616-466d-9ed8-eae2b9b6056b - - - - -] Provisioning complete for port 86f1034d-837d-4e67-ad5e-63d9642a0b2a triggered by entity L2. provisioning_complete /openstack/venvs/neutron-18.1.9/lib/python2.7/site-packages/neutron/db/provisioning_blocks.py:138
-----------------

The port we are talking about is "86f1034d-837d-4e67-ad5e-63d9642a0b2a" in the above logs.

------------
1.9/lib/python2.7/site-packages/neutron/notifiers/nova.py:242
2020-09-18 08:41:56.744 16567 DEBUG novaclient.v2.client [-] REQ: curl -g -i -X POST http://wtl-int.sandvine.cloud:8774/v2.1/os-server-external-events -H "Accept: application/json" -H "Content-Type: application/json" -H "User-Agent: python-novaclient" -H "X-Auth-Token: {SHA1}c8de3bb0dae419214d99c02879f89bd3a6a4dd78" -H "X-OpenStack-Nova-API-Version: 2.1" -d '{"events": [{"status": "completed", "tag": "6d456cbe-425a-4d17-86f6-2b77ab88a42f", "name": "network-vif-unplugged", "server_uuid": "713376a9-c354-4fb7-946c-e926c1cd9412"}, {"status": "completed", "tag": "298916ee-91a2-428f-86aa-c5ed5f034563", "name": "network-vif-unplugged", "server_uuid": "713376a9-c354-4fb7-946c-e926c1cd9412"}, {"status": "completed", "tag": "86f1034d-837d-4e67-ad5e-63d9642a0b2a", "name": "network-vif-unplugged", "server_uuid": "713376a9-c354-4fb7-946c-e926c1cd9412"}]}' _http_log_request /openstack/venvs/neutron-18.1.9/lib/python2.7/site-packages/keystoneauth1/session.py:448
2020-09-18 08:41:56.801 16567 DEBUG novaclient.v2.client [-] RESP BODY: {"events": [{"status": "completed", "tag": "6d456cbe-425a-4d17-86f6-2b77ab88a42f", "name": "network-vif-unplugged", "server_uuid": "713376a9-c354-4fb7-946c-e926c1cd9412", "code": 200}, {"status": "completed", "tag": "298916ee-91a2-428f-86aa-c5ed5f034563", "name": "network-vif-unplugged", "server_uuid": "713376a9-c354-4fb7-946c-e926c1cd9412", "code": 200}, {"status": "completed", "tag": "86f1034d-837d-4e67-ad5e-63d9642a0b2a", "name": "network-vif-unplugged", "server_uuid": "713376a9-c354-4fb7-946c-e926c1cd9412", "code": 200}]} _http_log_response /openstack/venvs/neutron-18.1.9/lib/python2.7/site-packages/keystoneauth1/session.py:511
2020-09-18 08:41:56.803 16567 INFO neutron.notifiers.nova [-] Nova event response: {u'status': u'completed', u'tag': u'86f1034d-837d-4e67-ad5e-63d9642a0b2a', u'name': u'network-vif-unplugged', u'server_uuid': u'713376a9-c354-4fb7-946c-e926c1cd9412', u'code': 200}
---------------------

-------------

2020-09-18 08:42:01.553 16363 DEBUG novaclient.v2.client [-] REQ: curl -g -i -X POST http://wtl-int.sandvine.cloud:8774/v2.1/os-server-external-events -H "Accept: application/json" -H "Content-Type: application/json" -H "User-Agent: python-novaclient" -H "X-Auth-Token: {SHA1}e28d38f7ee67e8258671b28e839dbdf049f13b48" -H "X-OpenStack-Nova-API-Version: 2.1" -d '{"events": [{"status": "completed", "tag": "6d456cbe-425a-4d17-86f6-2b77ab88a42f", "name": "network-vif-plugged", "server_uuid": "713376a9-c354-4fb7-946c-e926c1cd9412"}, {"status": "completed", "tag": "298916ee-91a2-428f-86aa-c5ed5f034563", "name": "network-vif-plugged", "server_uuid": "713376a9-c354-4fb7-946c-e926c1cd9412"}, {"status": "completed", "tag": "86f1034d-837d-4e67-ad5e-63d9642a0b2a", "name": "network-vif-plugged", "server_uuid": "713376a9-c354-4fb7-946c-e926c1cd9412"}]}' _http_log_request /openstack/venvs/neutron-18.1.9/lib/python2.7/site-packages/keystoneauth1/session.py:448
2020-09-18 08:42:01.608 16363 DEBUG novaclient.v2.client [-] RESP BODY: {"events": [{"status": "completed", "tag": "6d456cbe-425a-4d17-86f6-2b77ab88a42f", "name": "network-vif-plugged", "server_uuid": "713376a9-c354-4fb7-946c-e926c1cd9412", "code": 200}, {"status": "completed", "tag": "298916ee-91a2-428f-86aa-c5ed5f034563", "name": "network-vif-plugged", "server_uuid": "713376a9-c354-4fb7-946c-e926c1cd9412", "code": 200}, {"status": "completed", "tag": "86f1034d-837d-4e67-ad5e-63d9642a0b2a", "name": "network-vif-plugged", "server_uuid": "713376a9-c354-4fb7-946c-e926c1cd9412", "code": 200}]} _http_log_response /openstack/venvs/neutron-18.1.9/lib/python2.7/site-packages/keystoneauth1/session.py:511
2020-09-18 08:42:01.610 16363 INFO neutron.notifiers.nova [-] Nova event response: {u'status': u'completed', u'tag': u'86f1034d-837d-4e67-ad5e-63d9642a0b2a', u'name': u'network-vif-plugged', u'server_uuid': u'713376a9-c354-4fb7-946c-e926c1cd9412', u'code': 200}
------------

Please suggest a workaround if it is a known bug.

Thanks
Kumar

[YG Kumar (ygk-kmr)]

The base OS of the computes and controllers is Ubuntu 18

[Hongbin Lu (hongbin.lu)]

@YG Kumar,

Please post the relevant neutron config files. In addition, how do you create the VM with four nics? please post the VM configuration and the network/subnet/port of those four nics.

[Hongbin Lu (hongbin.lu)]

Neutron side needs more information to continue the troubleshooting.

[Balazs Gibizer (balazs-gibizer)]

Does the timing of the above network-vif-unplugged event correlates to the disappearance of the vnics from the guest?

Does this event comes at every restart of the guest?

I'm also setting this to Incomplete from Nova perspective.

[Launchpad Janitor (janitor)]

[Expired for neutron because there has been no activity for 60 days.]

[Launchpad Janitor (janitor)]

[Expired for OpenStack Compute (nova) because there has been no activity for 60 days.]