[R 2.20] SNAT: Instance VN not created sometimes leading to SNAT failures

[amit surana]

32 private VNs are connected to a public GW via a LR. Setting up/tearing down will sometimes cause the instance VN to not be created. Details below:

----

I see that the instance_vn is not created for some reason. The LR has been created and a private VN and GW have been attached to it. The service instance has been created with right interface in the GW VN, but because the instance VN isn't there, SNAT doesn't work.

Also seems like the message queue has pending messages on it that haven't been dequeued by svc-mon.

svc_mon.csol2-node5 2185
svc_mon.csol2-node8 4410

root@csol2-node5:/var/log/contrail# neutron net-list
+--------------------------------------+-------------------------+-------------------------------------------------+
| id | name | subnets |
+--------------------------------------+-------------------------+-------------------------------------------------+
| 4c4dace9-9584-4dd8-8200-2f90cdc42740 | ip-fabric | |
| 1ca51aae-7587-4ee0-ab68-dd2539dce72b | C_VN | a8ab1252-1f7a-46a2-a35a-969320f8580e 5.1.1.0/24 |
| d8d62093-1ca3-4e5e-a511-f8868932f9bf | default-virtual-network | |
| c58c620a-3d26-4cdb-bfee-a38997faeede | __link_local__ | |
| 867f4a38-d074-43d9-b8fb-514b4e8eeb4c | P_VN1 | d65badcb-0aae-4e85-8282-8528f4308524 6.1.1.0/24 |
| 2c3a5e5f-4f7d-4d35-bc0e-ad76093225b4 | GW1 | f2af2852-7681-4474-accf-645fc191e9a8 7.1.1.0/24 |
+--------------------------------------+-------------------------+-------------------------------------------------+
root@csol2-node5:/var/log/contrail# neutron router-list
+--------------------------------------+------+-----------------------------------------------------------------------------+
| id | name | external_gateway_info |
+--------------------------------------+------+-----------------------------------------------------------------------------+
| d98369dd-efae-46d0-9a79-a21d10ec5399 | R1 | {"network_id": "2c3a5e5f-4f7d-4d35-bc0e-ad76093225b4", "enable_snat": true} |
+--------------------------------------+------+-----------------------------------------------------------------------------+

Can you please take a look at the system and help debug this? The controller node is at 10.87.26.145 (.147 and .148 are the other 2 nodes of the cluster).

The service-monitor log shows this error:

127.0.0.1 - - [02/Jun/2015 07:16:12] "GET /Snh_SandeshUVECacheReq?x=NodeStatus HTTP/1.1" 200 3630
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/gevent/greenlet.py", line 327, in run
    result = self._run(*self.args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/cfgm_common/vnc_kombu.py", line 110, in _connection_watch
    self._conn.drain_events()
  File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 279, in drain_events
    return self.transport.drain_events(self.connection, **kwargs)
  File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 90, in drain_events
    return connection.drain_events(**kwargs)
  File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 322, in drain_events
    return amqp_method(channel, args, content)
  File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 1908, in _basic_deliver
    fun(msg)
  File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 589, in _receive_callback
    return on_m(message) if on_m else self.receive(decoded, message)
  File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 558, in receive
    [callback(body, message) for callback in callbacks]
  File "/usr/lib/python2.7/dist-packages/cfgm_common/vnc_kombu.py", line 137, in _subscribe
    self._subscribe_cb(body)
  File "/usr/lib/python2.7/dist-packages/svc_monitor/svc_monitor.py", line 285, in _vnc_subscribe_callback
    self._delete_interface_route_table(irt_id)
  File "/usr/lib/python2.7/dist-packages/svc_monitor/svc_monitor.py", line 735, in _delete_interface_route_table
    self._vnc_lib.interface_route_table_delete(id=irt_uuid)
  File "/usr/lib/python2.7/dist-packages/vnc_api/gen/vnc_api_client_gen.py", line 3652, in interface_route_table_delete
    content = self._request_server(rest.OP_DELETE, uri)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 368, in _request_server
    retry_count=retry_count)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 429, in _request
    raise HttpError(status, content)
HttpError: HTTP Status: 500 Content: Internal Server Error
<Greenlet at 0x7fab683914b0: <bound method VncKombuClientV2._connection_watch of <cfgm_common.vnc_kombu.VncKombuClientV2 object at 0x7fab684ccfd0>>> failed with HttpError

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/11424
Submitter: Rudra Rugge (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/11422
Submitter: Rudra Rugge (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/11422
Committed: http://github.org/Juniper/contrail-controller/commit/d46e8f07d23d54a535ee877d49610de07611355f
Submitter: Zuul
Branch: master

commit d46e8f07d23d54a535ee877d49610de07611355f
Author: Rudra Rugge <email address hidden>
Date: Tue Jun 9 09:20:37 2015 -0700

Traceback during update of local preference

Traceback seen during update of local preference because of
service template object not read from config_db.

Handle exceptions for rabbit message gevent thread. This was
causing the greenlet to exit.

Change-Id: Ie67225e5e09932af68be20aba49c01094899e1df
Closes-Bug: #1463477
Closes-Bug: #1463091

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/11424
Committed: http://github.org/Juniper/contrail-controller/commit/e295985a3c521e6028b5585eaab2c85232c5be1d
Submitter: Zuul
Branch: R2.20

commit e295985a3c521e6028b5585eaab2c85232c5be1d
Author: Rudra Rugge <email address hidden>
Date: Tue Jun 9 09:20:37 2015 -0700

Traceback during update of local preference

Traceback seen during update of local preference because of
service template object not read from config_db.

Handle exceptions for rabbit message gevent thread. This was
causing the greenlet to exit.

Change-Id: Ie67225e5e09932af68be20aba49c01094899e1df
Closes-Bug: #1463477
Closes-Bug: #1463091