API server errors preventing creation of routing/ service instance

[Vinod Nair]

LBAAS netns and haproxy config is not getting created. Looks like the api server error is the one preventing the ceration of routing / service instances..

api server log is as below

http://cmbu-sv02.englab.juniper.net/pxe/Standard/vin/contrail-api.log

version : 2.02 build 46 Juno 14.04

root@cs-scale-1:/opt/contrail/utils# neutron lb-vip-list
+--------------------------------------+------+----------+----------+----------------+--------+
| id | name | address | protocol | admin_state_up | status |
+--------------------------------------+------+----------+----------+----------------+--------+
| 0afa003c-585f-4aa4-963a-47afd6eb626b | VIP1 | 11.1.0.3 | HTTP | True | ACTIVE |
+--------------------------------------+------+----------+----------+----------------+--------+

root@cs-scale-1:/opt/contrail/utils# neutron port-list | grep 11.1.0.3
| 93264673-316e-4f01-a308-8eb0688ce5ef | 0afa003c-585f-4aa4-963a-47afd6eb626b | 02:93:26:46:73:31 | {"subnet_id": "624809fa-8d39-4165-ac5d-2f0c8dd275eb", "ip_address": "11.1.0.3"} |

http://10.87.141.10:8081/analytics/uves/service-instances
[]

Api.log on one of the config nodes is as below

tailf /var/log/contrail/contrail-api.log
06/09/2015 07:47:10 PM [cs-scale-1:contrail-api:Config:0]: __default__ [SYS_WARN]: VncApiError: Warning! InvalidRequestException(why='Keyspace names must be case-insensitively unique ("useragent" conflicts with "useragent")')
06/09/2015 07:47:10 PM [cs-scale-1:contrail-api:Config:0]: __default__ [SYS_WARN]: VncApiError: Warning! InvalidRequestException(why='Cannot add already existing column family "useragent_keyval_table" to keyspace "useragent"')
06/09/2015 07:47:10 PM [cs-scale-1:contrail-api:Config:0]: __default__ [SYS_WARN]: VncApiError: Warning! InvalidRequestException(why='Keyspace names must be case-insensitively unique ("config_db_uuid" conflicts with "config_db_uuid")')
06/09/2015 07:47:10 PM [cs-scale-1:contrail-api:Config:0]: __default__ [SYS_WARN]: VncApiError: Warning! InvalidRequestException(why='Cannot add already existing column family "obj_uuid_table" to keyspace "config_db_uuid"')
06/09/2015 07:47:10 PM [cs-scale-1:contrail-api:Config:0]: __default__ [SYS_WARN]: VncApiError: Warning! InvalidRequestException(why='Cannot add already existing column family "obj_fq_name_table" to keyspace "config_db_uuid"')
06/09/2015 07:47:10 PM [cs-scale-1:contrail-api:Config:0]: __default__ [SYS_NOTICE]: VncApiError: Cassandra connection ESTABLISHED
06/09/2015 07:47:10 PM [cs-scale-1:contrail-api:Config:0]: __default__ [SYS_NOTICE]: VncApiError: Connecting to zookeeper on 13.1.0.1:2181,13.1.0.2:2181,13.1.0.3:2181
06/09/2015 07:47:10 PM [cs-scale-1:contrail-api:Config:0]: __default__ [SYS_NOTICE]: VncApiError: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@13.1.0.10:5673//']
06/09/2015 07:47:10 PM [cs-scale-1:contrail-api:Config:0]: __default__ [SYS_ERR]: VncApiError: RabbitMQ connection down
06/09/2015 07:47:10 PM [cs-scale-1:contrail-api:Config:0]: __default__ [SYS_NOTICE]: VncApiError: RabbitMQ connection ESTABLISHED <Connection: amqp://guest@13.1.0.10:5673// at 0x7fb90219a150>

[Vinod Nair (vinodnair)]

When following sequence of operations were done, the rabbimq drainer greenlet died forever. The rabbitmq drainer should keep running i.e. handle catch-all Exception

1. patch oslo for some backports on all controllers
2. restart rabbitmq server
3. restart all nova services
[ contrail-api was NOT restarted]

13.1.0.2 - - [2015-06-09 17:42:49] "GET /virtual-routers?detail=True HTTP/1.1" 401 192 0.022938
ERROR:cs-scale-2:contrail-api:Config:0:__default__ [SYS_ERR]: VncApiError: RabbitMQ connection down
WARNING:cs-scale-2:contrail-api:Config:0:__default__ [SYS_NOTICE]: VncApiError: RabbitMQ connection ESTABLISHED <Connection: amqp://guest@13.1.0.10:5673// at 0x7f085e208610>
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/gevent/greenlet.py", line 327, in run
    result = self._run(*self.args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/cfgm_common/vnc_kombu.py", line 112, in _connection_watch
    self._reconnect()
  File "/usr/lib/python2.7/dist-packages/cfgm_common/vnc_kombu.py", line 99, in _reconnect
    callbacks=[self._subscribe])
  File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 357, in __init__
    self.revive(self.channel)
  File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 369, in revive
    self.declare()
  File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 379, in declare
    queue.declare()
  File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 505, in declare
    self.queue_declare(nowait, passive=False)
  File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 531, in queue_declare
    nowait=nowait)
  File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 1258, in queue_declare
    (50, 11), # Channel.queue_declare_ok
  File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 69, in wait
    return self.dispatch_method(method_sig, args, content)
  File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 87, in dispatch_method
    return amqp_method(self, args)
  File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 241, in _close
    reply_code, reply_text, (class_id, method_id), ChannelError,
NotFound: Queue.declare: (404) NOT_FOUND - home node 'rabbit@cs-scale-1-ctrl' of durable queue 'vnc_config.cs-scale-2-9100' in vhost '/' is down or inaccessible
<Greenlet at 0x7f085e2017d0: <bound method VncServerKombuClient._connection_watch of <vnc_cfg_api_server.vnc_cfg_ifmap.VncServerKombuClient object at 0x7f085f3ac490>>> failed with NotFound

WARNING:keystoneclient.middleware.auth_token:Unable to find authentication token in headers
13.1.0.3 - - [2015-06-09 17:46:04] "GET / HTTP/1.1" 401 192 0.000700
13.1.0.3 - - [2015-06-09 17:46:04] "GET / HTTP/1.1" 200 11310 0.017901
13.1.0.3 - - [2015-06-09 17:46:04] "GET /physical-routers?count=False&detail=False HTTP/1.1" 200 132 0.002907
WARNING:keystoneclient.middleware.auth_token:Unable to find authentication token in headers
13.1.0.2 - - [2015-06-09 17:46:07] "GET / HTTP/1.1" 401 192 0.000702
13.1.0.2 - - [2015-06-09 17:46:08] "GET / HTTP/1.1" 200 11310 0.214082

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/11594
Submitter: Sachin Bansal (<email address hidden>)