schema.err (traceback) & contrail-schema in initializing (Database:Cassandra connection down)

[Vijay Anand]

Build:
=====
Contrail version: 2.21(102) + duplicate instances patch

Test:
====
60 parallel instances (Left networks are unique, mgmt and right are shared so the policy is unique)

stack-list
=======
root@csp-sol-mustang:~# heat stack-list
ERROR: Timeout while waiting on RPC response - topic: "engine", RPC method: "list_stacks" info: "<unknown>"
root@csp-sol-mustang:~#
root@csp-sol-mustang:~# heat stack-list
ERROR: Timeout while waiting on RPC response - topic: "engine", RPC method: "list_stacks" info: "<unknown>"

Controller - CPU/memory utilization:
=============================
root@csp-sol-mustang:~# top

top - 11:58:13 up 1 day, 2:58, 1 user, load average: 40.64, 34.80, 21.97
Tasks: 618 total, 4 running, 612 sleeping, 0 stopped, 2 zombie
%Cpu(s): 41.7 us, 20.3 sy, 0.0 ni, 13.4 id, 24.3 wa, 0.0 hi, 0.3 si, 0.0 st
KiB Mem: 65704008 total, 65446328 used, 257680 free, 130356 buffers
KiB Swap: 67096572 total, 19243656 used, 47852916 free. 2720012 cached Mem

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
 2373 root 20 0 23.767g 0.010t 891388 S 140.6 16.3 210:51.29 java
 2380 contrail 20 0 1266116 415996 5844 S 117.1 0.6 167:14.31 contrail-collec
 1815 libvirt+ 20 0 4759956 2.976g 12664 S 95.6 4.7 1:05.15 qemu-system-x86
 4461 libvirt+ 20 0 4759956 31568 12592 S 77.1 0.0 0:05.59 qemu-system-x86
 2679 libvirt+ 20 0 4759956 40272 12668 S 73.1 0.1 0:52.62 qemu-system-x86
24433 libvirt+ 20 0 6055648 2.831g 12700 S 63.2 4.5 2:29.46 qemu-system-x86
 2369 root 20 0 697492 134448 5944 S 44.0 0.2 183:02.62 contrail-api
 2462 keystone 20 0 236928 60408 5100 R 34.1 0.1 135:11.45 keystone-all
 2363 contrail 20 0 1359916 62600 8272 S 22.2 0.1 19:13.30 contrail-contro
17726 libvirt+ 20 0 6760500 2.268g 12712 S 19.9 3.6 2:23.82 qemu-system-x86

Schema.err (traceback):
==================

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/schema_transformer/to_bgp.py", line 3749, in launch_arc
    transformer.process_poll_result(result)
  File "/usr/lib/python2.7/dist-packages/schema_transformer/to_bgp.py", line 3384, in process_poll_result
    prule, dynamic)
  File "/usr/lib/python2.7/dist-packages/schema_transformer/to_bgp.py", line 1088, in policy_to_acl_rule
    service_list)
  File "/usr/lib/python2.7/dist-packages/schema_transformer/to_bgp.py", line 465, in add_service_chain
    service_chain.destroy()
  File "/usr/lib/python2.7/dist-packages/schema_transformer/to_bgp.py", line 1951, in destroy
    vn2_obj.delete_routing_instance(service_ri2)
  File "/usr/lib/python2.7/dist-packages/schema_transformer/to_bgp.py", line 678, in delete_routing_instance
    ri.delete(self)
  File "/usr/lib/python2.7/dist-packages/schema_transformer/to_bgp.py", line 1547, in delete
    rt_cf.remove(ri_fq_name_str)
  File "/usr/lib/python2.7/dist-packages/schema_transformer/to_bgp.py", line 3560, in wrapper
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/pycassa/columnfamily.py", line 1090, in remove
    batch.send()
  File "/usr/lib/python2.7/dist-packages/schema_transformer/to_bgp.py", line 3560, in wrapper
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/pycassa/batch.py", line 146, in send
    allow_retries=self.allow_retries)
  File "/usr/lib/python2.7/dist-packages/pycassa/pool.py", line 153, in new_f
    return new_f(self, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/pycassa/pool.py", line 153, in new_f
    return new_f(self, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/pycassa/pool.py", line 153, in new_f
    return new_f(self, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/pycassa/pool.py", line 125, in new_f
    self._pool._replace_wrapper() # puts a new wrapper in the queue
  File "/usr/lib/python2.7/dist-packages/pycassa/pool.py", line 458, in _replace_wrapper
    conn = self._create_connection()
  File "/usr/lib/python2.7/dist-packages/pycassa/pool.py", line 431, in _create_connection
    (exc.__class__.__name__, exc))
AllServersUnavailable: An attempt was made to connect to each of the servers twice, but none of the attempts succeeded. The last failure was timeout: timed out

Contrail-schema in initializing state :
=============================
root@csp-sol-mustang:~# contrail-status
== Contrail vRouter ==
supervisor-vrouter: active
contrail-vrouter-agent active
contrail-vrouter-nodemgr active

== Contrail Control ==
supervisor-control: active
contrail-control active
contrail-control-nodemgr active
contrail-dns active
contrail-named active

== Contrail Analytics ==
supervisor-analytics: active
contrail-analytics-api active
contrail-analytics-nodemgr active
contrail-collector active
contrail-query-engine active
contrail-snmp-collector active
contrail-topology active

== Contrail Config ==
supervisor-config: active
contrail-api:0 active
contrail-config-nodemgr active
contrail-device-manager active
contrail-discovery:0 active
contrail-schema initializing (Database:Cassandra connection down) <<<<<<<<<
contrail-svc-monitor active
ifmap active

== Contrail Web UI ==
supervisor-webui: active
contrail-webui active
contrail-webui-middleware active

== Contrail Database ==
supervisor-database: active
contrail-database active
contrail-database-nodemgr active

== Contrail Support Services ==
supervisor-support-service: active
rabbitmq-server active

[Sachin Bansal (sbansal)]

I think this should have been fixed in 3.0 or later. I will mark it as fixed. If not, please reopen.