On November 9th contrail-schema failed to start in one of our clusters (contrail 2.21.3-47.el6). The problem is due to the number of network policies in the cluster.
To work around the issue we applied the following patch to /usr/lib/python2.6/site-packages/cfgm_common/zkclient.py.
def _zk_listener(self, state):
if state == KazooState.CONNECTED:
if self._election:
self._election.cancel()
# Update connection info
self._sandesh_connection_info_update(status='UP', message='')
elif state == KazooState.LOST:
# Lost the session with ZooKeeper Server
# Best of option we have is to exit the process and restart all
# over again
self._sandesh_connection_info_update(status='DOWN',
message='Connection to Zookeeper lost')
if self._lost_cb:
self._lost_cb()
else:
- os._exit(2)
+ pass
elif state == KazooState.SUSPENDED:
# Update connection info
self._sandesh_connection_info_update(status='INIT',
message = 'Connection to zookeeper lost. Retrying')
This allows contrail-schema to start but needless to say it's a very short term fix.
The problem can be recreated as follows,
1. Create 2 networks.
2. Create 2 policies with 100 rules each and attach to the networks created in step 1. (see note 1 below)
3. Stop and start the contrail-schema service. I started it interactively rather than using the service as it gave me a little more control.
/usr/bin/python /usr/bin/contrail-schema --conf_file /etc/contrail/contrail-schema.conf --conf_file /etc/contrail/contrail-keystone-auth.conf
4. Note the number of rules returned from /access-control-lists (see note 2) and the time it took contrail-schema to start (if it did actually start).
5. Repeat these steps until contrail-schema fails to start
In my testing contrail-schema failed to start when these numbers were reached,
- 263 virtual networks
- 717 network policies
- 98412 network policy rules as returned from /access-control-lists
- 2:40 time it takes contrail-schema to fail
*Note 1: Policy Details*
Each policy has 100 rules. Each rule looked like this,
source: local
source port: any
dest: 10.0.0.$i/32
dest_port: any
protocol: tcp
direction <>
action pass
$i runs from 1 to 100.
*Note 2: Counting rules*
curl -s -o acls.json -H "X-Auth-Token: $TOKEN" "http://[CONTRAIL CONTROLLER IP]:9100/access-control-lists?count=False&detail=True"
grep -o rule_uuid acls.json | wc -l
Review in progress for https:/
Submitter: <email address hidden> (<email address hidden>)