Policy Engine and API are locked for a long time

[Pierre Ettori]

If Congress is inserting a large dataset, all API access and policy engine rule are locked till the insertion is finished which can take mn depending on the volume of data to insert

[Alexander Yip (ayip)]

Pierre, the messages that look like this:

2014-11-16 08:42:48.557 15185 DEBUG congress.policy.runtime [-] Clas : Insert: ceilometer:stats("cpu_util", "4a1ab506-b71b-4372-8dcd-97cf0da5bc61", 0.0445943716575, 79, 47407.0, "2014-11-16T09:52:15", "2014-11-15T20:42:08", 0.0500834724541, 0.0382695507488, 0, "2014-11-16T09:52:15", "2014-11-16T09:52:15", 3.52295536094, "%") log /var/lib/congress/local/lib/python2.7/site-packages/congress/policy/runtime.py:45

occur when you insert a rule that depends on ceilometer:stats. Inserting the rule causes the runtime to subscribe to a new table:

2014-11-16 08:42:35.301 15185 DEBUG congress.dse.deepsix [-] engine:: Subscribing to new (service, table): (ceilometer, stats) log_debug /var/lib/congress/local/lib/python2.7/site-packages/congress/dse/deepsix.py:567

There are many of these messages, so I think this is what's happening during the lockout period. I suspect that the eventlet thread does not do I/O in this loop, and therefore the eventlet thread does not yield to other threads.

[Tim Hinrichs (thinrichs)]

We addressed this to some extent by speeding up the operations that cause locking. And we're adding support for high-availability to mitigate the effects. But there's still more to do here.