Comment 9 for bug 1498867

Revision history for this message
Dmitry Sutyagin (dsutyagin) wrote :

Looks like not fixed for 9.0, will double-check with the original scenario. Currently I'm doing tests on 1 controller out of 3 with no-quorum-policy=ignore to allow all pcs/crm resources to work. Everything works, but is significantly (> 10x) slower compared to when all controllers are running. Tried removing offline rabbit hosts from nova.conf -didn't help. Logs show the following timestamps for a given request:

[dsutyagin@dsutyagin-local var]$ grep req-33ba0db0-6072-429e-9245-9fc20462e260 log/nova/nova-api.log
2016-09-07 07:34:46.839 32065 DEBUG nova.api.openstack.wsgi [req-33ba0db0-6072-429e-9245-9fc20462e260 b30092137b424eeebb34d6e06b669f74 4ebb2779094a406db97e552a8d0f3ae2 - - -] Calling method '<bound method ServersController.detail of <nova.api.openstack.compute.servers.ServersController object at 0x7f6e7e163190>>' _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:699
2016-09-07 07:34:46.840 32065 DEBUG nova.compute.api [req-33ba0db0-6072-429e-9245-9fc20462e260 b30092137b424eeebb34d6e06b669f74 4ebb2779094a406db97e552a8d0f3ae2 - - -] Searching by: {'deleted': False} get_all /usr/lib/python2.7/dist-packages/nova/compute/api.py:2062
2016-09-07 07:34:53.244 32065 DEBUG oslo.cache._memcache_pool [req-33ba0db0-6072-429e-9245-9fc20462e260 b30092137b424eeebb34d6e06b669f74 4ebb2779094a406db97e552a8d0f3ae2 - - -] Memcached pool 140112431041168, thread 140112787171136: Marked host 10.20.124.196:11211 dead until 1473233993.09 _do_log /usr/lib/python2.7/dist-packages/oslo_cache/_memcache_pool.py:116
2016-09-07 07:34:53.245 32065 DEBUG oslo.cache._memcache_pool [req-33ba0db0-6072-429e-9245-9fc20462e260 b30092137b424eeebb34d6e06b669f74 4ebb2779094a406db97e552a8d0f3ae2 - - -] Memcached pool 140112431041168, thread 140112787171136: Marked host 10.20.124.195:11211 dead until 1473233990.09 _do_log /usr/lib/python2.7/dist-packages/oslo_cache/_memcache_pool.py:116
2016-09-07 07:34:53.347 32065 DEBUG nova.policy [req-33ba0db0-6072-429e-9245-9fc20462e260 b30092137b424eeebb34d6e06b669f74 4ebb2779094a406db97e552a8d0f3ae2 - - -] Policy check for os_compute_api:os-hide-server-addresses failed with credentials ...

There are other issues which slow down stuff as well - call from nova to neutron takes 5 seconds, Keystone is also slow, I'll look at these too.
But in this log section I see that nova does get_all and then in 7 seconds says about memcached, nothing in between with this req id.