Looks like not fixed for 9.0, will double-check with the original scenario. Currently I'm doing tests on 1 controller out of 3 with no-quorum-policy=ignore to allow all pcs/crm resources to work. Everything works, but is significantly (> 10x) slower compared to when all controllers are running. Tried removing offline rabbit hosts from nova.conf -didn't help. Logs show the following timestamps for a given request:
There are other issues which slow down stuff as well - call from nova to neutron takes 5 seconds, Keystone is also slow, I'll look at these too.
But in this log section I see that nova does get_all and then in 7 seconds says about memcached, nothing in between with this req id.
Looks like not fixed for 9.0, will double-check with the original scenario. Currently I'm doing tests on 1 controller out of 3 with no-quorum- policy= ignore to allow all pcs/crm resources to work. Everything works, but is significantly (> 10x) slower compared to when all controllers are running. Tried removing offline rabbit hosts from nova.conf -didn't help. Logs show the following timestamps for a given request:
[dsutyagin@ dsutyagin- local var]$ grep req-33ba0db0- 6072-429e- 9245-9fc20462e2 60 log/nova/ nova-api. log openstack. wsgi [req-33ba0db0- 6072-429e- 9245-9fc20462e2 60 b30092137b424ee ebb34d6e06b669f 74 4ebb2779094a406 db97e552a8d0f3a e2 - - -] Calling method '<bound method ServersControll er.detail of <nova.api. openstack. compute. servers. ServersControll er object at 0x7f6e7e163190>>' _process_stack /usr/lib/ python2. 7/dist- packages/ nova/api/ openstack/ wsgi.py: 699 6072-429e- 9245-9fc20462e2 60 b30092137b424ee ebb34d6e06b669f 74 4ebb2779094a406 db97e552a8d0f3a e2 - - -] Searching by: {'deleted': False} get_all /usr/lib/ python2. 7/dist- packages/ nova/compute/ api.py: 2062 _memcache_ pool [req-33ba0db0- 6072-429e- 9245-9fc20462e2 60 b30092137b424ee ebb34d6e06b669f 74 4ebb2779094a406 db97e552a8d0f3a e2 - - -] Memcached pool 140112431041168, thread 140112787171136: Marked host 10.20.124.196:11211 dead until 1473233993.09 _do_log /usr/lib/ python2. 7/dist- packages/ oslo_cache/ _memcache_ pool.py: 116 _memcache_ pool [req-33ba0db0- 6072-429e- 9245-9fc20462e2 60 b30092137b424ee ebb34d6e06b669f 74 4ebb2779094a406 db97e552a8d0f3a e2 - - -] Memcached pool 140112431041168, thread 140112787171136: Marked host 10.20.124.195:11211 dead until 1473233990.09 _do_log /usr/lib/ python2. 7/dist- packages/ oslo_cache/ _memcache_ pool.py: 116 6072-429e- 9245-9fc20462e2 60 b30092137b424ee ebb34d6e06b669f 74 4ebb2779094a406 db97e552a8d0f3a e2 - - -] Policy check for os_compute_ api:os- hide-server- addresses failed with credentials ...
2016-09-07 07:34:46.839 32065 DEBUG nova.api.
2016-09-07 07:34:46.840 32065 DEBUG nova.compute.api [req-33ba0db0-
2016-09-07 07:34:53.244 32065 DEBUG oslo.cache.
2016-09-07 07:34:53.245 32065 DEBUG oslo.cache.
2016-09-07 07:34:53.347 32065 DEBUG nova.policy [req-33ba0db0-
There are other issues which slow down stuff as well - call from nova to neutron takes 5 seconds, Keystone is also slow, I'll look at these too.
But in this log section I see that nova does get_all and then in 7 seconds says about memcached, nothing in between with this req id.