Detailed bug description:
stick on src is a cause of slow performance of kystone. We performed rally scenario which boots an instance with secgroup with 10 rules and a cinder volume, then migrate the instance. (http://paste.openstack.org/show/497202/)
When the option persists in /etc/haproxy/conf.d/020-keystone-1.cfg and /etc/haproxy/conf.d/030-keystone-2.cfg files (during rally test):
openstack user list - real 0m27.563s
nova list --all-tenant - real 0m47.577s
root@node-179:~# for i in 77 96 179; do grep keystone-2 /var/log/haproxy.log | grep -c node-$i; done
1124
313
2283
When I commented out two lines in the files (during rally test):
#stick on src
#stick-table type ip size 200k expire 2m
openstack user list - real 0m1.704s
nova list --all-tenant - real 0m6.011s
Round Robin works well
Steps to reproduce:
1. Fuel 9.0-308 has been deployed
2. curl -s 'https://review.openstack.org/gitweb?p=openstack/fuel-web.git;a=patch;h=6106dfa026b042dac26ed77354321115b78aae5b' | patch -b -d /usr/lib/python2.7/site-packages -p2
rpm -Uvh epel-release-latest-7.noarch.rpm
sed -i s/^enabled=1/enabled=0/g /etc/yum.repos.d/epel.repo
yum --enablerepo=epel install uwsgi uwsgi-plugin-python python-uwsgidecorator
service receiverd restart && service nailgun restart
due https://bugs.launchpad.net/fuel/+bug/1569859 and https://bugs.launchpad.net/fuel/+bug/1570509
fuel-agent and fuel-library have been updated: http://paste.openstack.org/show/496825/ due https://bugs.launchpad.net/fuel/+bug/1543233 and https://bugs.launchpad.net/fuel/+bug/1574999
3. patch has been applied to keep rotated logs http://paste.openstack.org/show/495857/
4. patch has been applied to increase rsyslog chunks to 4к http://paste.openstack.org/show/496901/
5. LMA, ElasticSearch, Grafana plugins have been installed:
yum -y install createrepo rpm rpm-build dpkg-devel git
easy_install pip
pip install fuel-plugin-builder
git clone https://github.com/openstack/fuel-plugin-lma-collector.git
fpb --check ./fuel-plugin-lma-collector
fpb --build ./fuel-plugin-lma-collector
fuel plugins --install ./fuel-plugin-lma-collector/*.noarch.rpm
git clone https://github.com/openstack/fuel-plugin-elasticsearch-kibana.git
fpb --check ./fuel-plugin-elasticsearch-kibana
fpb --build ./fuel-plugin-elasticsearch-kibana
fuel plugins --install ./fuel-plugin-elasticsearch-kibana/*.noarch.rpm
git clone https://github.com/openstack/fuel-plugin-influxdb-grafana.git
fpb --check ./fuel-plugin-influxdb-grafana
fpb --build ./fuel-plugin-influxdb-grafana
fuel plugins --install ./fuel-plugin-influxdb-grafana/*.noarch.rpm
patch -b -d /var/www/nailgun/plugins/lma_collector-0.10/ -p1 < lma.patch (http://paste.openstack.org/show/495328/)
fuel plugins --sync
6. deploy cluster 3 controllers, 20 computes+Ceph, 172 computes, vxlan+DVR, Ceph for all
7. Perform rally tests and execute openstack user list command during the testing
Expected results:
The command shows results in 5 seconds
Actual result:
The command shows results in ~50 seconds (timeout 60)
Reproducibility:
Each time
Workaround:
Comment out lines if /etc/haproxy/conf.d/020-keystone-1.cfg and /etc/haproxy/conf.d/030-keystone-2.cfg files:
#stick on src
#stick-table type ip size 200k expire 2m
Impact:
OpenStack cluster performance
Description of the environment:
Operation system: ubuntu
Versions of components: MOS 9.0
Reference architecture: 3 controllers, 20 computes+Ceph, 172 computes, vxlan+DVR, Ceph for all
Network model: vxlan+DVR
Related projects installed: LMA
Additional information:
-
Fix proposed to branch: master
Review: https:/