Enable KS token caching by default to avoid performance problems

[Roman Rufanov]

Detailed bug description: without KS token caching Horizon performance is bad (10-8 sec for Login or Network)
Steps to reproduce: install MOS 8 by default, login into Horizon, click on Network screen
Expected results: Login or Network should take under 2 sec
Actual result: Login screen=10 sec, Network screen=8 sec

Impact:
Enable KS token caching by default to avoid performance problems.
Without memcached servers KS performance on Horizon side is horrible.
With memcached it improved by factor of 2.

Workaround and Proposed changes:
What config files were changed? Nova.conf, glance-api.conf, neutron.conf, cinder.conf on OpenStack controllers.
How it was changed: for i in /etc/nova/nova.conf /etc/neutron/neutron.conf /etc/cinder/cinder.conf /etc/glance/glance-api.conf ; do sed -i '/\[keystone_authtoken]/a memcached_servers=192.168.210.75:11211,192.168.210.77:11211,192.168.210.78:11211' $i ; done

What was restarted: for service in nova-api neutron-server cinder-api glance-api; do restart $service; done

Applied, tested and helped for MOS 8 Customer

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/336139

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/338579

[Alex Schultz (alex-schultz)]

Not sure if 7.0 supports the keystone_authtoken/memcached_server options so it will need to be confirmed.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/336139
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=df416f53d2dcb21c700802f5c2602d19f6d6c70c
Submitter: Jenkins
Branch: master

commit df416f53d2dcb21c700802f5c2602d19f6d6c70c
Author: Alex Schultz <email address hidden>
Date: Thu Jun 30 11:48:41 2016 -0600

    Use memcache for keystone_authtoken

    This change updates where we calculate the memcache server information
    to the globals hiera data and updates all the openstack services to
    use the heira memcached_servers key for consistency. Additionally all
    api services have been updated to configure the memcached servers for
    the keystone_authtoken middleware configuration to improve the
    responsiveness of the service.

    Change-Id: Id1034e22d79c3ea6b25575d9bcf8e8750a02365d
    Closes-Bug: #1597512

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/344240

[Rodion Tikunov (rtikunov)]

For verification:
run on controller nodes
# sed -n -e "/^\[keystone_authtoken\]/,/^\s*\[/{/^memcached_server.*\=.*/p;}" /etc/nova/nova.conf /etc/neutron/neutron.conf /etc/cinder/cinder.conf /etc/glance/glance-api.conf
memcached_servers = 192.168.0.4:11211,192.168.0.7:11211,192.168.0.8:11211
memcached_servers=192.168.0.4:11211,192.168.0.7:11211,192.168.0.8:11211
memcached_servers = 192.168.0.4:11211,192.168.0.7:11211,192.168.0.8:11211
memcached_servers = 192.168.0.4:11211,192.168.0.7:11211,192.168.0.8:11211

[Rodion Tikunov (rtikunov)]

Milestone changed because fix is disputable for 8.0 and should be tested in environment with one node is down

[Boris Bobrov (bbobrov)]

I suspect this will lead to terrible performance degradation if a node goes down. Are there tests that would show "before" and "after" performance if 1 node is down?

I strongly advise against making this change to any env.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Alex Schultz (<email address hidden>) on branch: stable/8.0
Review: https://review.openstack.org/338579
Reason: needs a different solution as this one is not optimal

[OpenStack Infra (hudson-openstack)]

Change abandoned by Rodion Tikunov (<email address hidden>) on branch: stable/7.0
Review: https://review.openstack.org/344240
Reason: Cherry picked commit was abandoned

[Roman Rufanov (rrufanov)]

Ok - if that solution does not work - please propose another. Customer's env was not usable until this was enabled. We need to have this perform reasonable fast in default configuration. "node down" even is 1% when it becomes slow while "non-cashing and hence slow" is 99% which is very visible to a customers and they do not accept it. Please review and revise.

[Dmitry Pyzhov (dpyzhov)]

Let's ask QA team to carefully test the fix before merge it in 9.2

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/fuel-library 10.0.0rc1 release candidate.

[Roman Rufanov (rrufanov)]

Can 10x fix be packported to 9.2, 8 and 7? If not - what is proposed solution?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/395486

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/fuel-library 10.0.0 release.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Vladimir Kozhukalov (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/395486
Reason: Can not be backported to mitaka, since there are no necessary parameters in upstream modules for those versions that are used with mitaka version of fuel-library (e.g. ceilometer 8.2.0)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/395486
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=6d5911a30bfd950d0e55de7212850948cfc67dc2
Submitter: Jenkins
Branch: stable/mitaka

commit 6d5911a30bfd950d0e55de7212850948cfc67dc2
Author: Sergey Abramov <email address hidden>
Date: Wed Nov 9 12:43:30 2016 +0300

    Use memcache for keystone_authtoken

    This change updates where we calculate the memcache server information
    to the globals hiera data and updates all the openstack services to
    use the heira memcached_servers key for consistency. Additionally all
    api services have been updated to configure the memcached servers for
    the keystone_authtoken middleware configuration to improve the
    responsiveness of the service.

    Change-Id: Id1034e22d79c3ea6b25575d9bcf8e8750a02365d
    Closes-Bug: #1597512
    (cherry picked from commit df416f53d2dcb21c700802f5c2602d19f6d6c70c)

[Ekaterina Shutova (eshutova)]

verified on 9.2 snapshot #682:
cat /etc/fuel_build_id:
 495
cat /etc/fuel_build_number:
 495
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 mitaka-9.0
rpm -qa | egrep 'fuel|astute|network-checker|nailgun|packetary|shotgun':
 fuel-nailgun-9.0.0-1.mos8931.noarch
 network-checker-9.0.0-1.mos77.x86_64
 fuel-ostf-9.0.0-1.mos947.noarch
 fuel-agent-9.0.0-1.mos291.noarch
 python-packetary-9.0.0-1.mos161.noarch
 nailgun-mcagents-9.0.0-1.mos784.noarch
 fuel-setup-9.0.0-1.mos6359.noarch
 shotgun-9.0.0-1.mos90.noarch
 python-fuelclient-9.0.0-1.mos364.noarch
 fuelmenu-9.0.0-1.mos276.noarch
 fuel-provisioning-scripts-9.0.0-1.mos8931.noarch
 fuel-mirror-9.0.0-1.mos161.noarch
 fuel-notify-9.0.0-1.mos8685.noarch
 rubygem-astute-9.0.0-1.mos784.noarch
 fuel-release-9.0.0-1.mos6359.noarch
 fuel-misc-9.0.0-1.mos8685.noarch
 fuel-openstack-metadata-9.0.0-1.mos8931.noarch
 fuel-migrate-9.0.0-1.mos8685.noarch
 fuel-library9.0-9.0.0-1.mos8685.noarch
 fuel-bootstrap-cli-9.0.0-1.mos291.noarch
 fuel-ui-9.0.0-1.mos2854.noarch
 fuel-utils-9.0.0-1.mos8685.noarch
 fuel-9.0.0-1.mos6359.noarch

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/427733

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/427733
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=812b15bccf2d49714bcc098f38a4dde13d37890e
Submitter: Jenkins
Branch: stable/7.0

commit 812b15bccf2d49714bcc098f38a4dde13d37890e
Author: Alex Schultz <email address hidden>
Date: Wed Jul 6 16:15:15 2016 -0600

    Use memcache for keystone_authtoken

    That allow to cache keyston authtoken in local instance of memcached.
    Commit adds new global variable local_memcached_server and force all
    capable services use it. That should improve speed of operations in
    that services.
    Initial job has been started in LP#1597512. But that commit also
    include improvements from #1657727.

    Closes-Bug: #1597512
    Closes-bug: #1657727

    Change-Id: I6004a8366ddc639feb1aed55b6dfbaf626f82839

[Ekaterina Shutova (eshutova)]

Verified on mos 7.0 + mu7 updates.
Login takes ~4sec, tabs switch takes ~2sec.

[Ekaterina Shutova (eshutova)]

Verified on 10.0 #1578.
Login takes ~4sec, tabs switch takes ~4sec.
On controller:
root@node-3:~# sed -n -e "/^\[keystone_authtoken\]/,/^\s*\[/{/^memcached_server.*\=.*/p;}" /etc/nova/nova.conf /etc/neutron/neutron.conf /etc/cinder/cinder.conf /etc/glance/glance-api.conf
memcached_servers=10.109.11.4:11211
memcached_servers=10.109.11.4:11211
memcached_servers = 10.109.11.4:11211
memcached_servers = 10.109.11.4:11211