Setting of local memcached server for non-keystone services breaks some tempest tests

[Sergey Novikov]

Detailed bug description:

The commit https://github.com/openstack/fuel-library/commit/96e369880c441bdcf9ab30f196b4aabaafb9b921 sets memcached server to local one for non-keystone services. it affects some tempest test from object storage suite http://paste.openstack.org/show/599993/

Steps to reproduce:
 1. Deploy HA cluster (3 controller + 2 compute)
 2. Run the following tempest tests
- tempest.api.object_storage.test_object_formpost_negative.ObjectFormPostNegativeTest.test_post_object_using_form_expired
- tempest.api.object_storage.test_object_temp_url.ObjectTempUrlTest.test_get_object_using_temp_url_key_2
- tempest.scenario.test_object_storage_basic_ops.TestObjectStorageBasicOps.test_swift_basic_ops

Expected results:tests pass
Actual result: tests fail

Workaround:
1. Add external memcached servers in option "memcache_servers", in /etc/swift/proxy-server.conf on each controller node
for example, like this
memcache_servers = 10.168.0.7:11211,10.168.0.10:11211,10.168.0.11:11211
2. reboot the corresponding service on each controller node

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/440529

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/440529
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=f7e609e612a9fd95a2f44399315a73d744c1ef7c
Submitter: Jenkins
Branch: master

commit f7e609e612a9fd95a2f44399315a73d744c1ef7c
Author: Oleksiy Molchanov <email address hidden>
Date: Thu Mar 2 14:37:00 2017 +0200

    Revert setting local memcached server for swift proxy

    Switch back to using all available mamcached servers, because of
    failures during swift testing.

    Change-Id: I251b465dadd8ffe9a15b1a7092168d89acd2d72b
    Closes-Bug: 1666837

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/442349

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/442439

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/442439
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=54135a5d0bbc70de63678b71131dc5e773b13f5b
Submitter: Jenkins
Branch: stable/ocata

commit 54135a5d0bbc70de63678b71131dc5e773b13f5b
Author: Oleksiy Molchanov <email address hidden>
Date: Thu Mar 2 14:37:00 2017 +0200

    Revert setting local memcached server for swift proxy

    Switch back to using all available mamcached servers, because of
    failures during swift testing.

    Change-Id: I251b465dadd8ffe9a15b1a7092168d89acd2d72b
    Closes-Bug: 1666837
    (cherry picked from commit f7e609e612a9fd95a2f44399315a73d744c1ef7c)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/fuel-library 11.0.0.0rc2 release candidate.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/442349
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=ae92db53ad1344dc26026e612bb7cfa47aa4023b
Submitter: Jenkins
Branch: stable/mitaka

commit ae92db53ad1344dc26026e612bb7cfa47aa4023b
Author: Oleksiy Molchanov <email address hidden>
Date: Thu Mar 2 14:37:00 2017 +0200

    Revert setting local memcached server for swift proxy

    Switch back to using all available mamcached servers, because of
    failures during swift testing.

    Change-Id: I251b465dadd8ffe9a15b1a7092168d89acd2d72b
    Closes-Bug: 1666837
    (cherry picked from commit f7e609e612a9fd95a2f44399315a73d744c1ef7c)

[TatyanaGladysheva (tgladysheva)]

Verified on 9.2 + mu1 updates.

Actual results:
Before the fix issue is reproduced on 9.2.
After the fix three tempest tests are passed: http://paste.openstack.org/show/602657/.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/450870
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=41d4a11ee73337de2e9a5cf8fe983c6f0a114bc6
Submitter: Jenkins
Branch: stable/newton

commit 41d4a11ee73337de2e9a5cf8fe983c6f0a114bc6
Author: Vladimir Kuklin <email address hidden>
Date: Thu Jan 19 17:35:35 2017 +0300

    Set memcached server to local one for non-keyston services

    We misconfigured local cache for services with change
    https://review.openstack.org/#/q/Id1034e22d79c3ea6b25575d9bcf8e8750a02365d
    Thus, it becomes extremely slow when a controller is down.

    With this commit we revert things back to normal with local memcached
    for all openstack services leaving keystone memcached shared for tokens
    (this was thoroughly tested previously)

    This commit https://github.com/openstack/fuel-library/commit/a529033fdcb36ccea8cf0cc76339816ed31418c7
    pointed all non-keystone services to local memcached for keystone auth tokens, however it also
    pointed cache/memcache_servers in nova to local memcached. This led to regression in Nova.

    Revert setting local memcached server for swift proxy

    Switch back to using all available mamcached servers, because of
    failures during swift testing.

    Change-Id: I8f6bbf77d27f3d8976985241deb8a948984862f5
    Closes-bug: #1657727
    Closes-Bug: #1576218
    Closes-Bug: 1666837