Mirantis OpenStack

Keystone stops processing HTTP requests

Bug #1400326 reported by Ilya Shakhat
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Fix Committed
High
Alexander Makarov
Mirantis OpenStack 6.0

Bug Description

At some point Keystone stops processing HTTP requests and HAProxy moves the server out of pool. Once Keystones on all controllers stuck HAProxy starts to respond with HTTP 503 error.

Symptoms:
~~~~~~~~~
HAProxy log:
2014-12-08T11:01:06.135849+00:00 info: 192.168.0.23:34984 [08/Dec/2014:11:01:06.135] keystone-2 keystone-2/<NOSRV> 0/-1/-1/-1/0 503 212 - - SC-- 366/0/0/0/0 0/0 "POST /v2.0/tokens HTTP/1.1"

Keystone keeps accepting TCP connection, but hangs at HTTP level (meaning that telnet works, but curl hangs)

root@node-21:/var/log/keystone# strace -p 26330
Process 26330 attached - interrupt to quit
futex(0xe50420, FUTEX_WAIT_PRIVATE, 0, NULL

The last logged message from stuck process:
2014-12-05 17:01:20.749 26330 DEBUG keystone.common.cache._memcache_pool [-] Memcached pool 39391504, thread 139781728069376: Acquiring connection _debug_logger /usr/lib/python2.7/dist-packages/keystone/common/cache/_memcache_pool.py:91

Version:
~~~~~~
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.0"
  api: "1.0"
  build_number: "36"
  build_id: "2014-12-03_11-32-25"
  astute_sha: "16b252d93be6aaa73030b8100cf8c5ca6a970a91"
  fuellib_sha: "1eb704034c31a7679c6cfbf13579219c7da75e4b"
  ostf_sha: "7e79964ddb5092fc4568c6fb08a348bb326df2a8"
  nailgun_sha: "d2e732c5f54e35d0ed19f9a17489608dc1d11be8"
  fuelmain_sha: "7ab330b4958ab20955372e85de05e8732e8f6df2"

Tags: keystone scale
Ilya Shakhat (shakhat)
tags: added: keystone
Changed in mos:
importance: Undecided → Critical
Alexander Ignatov (aignatov)
tags: added: scale
Ilya Shakhat (shakhat)
Changed in mos:
assignee: nobody → MOS Keystone (mos-keystone)
Revision history for this message
Alexander Makarov (amakarov) wrote :

The reason seems to be memcached pool depletion:
it's build atop of python queue.Queue class, which locks thread if something tries to extract element from empty queue.

According to the log provided this is a possible case.

Changed in mos:
assignee: MOS Keystone (mos-keystone) → Alexander Makarov (amakarov)
Revision history for this message
Alexander Makarov (amakarov) wrote :

Update: depletion cannot cause the lock since in this case a new connection is created.
However, this new connection is returned to the pool afterwards.
Consequently there are +1 connection.
While there is a depletion check in the poll on the connection aquire, there is no such check on the release.

All this results in indefinite lock on connection release, if connection excess is high enough.

Revision history for this message
Alexander Makarov (amakarov) wrote :

simple way: set memcache_pool_maxsize to something very big

Alexander Makarov (amakarov)
Changed in mos:
milestone: none → 6.0
importance: Critical → High
status: New → In Progress
Revision history for this message
OSCI Robot (oscirobot) wrote :

RPM package keystone has been built for project openstack/keystone
Package version == 2014.2, package release == fuel6.0.mira10.git.bd14792.96eb21d

Changeset: https://review.fuel-infra.org/1281
project: openstack/keystone
branch: openstack-ci/fuel-6.0/2014.2
author: Alexander Makarov
committer: Alexander Makarov
subject: Memcache connection pool excess check
status: patchset-created

Files placed on repository:
openstack-keystone-2014.2-fuel6.0.mira10.git.bd14792.96eb21d.noarch.rpm
openstack-keystone-doc-2014.2-fuel6.0.mira10.git.bd14792.96eb21d.noarch.rpm
python-keystone-2014.2-fuel6.0.mira10.git.bd14792.96eb21d.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0-stable-1281/centos

Revision history for this message
OSCI Robot (oscirobot) wrote :

DEB package keystone has been built for project openstack/keystone
Package version == 2014.2, package release == fuel6.0~mira10+git.bd14792.96eb21d

Changeset: https://review.fuel-infra.org/1281
project: openstack/keystone
branch: openstack-ci/fuel-6.0/2014.2
author: Alexander Makarov
committer: Alexander Makarov
subject: Memcache connection pool excess check
status: patchset-created

Files placed on repository:
keystone-doc_2014.2-fuel6.0~mira10+git.bd14792.96eb21d_all.deb
keystone_2014.2-fuel6.0~mira10+git.bd14792.96eb21d_all.deb
python-keystone_2014.2-fuel6.0~mira10+git.bd14792.96eb21d_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0-stable-1281/ubuntu

Revision history for this message
OSCI Robot (oscirobot) wrote :

RPM package keystone has been built for project openstack/keystone
Package version == 2014.2, package release == fuel6.0.mira10.git.17464d9.96eb21d

Changeset: https://review.fuel-infra.org/1281
project: openstack/keystone
branch: openstack-ci/fuel-6.0/2014.2
author: Alexander Makarov
committer: Alexander Makarov
subject: Memcache connection pool excess check
status: patchset-created

Files placed on repository:
openstack-keystone-2014.2-fuel6.0.mira10.git.17464d9.96eb21d.noarch.rpm
openstack-keystone-doc-2014.2-fuel6.0.mira10.git.17464d9.96eb21d.noarch.rpm
python-keystone-2014.2-fuel6.0.mira10.git.17464d9.96eb21d.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0-stable-1281/centos

Revision history for this message
OSCI Robot (oscirobot) wrote :

DEB package keystone has been built for project openstack/keystone
Package version == 2014.2, package release == fuel6.0~mira10+git.17464d9.96eb21d

Changeset: https://review.fuel-infra.org/1281
project: openstack/keystone
branch: openstack-ci/fuel-6.0/2014.2
author: Alexander Makarov
committer: Alexander Makarov
subject: Memcache connection pool excess check
status: patchset-created

Files placed on repository:
keystone-doc_2014.2-fuel6.0~mira10+git.17464d9.96eb21d_all.deb
keystone_2014.2-fuel6.0~mira10+git.17464d9.96eb21d_all.deb
python-keystone_2014.2-fuel6.0~mira10+git.17464d9.96eb21d_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0-stable-1281/ubuntu

Revision history for this message
OSCI Robot (oscirobot) wrote :

RPM package keystone has been built for project openstack/keystone
Package version == 2014.2, package release == fuel6.0.mira10

Changeset: https://review.fuel-infra.org/1281
project: openstack/keystone
branch: openstack-ci/fuel-6.0/2014.2
author: Alexander Makarov
committer: Alexander Makarov
subject: Memcache connection pool excess check
status: change-merged

Files placed on repository:
openstack-keystone-2014.2-fuel6.0.mira10.noarch.rpm
openstack-keystone-doc-2014.2-fuel6.0.mira10.noarch.rpm
python-keystone-2014.2-fuel6.0.mira10.noarch.rpm

Changeset merged. Package placed on primary repository
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0-stable/centos

Revision history for this message
OSCI Robot (oscirobot) wrote :

DEB package keystone has been built for project openstack/keystone
Package version == 2014.2, package release == fuel6.0~mira10

Changeset: https://review.fuel-infra.org/1281
project: openstack/keystone
branch: openstack-ci/fuel-6.0/2014.2
author: Alexander Makarov
committer: Alexander Makarov
subject: Memcache connection pool excess check
status: change-merged

Files placed on repository:
keystone-doc_2014.2-fuel6.0~mira10_all.deb
keystone_2014.2-fuel6.0~mira10_all.deb
python-keystone_2014.2-fuel6.0~mira10_all.deb

Changeset merged. Package placed on primary repository
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0-stable/ubuntu

Alexander Makarov (amakarov)
Changed in mos:
status: In Progress → Fix Committed
Revision history for this message
OSCI Robot (oscirobot) wrote :

package keystone has been built for project openstack/keystone
Package version == 2014.2, package release == fuel6.1.mira13.git.bb829fb.a443a6c

Changeset: https://review.fuel-infra.org/4500
project: openstack/keystone
branch: openstack-ci/fuel-6.1/2014.2
author: Alexander Makarov
committer: Alexander Makarov
subject: Memcache connection pool excess check backport
status: patchset-created

Files placed on repository:

NOTE: Changeset is not merged, created temporary package repository.
 repository URL: /centos-fuel-6.1-stable-4500/

Revision history for this message
OSCI Robot (oscirobot) wrote :

package keystone has been built for project openstack/keystone
Package version == 2014.2, package release == fuel6.1~mira1+git.bb829fb.a443a6c

Changeset: https://review.fuel-infra.org/4500
project: openstack/keystone
branch: openstack-ci/fuel-6.1/2014.2
author: Alexander Makarov
committer: Alexander Makarov
subject: Memcache connection pool excess check backport
status: patchset-created

Files placed on repository:

NOTE: Changeset is not merged, created temporary package repository.
 repository URL: /trusty-fuel-6.1-stable-4500/

Revision history for this message
OSCI Robot (oscirobot) wrote :

RPM package keystone has been built for project openstack/keystone
Package version == 2014.2, package release == fuel6.0.mira11

Changeset: https://review.fuel-infra.org/5105
project: openstack/keystone
branch: openstack-ci/fuel-6.0-updates/2014.2
author: Alexander Makarov
committer: Alexander Makarov
subject: Correct initialization order for logging to use eventlet locks
status: change-merged

Files placed on repository:
openstack-keystone-2014.2-fuel6.0.mira11.noarch.rpm
openstack-keystone-doc-2014.2-fuel6.0.mira11.noarch.rpm
python-keystone-2014.2-fuel6.0.mira11.noarch.rpm

Changeset merged. Package placed on primary repository
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0-updates-stable/centos

Revision history for this message
OSCI Robot (oscirobot) wrote :

DEB package keystone has been built for project openstack/keystone
Package version == 2014.2, package release == fuel6.0~mira11

Changeset: https://review.fuel-infra.org/5105
project: openstack/keystone
branch: openstack-ci/fuel-6.0-updates/2014.2
author: Alexander Makarov
committer: Alexander Makarov
subject: Correct initialization order for logging to use eventlet locks
status: change-merged

Files placed on repository:
keystone-doc_2014.2-fuel6.0~mira11_all.deb
keystone_2014.2-fuel6.0~mira11_all.deb
python-keystone_2014.2-fuel6.0~mira11_all.deb

Changeset merged. Package placed on primary repository
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0-updates-stable/ubuntu

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.