Mirantis OpenStack

keystone doesn't respond if cluster is actively used

Bug #1413341 reported by Leontii Istomin
34
This bug affects 5 people
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Fix Released
Critical
Alexander Makarov
Mirantis OpenStack 6.1
5.1.x
Fix Released
Critical
Alex Ermolov
Mirantis OpenStack 5.1.1-mu-1
6.0.x
Fix Released
Critical
Alex Ermolov
Mirantis OpenStack 6.0-updates
6.1.x
Fix Released
Critical
Alexander Makarov
Mirantis OpenStack 6.0-mu-2

Bug Description

[root@fuel ~]# fuel --fuel-version
api: '1.0'
astute_sha: f7cda2171b0b677dfaeb59693d980a2d3ee4c3e0
auth_required: true
build_id: 2015-01-20_08-49-23
build_number: '38'
feature_groups:
- mirantis
fuellib_sha: 9aa913096fb93ea4847ee14bfaf33597326886f3
fuelmain_sha: 1ee1766a51bdb5bed75d5c2efdcaaa318118e439
nailgun_sha: 5f91157daa6798ff522ca9f6d34e7e135f150a90
ostf_sha: 3d2f44dcfa32d6ce0372cc64695e9edcc1913ea7
production: docker
release: 6.0.1
release_versions:
  2014.2-6.0:
    VERSION:
      api: '1.0'
      astute_sha: f7cda2171b0b677dfaeb59693d980a2d3ee4c3e0
      build_id: 2015-01-20_08-49-23
      build_number: '38'
      feature_groups:
      - mirantis
      fuellib_sha: 9aa913096fb93ea4847ee14bfaf33597326886f3
      fuelmain_sha: 1ee1766a51bdb5bed75d5c2efdcaaa318118e439
      nailgun_sha: 5f91157daa6798ff522ca9f6d34e7e135f150a90
      ostf_sha: 3d2f44dcfa32d6ce0372cc64695e9edcc1913ea7
      production: docker
      release: 6.0.1

Ubuntu, HA, neutron-gre, Ceilometer, debug, Ceph for volumes,images,ephemerals,objects
controllers: 3
computes: 97

root@node-47:~# keystone --debug user-list
DEBUG:keystoneclient.auth.identity.v2:Making authentication request to http://192.168.0.2:5000/v2.0/tokens
INFO:urllib3.connectionpool:Starting new HTTP connection (1): 192.168.0.2
DEBUG:urllib3.connectionpool:Setting read timeout to 600.0
DEBUG:urllib3.connectionpool:"POST /v2.0/tokens HTTP/1.1" 200 4102
DEBUG:keystoneclient.session:REQ: curl -i -X GET http://172.16.45.101:35357/v2.0/users -H "User-Agent: python-keystoneclient" -H "X-Auth-Token: TOKEN_REDACTED"
INFO:urllib3.connectionpool:Starting new HTTP connection (1): 172.16.45.101
DEBUG:urllib3.connectionpool:Setting read timeout to 600.0
DEBUG:urllib3.connectionpool:"GET /v2.0/users HTTP/1.1" 503 None
DEBUG:keystoneclient.session:RESP:
DEBUG:keystoneclient.session:Request returned failure status: 503
Service Unavailable (HTTP 503)

root@node-47:~# crm status
Last updated: Wed Jan 21 15:30:14 2015
Last change: Tue Jan 20 20:09:03 2015 via cibadmin on node-47
Stack: classic openais (with plugin)
Current DC: node-72 - partition with quorum
Version: 1.1.10-42f2063
3 Nodes configured, 3 expected votes
29 Resources configured

Online: [ node-47 node-71 node-72 ]

 vip__public (ocf::mirantis:ns_IPaddr2): Started node-71
 Clone Set: clone_ping_vip__public [ping_vip__public]
     Started: [ node-47 node-71 node-72 ]
 vip__management (ocf::mirantis:ns_IPaddr2): Started node-72
 Clone Set: clone_p_heat-engine [p_heat-engine]
     Started: [ node-47 node-71 node-72 ]
 Master/Slave Set: master_p_rabbitmq-server [p_rabbitmq-server]
     Masters: [ node-71 ]
     Slaves: [ node-47 node-72 ]
 Clone Set: clone_p_neutron-plugin-openvswitch-agent [p_neutron-plugin-openvswitch-agent]
     Started: [ node-47 node-71 node-72 ]
 p_neutron-dhcp-agent (ocf::mirantis:neutron-agent-dhcp): Started node-47
 Clone Set: clone_p_neutron-metadata-agent [p_neutron-metadata-agent]
     Started: [ node-47 node-71 node-72 ]
 Clone Set: clone_p_neutron-l3-agent [p_neutron-l3-agent]
     Started: [ node-47 node-71 node-72 ]
 Clone Set: clone_p_mysql [p_mysql]
     Started: [ node-47 node-71 node-72 ]
 Clone Set: clone_p_haproxy [p_haproxy]
     Started: [ node-47 node-71 node-72 ]

diagnostic snapshot is here https://drive.google.com/a/mirantis.com/file/d/0Bx4ptZV1Jt7hRTZlUWIzUk5PYmM/view?usp=sharing

Revision history for this message
Leontii Istomin (listomin) wrote :

it's baremetal installation

Leontii Istomin (listomin)
Changed in mos:
assignee: nobody → MOS Keystone (mos-keystone)
Leontii Istomin (listomin)
no longer affects: fuel
Revision history for this message
Alexander Makarov (amakarov) wrote :

Nova logs:

2015-01-21T05:48:05.160002+00:00 debug: REQ: curl -i -X GET http://192.168.0.2:35357/v3/auth/tokens -H "X-Subject-Token: TOKEN_REDACTED" -H "User-Agent: python-keystoneclient" -H "Accept: a
pplication/json" -H "X-Auth-Token: TOKEN_REDACTED"
2015-01-21T05:48:05.160002+00:00 debug: RESP:
2015-01-21T05:48:05.160002+00:00 debug: Request returned failure status: 504
2015-01-21T05:48:05.160002+00:00 err: Bad response code while validating token: 504
2015-01-21T05:48:05.160002+00:00 warning: Identity response: <html><body><h1>504 Gateway Time-out</h1>
The server didn't respond in time.
</body></html>
2015-01-21T05:48:05.160002+00:00 warning: Authorization failed for token
2015-01-21T05:48:05.160002+00:00 info: Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 459, in handle_one_response
    write(''.join(towrite))
  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 390, in write
    _writelines(towrite)
  File "/usr/lib/python2.7/socket.py", line 334, in writelines
    self.flush()
  File "/usr/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
  File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 357, in sendall
    tail = self.send(data, flags)
  File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 340, in send
    total_sent += fd.send(data[total_sent:], flags)
error: [Errno 104] Connection reset by peer

Meanwhile keystone log reports nothing.
My suspicion is: keystone locks for too long under high load as memcache lock in it sleeps ~0.5sec for every of 15 acquire attempts.

We can try to reduce this value (to 0 for ex.) and wait for "Maximum lock attempts on %s occurred." in keystone log.

Revision history for this message
Alexander Makarov (amakarov) wrote :

Let's try to minimize sleep between lock attempts.

Alexander Makarov (amakarov)
Changed in mos:
importance: Undecided → High
assignee: MOS Keystone (mos-keystone) → Alexander Makarov (amakarov)
Igor Marnat (imarnat)
summary: - keystone doesn't response if cluster is actively used
+ keystone doesn't respond if cluster is actively used
Revision history for this message
Alexander Makarov (amakarov) wrote :

Need a feedback on proposed solution (patch).

Changed in mos:
assignee: Alexander Makarov (amakarov) → Leontiy Istomin (listomin)
Revision history for this message
Leontii Istomin (listomin) wrote :

can't be reproduced even without the patch.

Changed in mos:
status: New → Invalid
Revision history for this message
Dina Belova (dbelova) wrote :

it was reproduced one more time. Please contact mod-scale team for more information. It relates to the 107th scale testing run.

Changed in mos:
status: Invalid → Confirmed
Revision history for this message
Alexander Makarov (amakarov) wrote :

We have system default ``open_files`` limit which is enough for human user but may harm service environment

keystone@node-85:~$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 256968
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 256968
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

We already had an issue resolved such a way: https://bugs.launchpad.net/fuel/+bug/1405657

Revision history for this message
Boris Bobrov (bbobrov) wrote :

In my opinion the bug is somehow related to memcache. We could debug it with gdb but we can't, because we have neither -gdb symbols nor gdb on the nodes.

Revision history for this message
Boris Bobrov (bbobrov) wrote :

The last messages from the hang keystone process is:

2015-01-31 13:21:49.214 12886 DEBUG keystone.common.cache._memcache_pool [-] Memcached pool 60709456, thread 140707178018560: Releasing connection 89609232 _debug_logger /usr/lib/python2.7/dist-packages/keystone/common/cache/_memcache_pool.py:102

There are no further entries from process 12886 and no entries mentioning connection 89609232

Revision history for this message
Boris Bobrov (bbobrov) wrote :

Since we cannot debug keystone properly, I've added some debug prints around memcache-related code and set [cache]debug_cache_backend = true. It was done on the lab itself on node-85. As discussed with Leontiy Istomin he will launch rally without resetting the lab to see how it goes.

Revision history for this message
Leontii Istomin (listomin) wrote :

I reproduced the issue:
root@node-32:~# keystone --debug --os-token xJrbBgdH --os-endpoint http://192.168.0.87:35357/v2.0/ user-list
WARNING: Bypassing authentication using a token & endpoint (authentication credentials are being ignored).
DEBUG:keystoneclient.session:REQ: curl -i -X GET http://192.168.0.87:35357/v2.0/users -H "User-Agent: python-keystoneclient" -H "X-Auth-Token: TOKEN_REDACTED"
INFO:urllib3.connectionpool:Starting new HTTP connection (1): 192.168.0.87
DEBUG:urllib3.connectionpool:Setting read timeout to 600.0
Unable to establish connection to http://192.168.0.87:35357/v2.0/users

As workaround we can just restart keystone service and after that it responds.

Revision history for this message
Leontii Istomin (listomin) wrote :

after nova rally tests two keystones from three hang. Memcached statistics bellow:
Controller-1: http://paste.openstack.org/show/166116/
Controller-2: http://paste.openstack.org/show/166117/ - keystone respond on this controller
Controller-3: http://paste.openstack.org/show/166118/

Revision history for this message
Jay Pipes (jaypipes) wrote :

Boris, why can't you just install GDB and strace manually on the node?

Revision history for this message
Jay Pipes (jaypipes) wrote :

It seems to me that Alexander was on the right track when he noted the ulimit on open files is 1024. Can we try adapting that to 4096 and re-running the tests? Seems that would be the easiest thing to change and re-test.

Revision history for this message
Boris Bobrov (bbobrov) wrote :

> why can't you just install GDB

Because python shipped was compiled with wrong options. Please see https://review.fuel-infra.org/#/c/1019/1 for more details.

Revision history for this message
Boris Bobrov (bbobrov) wrote :

I've added even more debug prints. I suggest to increase the limits and re-run the tests.

Revision history for this message
Leontii Istomin (listomin) wrote :

the following test was running on each controller:
service memcached restart; service keystone restart; while [ 1 == 1 ]; do date >> memcached_node-32.txt; timeout -k 2 5 echo stats | nc 192.168.0.34 11211 >> memcached_node-32.txt; sleep 30; done
I've changed open file limits on 32 and 85 nodes
root@node-85:~# ulimit -Hn 102400
root@node-85:~# ulimit -Sn 102400

user of memcached has increased limits too
root@node-32:~# ps aux | grep memcached
root 21049 0.0 0.0 9388 940 pts/8 S+ 08:54 0:00 grep --color=auto memcached
nobody 31596 0.3 0.1 951076 52140 ? Sl Feb03 3:38 /usr/bin/memcached -P /var/run/memcached.pid -m 30535 -l 192.168.0.34 -p 11211 -U 11211 -u nobody -c 8192 -t 12
root@node-32:~# su nobody -s /bin/bash -c "ulimit -Hn"
102400
root@node-32:~# su nobody -s /bin/bash -c "ulimit -Sn"
102400
root@node-32:~# su keystone -s /bin/bash -c "ulimit -Hn"
102400
root@node-32:~# su keystone -s /bin/bash -c "ulimit -Sn"
102400

the issue has been reproduced on each controller node:
root@node-32:~# for i in 34 80 87; do echo node-$i; keystone --debug --os-token xJrbBgdH --os-endpoint http://192.168.0.$i:35357/v2.0/ user-list; done
node-34
DEBUG:keystoneclient.session:REQ: curl -i -X GET http://192.168.0.34:35357/v2.0/users -H "User-Agent: python-keystoneclient" -H "X-Auth-Token: TOKEN_REDACTED"
INFO:urllib3.connectionpool:Starting new HTTP connection (1): 192.168.0.34
DEBUG:urllib3.connectionpool:Setting read timeout to 600.0
Unable to establish connection to http://192.168.0.34:35357/v2.0/users
node-80
DEBUG:keystoneclient.session:REQ: curl -i -X GET http://192.168.0.80:35357/v2.0/users -H "User-Agent: python-keystoneclient" -H "X-Auth-Token: TOKEN_REDACTED"
INFO:urllib3.connectionpool:Starting new HTTP connection (1): 192.168.0.80
DEBUG:urllib3.connectionpool:Setting read timeout to 600.0
Unable to establish connection to http://192.168.0.80:35357/v2.0/users
node-87
DEBUG:keystoneclient.session:REQ: curl -i -X GET http://192.168.0.87:35357/v2.0/users -H "User-Agent: python-keystoneclient" -H "X-Auth-Token: TOKEN_REDACTED"
INFO:urllib3.connectionpool:Starting new HTTP connection (1): 192.168.0.87
DEBUG:urllib3.connectionpool:Setting read timeout to 600.0
Unable to establish connection to http://192.168.0.87:35357/v2.0/users

here you can see that some processes didn't use a processor time
http://paste.openstack.org/show/166677/
http://paste.openstack.org/show/166678/
http://paste.openstack.org/show/166679/

Revision history for this message
Boris Bobrov (bbobrov) wrote :

I've updated Python from wget http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0-stable-1019/ubuntu/ and rebooted node-85. The updated python should contain debug info and we should be able to use gdb to debug.

Another thought: I think it might be related to eventlet used in our setup. I tried to reproduce the bug some time ago in my little environment with apache, but no luck. I didn't pay much attention to that difference then.

So, further actions:
1. Try again with python with enabled debug symbols
2. Get eventlet-based setup to my test env
3. Deploy an apache-powered keystone to the lab and try running tests again.

Revision history for this message
Davanum Srinivas (DIMS) (dims-v) wrote :

I totally vote for #3 this is the direction the keystone team wants everyone to use. devstack also uses apache by default

Dina Belova (dbelova)
Changed in mos:
importance: High → Critical
Revision history for this message
Alexander Makarov (amakarov) wrote :

+1 on #3

Revision history for this message
Davanum Srinivas (DIMS) (dims-v) wrote :

If we really think keystone+eventlet is ok but python-memcacheclient is causing issues, we can try the swift's MemcacheRing client implementation:

http://git.openstack.org/cgit/openstack/swift/tree/swift/common/memcached.py#n161

Revision history for this message
Boris Bobrov (bbobrov) wrote :

http://paste.openstack.org/show/W0jkpOKeEOzrDcrBHGZH/ -- first lines of py-bt output.

Revision history for this message
Boris Bobrov (bbobrov) wrote :

http://bugs.python.org/issue6721 -- first guess, will keep on verifying

Revision history for this message
Boris Bobrov (bbobrov) wrote :

After more googling I could found more issues regarding logging and eventlet. For example, there is patch to nova https://review.openstack.org/#/c/22162/ (not accepted though), with description of the problem.

Anyway, this needs further investigation. I am not sure that we use tpool anywhere in Keystone.

Revision history for this message
Davanum Srinivas (DIMS) (dims-v) wrote :

One more option - use the PylibmcBackend or another backend
https://bitbucket.org/zzzeek/dogpile.cache/src/a75835875b9c94bcc018f38d4fc829ec673c9371/dogpile/cache/backends/memcached.py?at=master

This will eliminate python-memcached as source of error.

Revision history for this message
Boris Bobrov (bbobrov) wrote :

Davanum, now I think that it is not memcache-related issue. It always hung because of logs, and logs were most actively used in memcache pool. I think that we can hit the issue in any other module under certain circumstances.

Revision history for this message
Aleksandr Shaposhnikov (alashai8) wrote :

Hi guys, here is a little bit more info.
root@node-8:~# wget http://192.168.0.10:5000/http://192.168.0.2:5000/v2.0/tokens
--2015-02-09 22:26:43-- http://192.168.0.10:5000/http://192.168.0.2:5000/v2.0/tokens
Connecting to 192.168.0.10:5000... connected.
HTTP request sent, awaiting response... ^C
root@node-8:~# wget http://192.168.0.10:5000/v2.0/tokens
--2015-02-09 22:26:53-- http://192.168.0.10:5000/v2.0/tokens
Connecting to 192.168.0.10:5000... connected.
HTTP request sent, awaiting response... Read error (Connection timed out) in headers.
Retrying.

--2015-02-09 22:27:15-- (try: 2) http://192.168.0.10:5000/v2.0/tokens
Connecting to 192.168.0.10:5000... connected.
HTTP request sent, awaiting response... Read error (Connection timed out) in headers.
Retrying.

--2015-02-09 22:27:30-- (try: 3) http://192.168.0.10:5000/v2.0/tokens
Connecting to 192.168.0.10:5000... connected.
HTTP request sent, awaiting response... Read error (Connection timed out) in headers.
Retrying.

--2015-02-09 22:28:02-- (try: 4) http://192.168.0.10:5000/v2.0/tokens
Connecting to 192.168.0.10:5000... connected.
HTTP request sent, awaiting response... Read error (Connection timed out) in headers.
Retrying.

^C
root@node-8:~# ^C
root@node-8:~# netstat -anp |grep -c 11211
4257

I think that this is an issues with using of memcache or to be more specific with keystone to memcache connection_pool.
There are 4.5k connections to the memcache that for me means only one thing: we're not reusing them. So I will recommend to take a look in our keystone.cache.memcache_pool implementation.

Revision history for this message
Boris Bobrov (bbobrov) wrote :

We believe that the issue is in logging. I've inserted a 'return' statement in the beginning of https://github.com/openstack/keystone/blob/master/keystone/common/cache/_memcache_pool.py#L97 on nodes 44 and 49.

Revision history for this message
Alexander Makarov (amakarov) wrote :

Confirming logging issue:

Python logging used in oslo.log consists of a bunch of logger objects for different modules.
Any logger object calls handle() method of every logging handler registered in the logger or any of it's parents.
All Handlers are derived from python logging.Handler class.
logging.Handler.handle() implemented as a critical section using a lock.
The lock used in logging.Handler.handle() is created by logging.Handler.createLock().
The lock is threading.RLock() which is reentrant allowing owner thread to attempt aquiring lock without blocking.
Thread is identified using _get_ident() call which is patched by eventlet making RLock() non-reentrant.

TL;DR
Eventlet monkeypatching breakes threading.RLock making it non-reentrant.

Revision history for this message
Alexander Makarov (amakarov) wrote :

Logging handler substitution with fixed locking: https://review.fuel-infra.org/#/c/3054

Davanum Srinivas (DIMS) (dims-v)
Changed in oslo.log:
status: New → Confirmed
importance: Undecided → Medium
assignee: nobody → Davanum Srinivas (DIMS) (dims-v)
Revision history for this message
Boris Bobrov (bbobrov) wrote :

Davanum, please don't do anything yet. We have another suspect now.

Revision history for this message
Alexander Makarov (amakarov) wrote :

Correct solution is here: https://review.fuel-infra.org/#/c/3060/
Root cause of the problem is that eventlet patching occures after logging system initialization thus leaving all the locks used in logging handlers non-patched.

Revision history for this message
Davanum Srinivas (DIMS) (dims-v) wrote :

Alexander, what do you think of this approach?
https://review.openstack.org/#/c/154630/

Davanum Srinivas (DIMS) (dims-v)
Changed in keystone:
assignee: nobody → Davanum Srinivas (DIMS) (dims-v)
Revision history for this message
Alexander Makarov (amakarov) wrote :

DIMS, I'd rather leave patching where it is as we actually don't know whether we need eventlet or not up until config is parsed.

Revision history for this message
Alexander Makarov (amakarov) wrote :

BTW I'm not sure if it is an oslo.log bug. Having reconfigure feature in it would be nice though.

Revision history for this message
Roman Podoliaka (rpodolyaka) wrote :

@Alexander, apparently we do - keystone-all is only used when keystone wsgi app is run by eventlet

Revision history for this message
Davanum Srinivas (DIMS) (dims-v) wrote :

Alexander, I can confirm @Roman's comment. keystone-all is *never* used under Apache.

no longer affects: keystone
no longer affects: oslo.log
Revision history for this message
Davanum Srinivas (DIMS) (dims-v) wrote :

Alex, one more thing, if we don't do this right (really early), other subtle things could go wrong. logging may not be the only thing that initialized stuff before eventlet patching.

fyi, Ben is writing up some best practices https://review.openstack.org/#/c/154642/ i ran this by Ben and Joshua as well.

Revision history for this message
Alexander Makarov (amakarov) wrote :

@DIMS, yes I saw this guide and completely agree that patching must be done ASAP.
The problem I see is that we can't go to community and say: "Guys, we here don't use keystone-all in single-threaded mode, let's throw it away!" It will be a discussion at least if not just rejected for some obvious reason.

Alexander Makarov (amakarov)
Changed in mos:
status: Confirmed → In Progress
Revision history for this message
Jay Pipes (jaypipes) wrote :

When I go to https://review.fuel-infra.org/#/c/3060/, I get an error:

The page you requested was not found, or you do not have permission to view this page.

Revision history for this message
OSCI Robot (oscirobot) wrote :

package keystone has been built for project openstack/keystone
Package version == 2014.2, package release == fuel6.1.mira12.git.a5b1f59.96eb21d

Changeset: https://review.fuel-infra.org/3060
project: openstack/keystone
branch: openstack-ci/fuel-6.1/2014.2
author: Alexander Makarov
committer: Boris Bobrov
subject: Correct initialization order for logging to use eventlet locks
status: patchset-created

Files placed on repository:

NOTE: Changeset is not merged, created temporary package repository.
 repository URL: /centos-fuel-6.1-stable-3060/

Revision history for this message
OSCI Robot (oscirobot) wrote :

package keystone has been built for project openstack/keystone
Package version == 2014.2, package release == fuel6.1~mira12+git.a5b1f59.96eb21d

Changeset: https://review.fuel-infra.org/3060
project: openstack/keystone
branch: openstack-ci/fuel-6.1/2014.2
author: Alexander Makarov
committer: Boris Bobrov
subject: Correct initialization order for logging to use eventlet locks
status: patchset-created

Files placed on repository:

NOTE: Changeset is not merged, created temporary package repository.
 repository URL: /ubuntu-fuel-6.1-stable-3060/

Alexander Makarov (amakarov)
Changed in mos:
assignee: Leontiy Istomin (listomin) → Alexander Makarov (amakarov)
Dmitry Savenkov (dsavenkov)
Changed in mos:
milestone: none → 6.0.1
Revision history for this message
Dmitry Mescheryakov (dmitrymex) wrote :

Jay, please contact me if you need access to the MOS repos.

Revision history for this message
Dina Belova (dbelova) wrote :

Why is the fix proposed for 6.1 only? It should be proposed for both 6.0.1 and 6.1 - it was reproduced on both of them.

Revision history for this message
OSCI Robot (oscirobot) wrote :

RPM package keystone has been built for project openstack/keystone
Package version == 2014.2, package release == fuel6.0.1.mira13.git.213b9d5.96eb21d

Changeset: https://review.fuel-infra.org/3569
project: openstack/keystone
branch: openstack-ci/fuel-6.0.1/2014.2
author: Boris Bobrov
committer: Boris Bobrov
subject: Correct initialization order for logging to use eventlet locks
status: patchset-created

Files placed on repository:
openstack-keystone-2014.2-fuel6.0.1.mira13.git.213b9d5.96eb21d.noarch.rpm
openstack-keystone-doc-2014.2-fuel6.0.1.mira13.git.213b9d5.96eb21d.noarch.rpm
python-keystone-2014.2-fuel6.0.1.mira13.git.213b9d5.96eb21d.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0.1-stable-3569/centos

Revision history for this message
OSCI Robot (oscirobot) wrote :

DEB package keystone has been built for project openstack/keystone
Package version == 2014.2, package release == fuel6.0.1~mira13+git.213b9d5.96eb21d

Changeset: https://review.fuel-infra.org/3569
project: openstack/keystone
branch: openstack-ci/fuel-6.0.1/2014.2
author: Boris Bobrov
committer: Boris Bobrov
subject: Correct initialization order for logging to use eventlet locks
status: patchset-created

Files placed on repository:
keystone-doc_2014.2-fuel6.0.1~mira13+git.213b9d5.96eb21d_all.deb
keystone_2014.2-fuel6.0.1~mira13+git.213b9d5.96eb21d_all.deb
python-keystone_2014.2-fuel6.0.1~mira13+git.213b9d5.96eb21d_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0.1-stable-3569/ubuntu

Revision history for this message
OSCI Robot (oscirobot) wrote :

RPM package keystone has been built for project openstack/keystone
Package version == 2014.2, package release == fuel6.0.1.mira13

Changeset: https://review.fuel-infra.org/3569
project: openstack/keystone
branch: openstack-ci/fuel-6.0.1/2014.2
author: Boris Bobrov
committer: Boris Bobrov
subject: Correct initialization order for logging to use eventlet locks
status: change-merged

Files placed on repository:
openstack-keystone-2014.2-fuel6.0.1.mira13.noarch.rpm
openstack-keystone-doc-2014.2-fuel6.0.1.mira13.noarch.rpm
python-keystone-2014.2-fuel6.0.1.mira13.noarch.rpm

Changeset merged. Package placed on primary repository
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0.1-stable/centos

Revision history for this message
OSCI Robot (oscirobot) wrote :

DEB package keystone has been built for project openstack/keystone
Package version == 2014.2, package release == fuel6.0.1~mira13

Changeset: https://review.fuel-infra.org/3569
project: openstack/keystone
branch: openstack-ci/fuel-6.0.1/2014.2
author: Boris Bobrov
committer: Boris Bobrov
subject: Correct initialization order for logging to use eventlet locks
status: change-merged

Files placed on repository:
keystone-doc_2014.2-fuel6.0.1~mira13_all.deb
keystone_2014.2-fuel6.0.1~mira13_all.deb
python-keystone_2014.2-fuel6.0.1~mira13_all.deb

Changeset merged. Package placed on primary repository
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0.1-stable/ubuntu

Revision history for this message
OSCI Robot (oscirobot) wrote :

package keystone has been built for project openstack/keystone
Package version == 2014.2, package release == fuel6.1.mira12.git.b541c69.96eb21d

Changeset: https://review.fuel-infra.org/3060
project: openstack/keystone
branch: openstack-ci/fuel-6.1/2014.2
author: Alexander Makarov
committer: Alexander Makarov
subject: Correct initialization order for logging to use eventlet locks.
status: patchset-created

Files placed on repository:

NOTE: Changeset is not merged, created temporary package repository.
 repository URL: /centos-fuel-6.1-stable-3060/

Revision history for this message
OSCI Robot (oscirobot) wrote :

package keystone has been built for project openstack/keystone
Package version == 2014.2, package release == fuel6.1~mira12+git.b541c69.96eb21d

Changeset: https://review.fuel-infra.org/3060
project: openstack/keystone
branch: openstack-ci/fuel-6.1/2014.2
author: Alexander Makarov
committer: Alexander Makarov
subject: Correct initialization order for logging to use eventlet locks.
status: patchset-created

Files placed on repository:

NOTE: Changeset is not merged, created temporary package repository.
 repository URL: /ubuntu-fuel-6.1-stable-3060/

Revision history for this message
OSCI Robot (oscirobot) wrote :

package keystone has been built for project openstack/keystone
Package version == 2014.2, package release == fuel6.1.mira12

Changeset: https://review.fuel-infra.org/3060
project: openstack/keystone
branch: openstack-ci/fuel-6.1/2014.2
author: Alexander Makarov
committer: Alexander Makarov
subject: Correct initialization order for logging to use eventlet locks.
status: change-merged

Files placed on repository:

Changeset merged. Package placed on primary repository
 repository URL: /centos-fuel-6.1-stable/

Revision history for this message
OSCI Robot (oscirobot) wrote :

package keystone has been built for project openstack/keystone
Package version == 2014.2, package release == fuel6.1~mira12

Changeset: https://review.fuel-infra.org/3060
project: openstack/keystone
branch: openstack-ci/fuel-6.1/2014.2
author: Alexander Makarov
committer: Alexander Makarov
subject: Correct initialization order for logging to use eventlet locks.
status: change-merged

Files placed on repository:

Changeset merged. Package placed on primary repository
 repository URL: /ubuntu-fuel-6.1-stable/

Alexander Makarov (amakarov)
Changed in mos:
status: In Progress → Fix Committed
Revision history for this message
OSCI Robot (oscirobot) wrote :

RPM package keystone has been built for project openstack/keystone
Package version == 2014.1.3, package release == fuel5.1.2.mira5.git.e464d0f.823fcc4

Changeset: https://review.fuel-infra.org/3641
project: openstack/keystone
branch: openstack-ci/fuel-5.1.2/2014.1.1
author: Boris Bobrov
committer: Boris Bobrov
subject: Correct initialization order for logging to use eventlet locks
status: patchset-created

Files placed on repository:
openstack-keystone-2014.1.3-fuel5.1.2.mira5.git.e464d0f.823fcc4.noarch.rpm
openstack-keystone-doc-2014.1.3-fuel5.1.2.mira5.git.e464d0f.823fcc4.noarch.rpm
python-keystone-2014.1.3-fuel5.1.2.mira5.git.e464d0f.823fcc4.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1.2-stable-3641/centos

Revision history for this message
OSCI Robot (oscirobot) wrote :

DEB package keystone has been built for project openstack/keystone
Package version == 2014.1.3, package release == fuel5.1.2~mira3+git.e464d0f.823fcc4

Changeset: https://review.fuel-infra.org/3641
project: openstack/keystone
branch: openstack-ci/fuel-5.1.2/2014.1.1
author: Boris Bobrov
committer: Boris Bobrov
subject: Correct initialization order for logging to use eventlet locks
status: patchset-created

Files placed on repository:
keystone-doc_2014.1.3-fuel5.1.2~mira3+git.e464d0f.823fcc4_all.deb
keystone_2014.1.3-fuel5.1.2~mira3+git.e464d0f.823fcc4_all.deb
python-keystone_2014.1.3-fuel5.1.2~mira3+git.e464d0f.823fcc4_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1.2-stable-3641/ubuntu

Revision history for this message
Boris Bobrov (bbobrov) wrote :

I have proposed a change to stable/icehouse branch of upstream Keystone, https://review.openstack.org/#/c/156649/ .

Dmitry Borodaenko (angdraug)
tags: added: release-notes
Revision history for this message
OSCI Robot (oscirobot) wrote :

RPM package keystone has been built for project openstack/keystone
Package version == 2014.1.3, package release == fuel5.1.1.mira5.git.ed49319.823fcc4

Changeset: https://review.fuel-infra.org/4421
project: openstack/keystone
branch: openstack-ci/fuel-5.1.1-updates/2014.1.1
author: Alex Ermolov
committer: Alex Ermolov
subject: Correct initialization order for logging to use eventlet locks
status: patchset-created

Files placed on repository:
openstack-keystone-2014.1.3-fuel5.1.1.mira5.git.ed49319.823fcc4.noarch.rpm
openstack-keystone-doc-2014.1.3-fuel5.1.1.mira5.git.ed49319.823fcc4.noarch.rpm
python-keystone-2014.1.3-fuel5.1.1.mira5.git.ed49319.823fcc4.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1.1-updates-stable-4421/centos

Revision history for this message
OSCI Robot (oscirobot) wrote :

DEB package keystone has been built for project openstack/keystone
Package version == 2014.1.3, package release == fuel5.1.1~mira3+git.ed49319.823fcc4

Changeset: https://review.fuel-infra.org/4421
project: openstack/keystone
branch: openstack-ci/fuel-5.1.1-updates/2014.1.1
author: Alex Ermolov
committer: Alex Ermolov
subject: Correct initialization order for logging to use eventlet locks
status: patchset-created

Files placed on repository:
keystone-doc_2014.1.3-fuel5.1.1~mira3+git.ed49319.823fcc4_all.deb
keystone_2014.1.3-fuel5.1.1~mira3+git.ed49319.823fcc4_all.deb
python-keystone_2014.1.3-fuel5.1.1~mira3+git.ed49319.823fcc4_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1.1-updates-stable-4421/ubuntu

Revision history for this message
OSCI Robot (oscirobot) wrote :

RPM package keystone has been built for project openstack/keystone
Package version == 2014.1.3, package release == fuel5.1.1.mira5

Changeset: https://review.fuel-infra.org/4421
project: openstack/keystone
branch: openstack-ci/fuel-5.1.1-updates/2014.1.1
author: Alex Ermolov
committer: Alex Ermolov
subject: Correct initialization order for logging to use eventlet locks
status: change-merged

Files placed on repository:
openstack-keystone-2014.1.3-fuel5.1.1.mira5.noarch.rpm
openstack-keystone-doc-2014.1.3-fuel5.1.1.mira5.noarch.rpm
python-keystone-2014.1.3-fuel5.1.1.mira5.noarch.rpm

Changeset merged. Package placed on primary repository
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1.1-updates-stable/centos

Revision history for this message
OSCI Robot (oscirobot) wrote :

DEB package keystone has been built for project openstack/keystone
Package version == 2014.1.3, package release == fuel5.1.1~mira3

Changeset: https://review.fuel-infra.org/4421
project: openstack/keystone
branch: openstack-ci/fuel-5.1.1-updates/2014.1.1
author: Alex Ermolov
committer: Alex Ermolov
subject: Correct initialization order for logging to use eventlet locks
status: change-merged

Files placed on repository:
keystone-doc_2014.1.3-fuel5.1.1~mira3_all.deb
keystone_2014.1.3-fuel5.1.1~mira3_all.deb
python-keystone_2014.1.3-fuel5.1.1~mira3_all.deb

Changeset merged. Package placed on primary repository
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1.1-updates-stable/ubuntu

Revision history for this message
Miroslav Anashkin (manashkin) wrote :

Got 5.x customer who encountered this.
Please back-port to 5.1.2

tags: added: customer-found
Revision history for this message
OSCI Robot (oscirobot) wrote :

RPM package keystone has been built for project openstack/keystone
Package version == 2014.1.3, package release == fuel5.1.2.mira5.git.8ce880b.823fcc4

Changeset: https://review.fuel-infra.org/3641
project: openstack/keystone
branch: openstack-ci/fuel-5.1.2/2014.1.1
author: Boris Bobrov
committer: Boris Bobrov
subject: Correct initialization order for logging to use eventlet locks
status: patchset-created

Files placed on repository:
openstack-keystone-2014.1.3-fuel5.1.2.mira5.git.8ce880b.823fcc4.noarch.rpm
openstack-keystone-doc-2014.1.3-fuel5.1.2.mira5.git.8ce880b.823fcc4.noarch.rpm
python-keystone-2014.1.3-fuel5.1.2.mira5.git.8ce880b.823fcc4.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1.2-stable-3641/centos

Revision history for this message
OSCI Robot (oscirobot) wrote :

DEB package keystone has been built for project openstack/keystone
Package version == 2014.1.3, package release == fuel5.1.2~mira3+git.8ce880b.823fcc4

Changeset: https://review.fuel-infra.org/3641
project: openstack/keystone
branch: openstack-ci/fuel-5.1.2/2014.1.1
author: Boris Bobrov
committer: Boris Bobrov
subject: Correct initialization order for logging to use eventlet locks
status: patchset-created

Files placed on repository:
keystone-doc_2014.1.3-fuel5.1.2~mira3+git.8ce880b.823fcc4_all.deb
keystone_2014.1.3-fuel5.1.2~mira3+git.8ce880b.823fcc4_all.deb
python-keystone_2014.1.3-fuel5.1.2~mira3+git.8ce880b.823fcc4_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1.2-stable-3641/ubuntu

Revision history for this message
OSCI Robot (oscirobot) wrote :

RPM package keystone has been built for project openstack/keystone
Package version == 2014.2, package release == fuel6.0.mira11.git.6b6590c.96eb21d

Changeset: https://review.fuel-infra.org/5105
project: openstack/keystone
branch: openstack-ci/fuel-6.0-updates/2014.2
author: Alexander Makarov
committer: Alexander Makarov
subject: Correct initialization order for logging to use eventlet locks
status: patchset-created

Files placed on repository:
openstack-keystone-2014.2-fuel6.0.mira11.git.6b6590c.96eb21d.noarch.rpm
openstack-keystone-doc-2014.2-fuel6.0.mira11.git.6b6590c.96eb21d.noarch.rpm
python-keystone-2014.2-fuel6.0.mira11.git.6b6590c.96eb21d.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0-updates-stable-5105/centos

Revision history for this message
OSCI Robot (oscirobot) wrote :

DEB package keystone has been built for project openstack/keystone
Package version == 2014.2, package release == fuel6.0~mira11+git.6b6590c.96eb21d

Changeset: https://review.fuel-infra.org/5105
project: openstack/keystone
branch: openstack-ci/fuel-6.0-updates/2014.2
author: Alexander Makarov
committer: Alexander Makarov
subject: Correct initialization order for logging to use eventlet locks
status: patchset-created

Files placed on repository:
keystone-doc_2014.2-fuel6.0~mira11+git.6b6590c.96eb21d_all.deb
keystone_2014.2-fuel6.0~mira11+git.6b6590c.96eb21d_all.deb
python-keystone_2014.2-fuel6.0~mira11+git.6b6590c.96eb21d_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0-updates-stable-5105/ubuntu

Revision history for this message
OSCI Robot (oscirobot) wrote :

RPM package keystone has been built for project openstack/keystone
Package version == 2014.1.3, package release == fuel5.1.2.mira6

Changeset: https://review.fuel-infra.org/5178
project: openstack/keystone
branch: openstack-ci/fuel-5.1.2/2014.1.1
author: Alexander Makarov
committer: Alexander Makarov
subject: Place sql backend initialization after eventlet monkey-patching
status: change-merged

Files placed on repository:
openstack-keystone-2014.1.3-fuel5.1.2.mira6.noarch.rpm
openstack-keystone-doc-2014.1.3-fuel5.1.2.mira6.noarch.rpm
python-keystone-2014.1.3-fuel5.1.2.mira6.noarch.rpm

Changeset merged. Package placed on primary repository
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1.2-stable/centos

Revision history for this message
OSCI Robot (oscirobot) wrote :

DEB package keystone has been built for project openstack/keystone
Package version == 2014.1.3, package release == fuel5.1.2~mira4

Changeset: https://review.fuel-infra.org/5178
project: openstack/keystone
branch: openstack-ci/fuel-5.1.2/2014.1.1
author: Alexander Makarov
committer: Alexander Makarov
subject: Place sql backend initialization after eventlet monkey-patching
status: change-merged

Files placed on repository:
keystone-doc_2014.1.3-fuel5.1.2~mira4_all.deb
keystone_2014.1.3-fuel5.1.2~mira4_all.deb
python-keystone_2014.1.3-fuel5.1.2~mira4_all.deb

Changeset merged. Package placed on primary repository
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1.2-stable/ubuntu

Revision history for this message
OSCI Robot (oscirobot) wrote :

RPM package keystone has been built for project openstack/keystone
Package version == 2014.1.3, package release == fuel5.1.1.mira6

Changeset: https://review.fuel-infra.org/5179
project: openstack/keystone
branch: openstack-ci/fuel-5.1.1-updates/2014.1.1
author: Alexander Makarov
committer: Alexander Makarov
subject: Place sql initialization after eventlet monkey-patching
status: change-merged

Files placed on repository:
openstack-keystone-2014.1.3-fuel5.1.1.mira6.noarch.rpm
openstack-keystone-doc-2014.1.3-fuel5.1.1.mira6.noarch.rpm
python-keystone-2014.1.3-fuel5.1.1.mira6.noarch.rpm

Changeset merged. Package placed on primary repository
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1.1-updates-stable/centos

Revision history for this message
OSCI Robot (oscirobot) wrote :

DEB package keystone has been built for project openstack/keystone
Package version == 2014.1.3, package release == fuel5.1.1~mira4

Changeset: https://review.fuel-infra.org/5179
project: openstack/keystone
branch: openstack-ci/fuel-5.1.1-updates/2014.1.1
author: Alexander Makarov
committer: Alexander Makarov
subject: Place sql initialization after eventlet monkey-patching
status: change-merged

Files placed on repository:
keystone-doc_2014.1.3-fuel5.1.1~mira4_all.deb
keystone_2014.1.3-fuel5.1.1~mira4_all.deb
python-keystone_2014.1.3-fuel5.1.1~mira4_all.deb

Changeset merged. Package placed on primary repository
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1.1-updates-stable/ubuntu

Revision history for this message
Leontii Istomin (listomin) wrote :

verified with 6.1-233

Revision history for this message
Boris Bobrov (bbobrov) wrote :

wow, wait, afaik, "fix released" will be after 6.1 release

Olena Logvinova (ologvinova)
tags: added: release-notes-done
removed: release-notes
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to openstack/keystone (openstack-ci/fuel-7.0/2015.1.0)

Fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: Alexander Makarov <email address hidden>
Review: https://review.fuel-infra.org/8178

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to openstack/keystone (openstack-ci/fuel-5.1-updates/2014.1.1)

Fix proposed to branch: openstack-ci/fuel-5.1-updates/2014.1.1
Change author: Alex Ermolov <email address hidden>
Review: https://review.fuel-infra.org/8853

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix merged to openstack/keystone (openstack-ci/fuel-5.1-updates/2014.1.1)

Reviewed: https://review.fuel-infra.org/8853
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-5.1-updates/2014.1.1

Commit: f3f40843143ca45de053ca7cce611e3955d12779
Author: Alex Ermolov <email address hidden>
Date: Mon Jul 20 10:19:44 2015

Correct initialization order for logging to use eventlet locks

Change-Id: I52910453c46778b1d80c65875f0e970a2c324fe0
Closes-Bug: 1413341

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Change abandoned on openstack/keystone (openstack-ci/fuel-7.0/2015.1.0)

Change abandoned by Alexander Makarov <email address hidden> on branch: openstack-ci/fuel-7.0/2015.1.0
Review: https://review.fuel-infra.org/8178

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.