| 2016-04-18 12:05:24 |
Mikhail Chernik |
bug |
|
|
added bug |
| 2016-04-18 12:55:34 |
Boris Bobrov |
mos: assignee |
Boris Bobrov (bbobrov) |
MOS Keystone (mos-keystone) |
|
| 2016-04-18 13:14:03 |
Dina Belova |
tags |
scale |
area-keystone scale |
|
| 2016-04-18 13:14:17 |
Dina Belova |
mos: status |
New |
Confirmed |
|
| 2016-04-18 13:14:22 |
Dina Belova |
mos: importance |
Undecided |
High |
|
| 2016-04-18 13:14:25 |
Dina Belova |
mos: milestone |
|
9.0 |
|
| 2016-04-18 13:16:49 |
Bug Checker Bot |
tags |
area-keystone scale |
area-keystone need-info scale |
|
| 2016-04-18 19:36:47 |
Leontii Istomin |
description |
Environment:
Reproduced on RackSpace lab, 3 controllers, 197 computes, VxLAN+DVR, MOS 9.0 ISO 188
Detailed description:
Keystone caches the whole revoke tree, which can exceed the 1M memcached object size limit if huge number of tokens get revoked at the same time (details: https://github.com/lericson/pylibmc/issues/184) .
After that keystone breaks its operation and cluster in not usable.
Keystone error:
2016-04-18 09:44:57.484 33105 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/revoke/core.py", line 225, in check_token
2016-04-18 09:44:57.484 33105 ERROR keystone.common.wsgi if self._get_revoke_tree().is_revoked(token_values):
Steps to reproduce:
run rally scenario KeystoneBasic.add_and_remove_user_role, against large cluster. Example scenario:
{
"kw": {
"runner": {
"type": "constant",
"concurrency": 20,
"times": 1970
},
"sla": {
"failure_rate": {
"max": 0
}
},
"context": {
"api_versions": {
"keystone": {
"version": 2
}
}
}
},
"name": "KeystoneBasic.add_and_remove_user_role",
"pos": 0
}
diagnostic snapshot will be added later |
Environment:
Reproduced on RackSpace lab, 3 controllers, 197 computes, VxLAN+DVR, MOS 9.0 ISO 188
Detailed description:
Keystone caches the whole revoke tree, which can exceed the 1M memcached object size limit if huge number of tokens get revoked at the same time (details: https://github.com/lericson/pylibmc/issues/184) .
After that keystone breaks its operation and cluster in not usable.
Keystone error:
2016-04-18 09:44:57.484 33105 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/revoke/core.py", line 225, in check_token
2016-04-18 09:44:57.484 33105 ERROR keystone.common.wsgi if self._get_revoke_tree().is_revoked(token_values):
Steps to reproduce:
1. set backend = dogpile.cache.pylibmc' in [cache] section in /etc/keystone/keystone.conf
2. Perform raly tests. All rally tests was failed excluding only three of them (results of the three tests are attached - rally_report.html)
3. found the following bug https://bugs.launchpad.net/mos/+bug/1571626
4. tried http://paste.openstack.org/show/494498/ + [revoke]caching=False in keystone.conf - te same behavior
run rally scenario KeystoneBasic.add_and_remove_user_role, against large cluster. Example scenario:
{
"kw": {
"runner": {
"type": "constant",
"concurrency": 20,
"times": 1970
},
"sla": {
"failure_rate": {
"max": 0
}
},
"context": {
"api_versions": {
"keystone": {
"version": 2
}
}
}
},
"name": "KeystoneBasic.add_and_remove_user_role",
"pos": 0
}
diagnostic snapshot will be added later |
|
| 2016-04-18 19:38:48 |
Leontii Istomin |
description |
Environment:
Reproduced on RackSpace lab, 3 controllers, 197 computes, VxLAN+DVR, MOS 9.0 ISO 188
Detailed description:
Keystone caches the whole revoke tree, which can exceed the 1M memcached object size limit if huge number of tokens get revoked at the same time (details: https://github.com/lericson/pylibmc/issues/184) .
After that keystone breaks its operation and cluster in not usable.
Keystone error:
2016-04-18 09:44:57.484 33105 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/revoke/core.py", line 225, in check_token
2016-04-18 09:44:57.484 33105 ERROR keystone.common.wsgi if self._get_revoke_tree().is_revoked(token_values):
Steps to reproduce:
1. set backend = dogpile.cache.pylibmc' in [cache] section in /etc/keystone/keystone.conf
2. Perform raly tests. All rally tests was failed excluding only three of them (results of the three tests are attached - rally_report.html)
3. found the following bug https://bugs.launchpad.net/mos/+bug/1571626
4. tried http://paste.openstack.org/show/494498/ + [revoke]caching=False in keystone.conf - te same behavior
run rally scenario KeystoneBasic.add_and_remove_user_role, against large cluster. Example scenario:
{
"kw": {
"runner": {
"type": "constant",
"concurrency": 20,
"times": 1970
},
"sla": {
"failure_rate": {
"max": 0
}
},
"context": {
"api_versions": {
"keystone": {
"version": 2
}
}
}
},
"name": "KeystoneBasic.add_and_remove_user_role",
"pos": 0
}
diagnostic snapshot will be added later |
Environment:
Reproduced on RackSpace lab, 3 controllers, 197 computes, VxLAN+DVR, MOS 9.0 ISO 188
Detailed description:
Keystone caches the whole revoke tree, which can exceed the 1M memcached object size limit if huge number of tokens get revoked at the same time (details: https://github.com/lericson/pylibmc/issues/184) .
After that keystone breaks its operation and cluster in not usable.
Keystone error:
2016-04-18 09:44:57.484 33105 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/revoke/core.py", line 225, in check_token
2016-04-18 09:44:57.484 33105 ERROR keystone.common.wsgi if self._get_revoke_tree().is_revoked(token_values):
Steps to reproduce:
1. set backend = dogpile.cache.pylibmc' in [cache] section in /etc/keystone/keystone.conf
2. Perform raly tests. All rally tests was failed excluding only three of them (results of the three tests are attached - rally_report.html)
3. found the following bug https://bugs.launchpad.net/mos/+bug/1571626
4. tried http://paste.openstack.org/show/494498/ + [revoke]caching=False in keystone.conf - No error any more, but request takes more then 60 sec and we get 504 even if request "opestack user list"
run rally scenario KeystoneBasic.add_and_remove_user_role, against large cluster. Example scenario:
{
"kw": {
"runner": {
"type": "constant",
"concurrency": 20,
"times": 1970
},
"sla": {
"failure_rate": {
"max": 0
}
},
"context": {
"api_versions": {
"keystone": {
"version": 2
}
}
}
},
"name": "KeystoneBasic.add_and_remove_user_role",
"pos": 0
}
diagnostic snapshot will be added later |
|
| 2016-04-18 19:40:58 |
Leontii Istomin |
description |
Environment:
Reproduced on RackSpace lab, 3 controllers, 197 computes, VxLAN+DVR, MOS 9.0 ISO 188
Detailed description:
Keystone caches the whole revoke tree, which can exceed the 1M memcached object size limit if huge number of tokens get revoked at the same time (details: https://github.com/lericson/pylibmc/issues/184) .
After that keystone breaks its operation and cluster in not usable.
Keystone error:
2016-04-18 09:44:57.484 33105 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/revoke/core.py", line 225, in check_token
2016-04-18 09:44:57.484 33105 ERROR keystone.common.wsgi if self._get_revoke_tree().is_revoked(token_values):
Steps to reproduce:
1. set backend = dogpile.cache.pylibmc' in [cache] section in /etc/keystone/keystone.conf
2. Perform raly tests. All rally tests was failed excluding only three of them (results of the three tests are attached - rally_report.html)
3. found the following bug https://bugs.launchpad.net/mos/+bug/1571626
4. tried http://paste.openstack.org/show/494498/ + [revoke]caching=False in keystone.conf - No error any more, but request takes more then 60 sec and we get 504 even if request "opestack user list"
run rally scenario KeystoneBasic.add_and_remove_user_role, against large cluster. Example scenario:
{
"kw": {
"runner": {
"type": "constant",
"concurrency": 20,
"times": 1970
},
"sla": {
"failure_rate": {
"max": 0
}
},
"context": {
"api_versions": {
"keystone": {
"version": 2
}
}
}
},
"name": "KeystoneBasic.add_and_remove_user_role",
"pos": 0
}
diagnostic snapshot will be added later |
Environment:
Reproduced on RackSpace lab, 3 controllers, 197 computes, VxLAN+DVR, MOS 9.0 ISO 188
Detailed description:
Keystone caches the whole revoke tree, which can exceed the 1M memcached object size limit if huge number of tokens get revoked at the same time (details: https://github.com/lericson/pylibmc/issues/184) .
from keystone adimn log file http://paste.openstack.org/show/494384/
After that keystone breaks its operation and cluster in not usable.
Keystone error:
2016-04-18 09:44:57.484 33105 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/revoke/core.py", line 225, in check_token
2016-04-18 09:44:57.484 33105 ERROR keystone.common.wsgi if self._get_revoke_tree().is_revoked(token_values):
Steps to reproduce:
1. set backend = dogpile.cache.pylibmc' in [cache] section in /etc/keystone/keystone.conf
2. Perform raly tests. All rally tests was failed excluding only three of them (results of the three tests are attached - rally_report.html)
3. found the following bug https://bugs.launchpad.net/mos/+bug/1571626
4. tried http://paste.openstack.org/show/494498/ + [revoke]caching=False in keystone.conf - No error any more, but request takes more then 60 sec and we get 504 even if request "opestack user list"
run rally scenario KeystoneBasic.add_and_remove_user_role, against large cluster. Example scenario:
{
"kw": {
"runner": {
"type": "constant",
"concurrency": 20,
"times": 1970
},
"sla": {
"failure_rate": {
"max": 0
}
},
"context": {
"api_versions": {
"keystone": {
"version": 2
}
}
}
},
"name": "KeystoneBasic.add_and_remove_user_role",
"pos": 0
}
diagnostic snapshot will be added later |
|
| 2016-04-18 19:55:41 |
Leontii Istomin |
description |
Environment:
Reproduced on RackSpace lab, 3 controllers, 197 computes, VxLAN+DVR, MOS 9.0 ISO 188
Detailed description:
Keystone caches the whole revoke tree, which can exceed the 1M memcached object size limit if huge number of tokens get revoked at the same time (details: https://github.com/lericson/pylibmc/issues/184) .
from keystone adimn log file http://paste.openstack.org/show/494384/
After that keystone breaks its operation and cluster in not usable.
Keystone error:
2016-04-18 09:44:57.484 33105 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/revoke/core.py", line 225, in check_token
2016-04-18 09:44:57.484 33105 ERROR keystone.common.wsgi if self._get_revoke_tree().is_revoked(token_values):
Steps to reproduce:
1. set backend = dogpile.cache.pylibmc' in [cache] section in /etc/keystone/keystone.conf
2. Perform raly tests. All rally tests was failed excluding only three of them (results of the three tests are attached - rally_report.html)
3. found the following bug https://bugs.launchpad.net/mos/+bug/1571626
4. tried http://paste.openstack.org/show/494498/ + [revoke]caching=False in keystone.conf - No error any more, but request takes more then 60 sec and we get 504 even if request "opestack user list"
run rally scenario KeystoneBasic.add_and_remove_user_role, against large cluster. Example scenario:
{
"kw": {
"runner": {
"type": "constant",
"concurrency": 20,
"times": 1970
},
"sla": {
"failure_rate": {
"max": 0
}
},
"context": {
"api_versions": {
"keystone": {
"version": 2
}
}
}
},
"name": "KeystoneBasic.add_and_remove_user_role",
"pos": 0
}
diagnostic snapshot will be added later |
Environment:
Reproduced on RackSpace lab, 3 controllers, 197 computes, VxLAN+DVR, MOS 9.0 ISO 188
Detailed description:
Keystone caches the whole revoke tree, which can exceed the 1M memcached object size limit if huge number of tokens get revoked at the same time (details: https://github.com/lericson/pylibmc/issues/184) .
from keystone adimn log file http://paste.openstack.org/show/494384/
After that keystone breaks its operation and cluster in not usable.
Keystone error:
2016-04-18 09:44:57.484 33105 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/revoke/core.py", line 225, in check_token
2016-04-18 09:44:57.484 33105 ERROR keystone.common.wsgi if self._get_revoke_tree().is_revoked(token_values):
Steps to reproduce:
1. set backend = dogpile.cache.pylibmc' in [cache] section in /etc/keystone/keystone.conf
2. Perform raly tests. All rally tests was failed excluding only three of them (results of the three tests are attached - rally_report.html)
3. found the following bug https://bugs.launchpad.net/mos/+bug/1571626
4. tried http://paste.openstack.org/show/494498/ + [revoke]caching=False in keystone.conf - No error any more, but request takes more then 60 sec and we get 504 even if request "opestack user list"
run rally scenario KeystoneBasic.add_and_remove_user_role, against large cluster. Example scenario:
{
"kw": {
"runner": {
"type": "constant",
"concurrency": 20,
"times": 1970
},
"sla": {
"failure_rate": {
"max": 0
}
},
"context": {
"api_versions": {
"keystone": {
"version": 2
}
}
}
},
"name": "KeystoneBasic.add_and_remove_user_role",
"pos": 0
}
diagnostic snapshot: http://mos-scale-share.mirantis.com/fuel-snapshot-2016-04-17_17-47-57.tar.xz |
|
| 2016-04-19 08:56:07 |
Leontii Istomin |
description |
Environment:
Reproduced on RackSpace lab, 3 controllers, 197 computes, VxLAN+DVR, MOS 9.0 ISO 188
Detailed description:
Keystone caches the whole revoke tree, which can exceed the 1M memcached object size limit if huge number of tokens get revoked at the same time (details: https://github.com/lericson/pylibmc/issues/184) .
from keystone adimn log file http://paste.openstack.org/show/494384/
After that keystone breaks its operation and cluster in not usable.
Keystone error:
2016-04-18 09:44:57.484 33105 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/revoke/core.py", line 225, in check_token
2016-04-18 09:44:57.484 33105 ERROR keystone.common.wsgi if self._get_revoke_tree().is_revoked(token_values):
Steps to reproduce:
1. set backend = dogpile.cache.pylibmc' in [cache] section in /etc/keystone/keystone.conf
2. Perform raly tests. All rally tests was failed excluding only three of them (results of the three tests are attached - rally_report.html)
3. found the following bug https://bugs.launchpad.net/mos/+bug/1571626
4. tried http://paste.openstack.org/show/494498/ + [revoke]caching=False in keystone.conf - No error any more, but request takes more then 60 sec and we get 504 even if request "opestack user list"
run rally scenario KeystoneBasic.add_and_remove_user_role, against large cluster. Example scenario:
{
"kw": {
"runner": {
"type": "constant",
"concurrency": 20,
"times": 1970
},
"sla": {
"failure_rate": {
"max": 0
}
},
"context": {
"api_versions": {
"keystone": {
"version": 2
}
}
}
},
"name": "KeystoneBasic.add_and_remove_user_role",
"pos": 0
}
diagnostic snapshot: http://mos-scale-share.mirantis.com/fuel-snapshot-2016-04-17_17-47-57.tar.xz |
Environment:
Reproduced on RackSpace lab, 3 controllers, 197 computes, VxLAN+DVR, MOS 9.0 ISO 188
Detailed description:
Keystone caches the whole revoke tree, which can exceed the 1M memcached object size limit if huge number of tokens get revoked at the same time (details: https://github.com/lericson/pylibmc/issues/184) .
from keystone adimn log file http://paste.openstack.org/show/494384/
After that keystone breaks its operation and cluster in not usable.
Keystone error:
2016-04-18 09:44:57.484 33105 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/revoke/core.py", line 225, in check_token
2016-04-18 09:44:57.484 33105 ERROR keystone.common.wsgi if self._get_revoke_tree().is_revoked(token_values):
Steps to reproduce:
1. set backend = dogpile.cache.pylibmc' in [cache] section in /etc/keystone/keystone.conf
2. Perform raly tests. All rally tests was failed excluding only three of them (results of the three tests are attached - rally_report.html)
3. found the following bug https://bugs.launchpad.net/mos/+bug/1571626
4. tried http://paste.openstack.org/show/494498/ + [revoke]caching=False in keystone.conf - No error any more, but request takes more then 60 sec and we get 504 even if request "opestack user list"
run rally scenario KeystoneBasic.add_and_remove_user_role, against large cluster. Example scenario:
{
"kw": {
"runner": {
"type": "constant",
"concurrency": 20,
"times": 1970
},
"sla": {
"failure_rate": {
"max": 0
}
},
"context": {
"api_versions": {
"keystone": {
"version": 2
}
}
}
},
"name": "KeystoneBasic.add_and_remove_user_role",
"pos": 0
}
diagnostic snapshot: http://mos-scale-share.mirantis.com/fuel-snapshot-2016-04-17_17-47-57.tar.xz
etc and log folders from controller nodes: http://mos-scale-share.mirantis.com/controller-data.tar.xz |
|
| 2016-04-19 09:14:46 |
Fuel Devops McRobotson |
mos/10.0.x: importance |
Undecided |
High |
|
| 2016-04-19 09:14:46 |
Fuel Devops McRobotson |
mos/10.0.x: status |
New |
Confirmed |
|
| 2016-04-19 09:14:46 |
Fuel Devops McRobotson |
mos/10.0.x: milestone |
|
10.0 |
|
| 2016-04-19 09:14:46 |
Fuel Devops McRobotson |
mos/10.0.x: assignee |
|
MOS Keystone (mos-keystone) |
|
| 2016-04-19 14:00:42 |
Alexander Petrov |
bug |
|
|
added subscriber Alexander Petrov |
| 2016-04-19 20:05:37 |
Sergey Shevorakov |
tags |
area-keystone need-info scale |
area-keystone blocker-for-qa need-info scale |
|
| 2016-04-20 13:01:13 |
Boris Bobrov |
mos: assignee |
MOS Keystone (mos-keystone) |
MOS Puppet Team (mos-puppet) |
|
| 2016-04-21 07:19:43 |
Bogdan Dobrelya |
bug task added |
|
fuel |
|
| 2016-04-21 07:19:51 |
Bogdan Dobrelya |
nominated for series |
|
fuel/mitaka |
|
| 2016-04-21 07:19:51 |
Bogdan Dobrelya |
bug task added |
|
fuel/mitaka |
|
| 2016-04-21 07:19:51 |
Bogdan Dobrelya |
nominated for series |
|
fuel/newton |
|
| 2016-04-21 07:19:51 |
Bogdan Dobrelya |
bug task added |
|
fuel/newton |
|
| 2016-04-21 07:19:58 |
Bogdan Dobrelya |
fuel/newton: status |
New |
In Progress |
|
| 2016-04-21 07:20:01 |
Bogdan Dobrelya |
fuel/mitaka: importance |
Undecided |
High |
|
| 2016-04-21 07:20:04 |
Bogdan Dobrelya |
fuel/newton: importance |
Undecided |
High |
|
| 2016-04-21 07:20:06 |
Bogdan Dobrelya |
fuel/newton: assignee |
|
Dmitry Ilyin (idv1985) |
|
| 2016-04-21 07:20:09 |
Bogdan Dobrelya |
fuel/newton: milestone |
|
10.0 |
|
| 2016-04-21 07:20:11 |
Bogdan Dobrelya |
fuel/mitaka: milestone |
|
9.0 |
|
| 2016-04-21 07:20:16 |
Bogdan Dobrelya |
fuel/mitaka: assignee |
|
Fuel Library Team (fuel-library) |
|
| 2016-04-21 07:20:20 |
Bogdan Dobrelya |
fuel/mitaka: status |
New |
Triaged |
|
| 2016-04-21 10:15:43 |
Matthew Mosesohn |
mos/10.0.x: assignee |
MOS Keystone (mos-keystone) |
MOS Puppet Team (mos-puppet) |
|
| 2016-04-21 10:15:49 |
Matthew Mosesohn |
fuel/mitaka: assignee |
Fuel Library Team (fuel-library) |
MOS Puppet Team (mos-puppet) |
|
| 2016-04-21 10:15:53 |
Matthew Mosesohn |
fuel/newton: assignee |
Dmitry Ilyin (idv1985) |
MOS Puppet Team (mos-puppet) |
|
| 2016-04-22 09:18:42 |
Ivan Berezovskiy |
mos/10.0.x: assignee |
MOS Puppet Team (mos-puppet) |
Dmitry Ilyin (idv1985) |
|
| 2016-04-22 09:18:46 |
Ivan Berezovskiy |
mos: assignee |
MOS Puppet Team (mos-puppet) |
Dmitry Ilyin (idv1985) |
|
| 2016-04-22 09:18:52 |
Ivan Berezovskiy |
fuel/newton: assignee |
MOS Puppet Team (mos-puppet) |
Dmitry Ilyin (idv1985) |
|
| 2016-04-22 09:18:56 |
Ivan Berezovskiy |
fuel/mitaka: assignee |
MOS Puppet Team (mos-puppet) |
Dmitry Ilyin (idv1985) |
|
| 2016-04-22 09:19:01 |
Ivan Berezovskiy |
mos/10.0.x: status |
Confirmed |
In Progress |
|
| 2016-04-22 14:33:44 |
OpenStack Infra |
fuel: status |
In Progress |
Fix Committed |
|
| 2016-04-26 17:01:47 |
OpenStack Infra |
fuel/mitaka: status |
Triaged |
In Progress |
|
| 2016-04-27 07:30:39 |
OpenStack Infra |
fuel/mitaka: status |
In Progress |
Fix Committed |
|
| 2016-04-27 12:40:14 |
Ivan Berezovskiy |
mos/10.0.x: status |
In Progress |
Fix Committed |
|
| 2016-04-27 12:40:16 |
Ivan Berezovskiy |
mos: status |
Confirmed |
Fix Committed |
|
| 2016-06-10 11:32:33 |
Andrew Kalach |
mos: status |
Fix Committed |
Fix Released |
|
| 2016-06-10 14:27:30 |
Michael Semenov |
fuel/mitaka: status |
Fix Committed |
Fix Released |
|
