heat-api hot spin
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Mirantis OpenStack |
Fix Released
|
High
|
Sergey Kolekonov | ||
6.0.x |
Fix Released
|
High
|
Denis Meltsaykin | ||
6.1.x |
Fix Released
|
High
|
Sergey Kolekonov |
Bug Description
Process heat-api is in hot spin state, it steadily consumes 85-95% cpu and doesn't accept http connections.
Strace shows the series of messages «GreenSocket object has no attribute fd» in GreenSocket.__del__ and maximum recursion depth exceeded in subclass check.
DEPLOYMENT: Centos + HA + Ceilo + Sahara + Murano
UPDATED:
This seems to be an issue with old python-requests package. Upgrade python-requests to 2.1.0+ should fix the problem. Only CentOS is affected, as Ubuntu uses the newer version.
Ilya Shakhat (shakhat) wrote : | #1 |
tags: | added: heat |
tags: | added: scale |
Ilya Shakhat (shakhat) wrote : | #2 |
[root@node-32 ~]# ps aux | grep 30707
root 21881 0.0 0.0 103244 880 pts/0 S+ 20:43 0:00 grep 30707
heat 30707 80.4 0.1 330068 57484 ? R Dec21 1540:41 /usr/bin/python /usr/bin/heat-api --config-file /etc/heat/heat.conf
[root@node-32 ~]# tail /var/log/
2014-12-21 14:49:06.340 30707 DEBUG keystonemiddlew
2014-12-21 14:49:06.343 30707 DEBUG heat.openstack.
2014-12-21 14:49:06.344 30707 DEBUG routes.middleware [req-225eecb3-
2014-12-21 14:49:06.344 30707 DEBUG routes.middleware [req-225eecb3-
2014-12-21 14:49:06.345 30707 DEBUG routes.middleware [req-225eecb3-
2014-12-21 14:49:06.346 30707 DEBUG root [req-225eecb3-
2014-12-21 14:49:06.415 30707 DEBUG heat.api.
2014-12-21 14:49:06.417 30707 INFO heat.common.
2014-12-21 14:49:06.419 30707 INFO urllib3.
2014-12-21 14:49:06.623 30707 DEBUG urllib3.
[root@node-32 ~]# netstat -aplnt | grep 30707
tcp 0 0 0.0.0.0:8004 0.0.0.0:* LISTEN 30707/python
tcp 1271 0 172.16.44.17:47014 93.184.216.34:80 CLOSE_WAIT 30707/python
[root@node-32 ~]# lsof -i | grep 30707
heat-api 30707 heat 7u IPv4 548124 0t0 TCP *:8004 (LISTEN)
heat-api 30707 heat 11u IPv4 22173539 0t0 TCP 172.16.
Ilya Shakhat (shakhat) wrote : | #3 |
strace output:
write(2, "Exception ", 10) = 10
write(2, "RuntimeError", 12) = 12
write(2, ": ", 2) = 2
write(2, "'", 1) = 1
write(2, "m", 1) = 1
write(2, "a", 1) = 1
write(2, "x", 1) = 1
write(2, "i", 1) = 1
write(2, "m", 1) = 1
write(2, "u", 1) = 1
write(2, "m", 1) = 1
write(2, " ", 1) = 1
write(2, "r", 1) = 1
write(2, "e", 1) = 1
write(2, "c", 1) = 1
write(2, "u", 1) = 1
write(2, "r", 1) = 1
write(2, "s", 1) = 1
write(2, "i", 1) = 1
write(2, "o", 1) = 1
write(2, "n", 1) = 1
write(2, " ", 1) = 1
write(2, "d", 1) = 1
write(2, "e", 1) = 1
write(2, "p", 1) = 1
write(2, "t", 1) = 1
write(2, "h", 1) = 1
write(2, " ", 1) = 1
write(2, "e", 1) = 1
write(2, "x", 1) = 1
write(2, "c", 1) = 1
write(2, "e", 1) = 1
write(2, "e", 1) = 1
write(2, "d", 1) = 1
write(2, "e", 1) = 1
write(2, "d", 1) = 1
write(2, " ", 1) = 1
write(2, "i", 1) = 1
write(2, "n", 1) = 1
write(2, " ", 1) = 1
write(2, "_", 1) = 1
write(2, "_", 1) = 1
write(2, "s", 1) = 1
write(2, "u", 1) = 1
write(2, "b", 1) = 1
write(2, "c", 1) = 1
write(2, "l", 1) = 1
write(2, "a", 1) = 1
write(2, "s", 1) = 1
write(2, "s", 1) = 1
write(2, "c", 1) = 1
write(2, "h", 1) = 1
write(2, "e", 1) = 1
write(2, "c", 1) = 1
write(2, "k", 1) = 1
write(2, "_", 1) = 1
write(2, "_", 1) = 1
write(2, "'", 1) = 1
write(2, " in ", 4) = 4
write(2, "<type 'exceptions.
write(2, " ignored\n", 9) = 9
write(2, "Exception ", 10) = 10
write(2, "AttributeError", 14) = 14
write(2, ": ", 2) = 2
write(2, "\"", 1) = 1
write(2, "'", 1) = 1
write(2, "G", 1) = 1
write(2, "r", 1) = 1
write(2, "e", 1) = 1
write(2, "e", 1) = 1
write(2, "n", 1) = 1
write(2, "S", 1) = 1...
Ilya Shakhat (shakhat) wrote : | #4 |
It looks suspicious that the last message in log is connection to www.example.com, which IP is 93.184.216.34 and the corresponding connection is still alive.
Changed in mos: | |
assignee: | nobody → MOS Heat (mos-heat) |
description: | updated |
Ilya Shakhat (shakhat) wrote : | #5 |
Pavlo Shchelokovskyy (pshchelo) wrote : Re: [Bug 1404983] Re: heat-api hot spin | #6 |
What was using heat API? I ask since the last lines in both logs are heat
being asked to validate a template at http://
and trying to fetch this url - looks like some dummy from unittest or the
like.
Pavlo Shchelokovskyy
Software Engineer
Mirantis Inc
www.mirantis.com
On Mon, Dec 22, 2014 at 11:05 PM, Ilya Shakhat <email address hidden>
wrote:
> Logs from controllers with the issue
>
> ** Attachment added: "heat-logs.tgz"
>
> https:/
>
> --
> You received this bug notification because you are a member of MOS Heat,
> which is a bug assignee.
> https:/
>
> Title:
> heat-api hot spin
>
> Status in Mirantis OpenStack:
> New
>
> Bug description:
> Process heat-api is in hot spin state, it steadily consumes 85-95% cpu
> and doesn't accept http connections.
>
> Strace shows the series of messages «GreenSocket object has no
> attribute fd» in GreenSocket.__del__ and maximum recursion depth
> exceeded in subclass check.
>
> DEPLOYMENT: Centos + HA + Ceilo + Sahara + Murano
>
> To manage notifications about this bug go to:
> https:/
>
Sergey Kraynev (skraynev) wrote : | #7 |
The root cause of this problem is two tempest negative tests
(https:/
In this tests we try to connect to example.com and fetch some templates.
Sometimes it leads to hanging heat-api service.
Pavlo Shchelokovskyy (pshchelo) wrote : | #8 |
Couple of bugs in requests and urllib3 that might be related
We use urllib3 v 1.7 and requests 1.2.3
https:/
https:/
Sergey Kraynev (skraynev) wrote : | #9 |
Pavlo:
juno requirements for heat tell, that:
requests>
so it is possible to use new versions of "requests" library.
Also urllib3: is not mentioned in juno requirements (not for Heat, not in global-
It appeared only in master branch of global requirements:
urllib3>=1.8.3 ( https:/
Pavlo Shchelokovskyy (pshchelo) wrote : | #10 |
So, I would state that this is caused by old urllib3, and updating it to >1.8 should prevent such issues in future.
Sergey Kraynev (skraynev) wrote : | #11 |
We reproduced it manually. Steps to reproduce:
1. Get Auth Token and Token id from response:
curl -d '{"auth"
2. reproduce requests:
curl -iv http://
3. result heat-api goes to down.....
Sergey Kraynev (skraynev) wrote : | #12 |
Small typo on step 2 need to specify content_type:
curl -iv http://
Sergey Kraynev (skraynev) wrote : | #13 |
Also:
I tried to reproduce it on my devstack and it works correct (without any problem).
note my requirements:
Name: requests
Version: 2.5.0
Name: urllib3
Version: 1.10
requirements for our fuel deployment:
Name: urllib3
Version: 1.7
Name: requests
Version: 1.2.3
Sergey Kraynev (skraynev) wrote : | #14 |
Checked for fuel deployment:
upgrading requests from 1.2.3 -> 2.5.0 (last) fixes the issue.
Changed in mos: | |
milestone: | none → 6.0.1 |
Pavlo Shchelokovskyy (pshchelo) wrote : | #15 |
we should probably report this in heat and backport it at least to Juno
Sergey Kraynev (skraynev) wrote : | #16 |
IMPORTANT NOTE:
Current bug is actual for Centos OS. For Ubuntu requests package is newer:
root@node-85:~# dpkg -l | grep requests
ii python-requests 2.2.1-1 elegant and simple HTTP library for Python, built for human beings
Ilya Shakhat (shakhat) wrote : | #17 |
CentOS:
[root@node-1 ~]# rpm -qa | grep reque
python-
OSCI Robot (oscirobot) wrote : | #18 |
RPM package python-eventlet has been built for project packages/
Package version == 0.16, package release == 1
Changeset: https:/
project: packages/
branch: 6.1
author: Roman Podoliaka
committer: Ivan Udovichenko
subject: Update to the latest upstream version
status: patchset-created
Files placed on repository:
python-
python-
NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://
OSCI Robot (oscirobot) wrote : | #19 |
RPM package python-eventlet has been built for project packages/
Package version == 0.16, package release == 1
Changeset: https:/
project: packages/
branch: 6.0.1
author: Roman Podoliaka
committer: Ivan Udovichenko
subject: Update to the latest upstream version
status: patchset-created
Files placed on repository:
python-
python-
NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://
OSCI Robot (oscirobot) wrote : | #21 |
package python-eventlet has been built for project packages/
Package version == 0.16.1, package release == 1
Changeset: https:/
project: packages/
branch: 6.1
author: Roman Podoliaka
committer: Ivan Udovichenko
subject: Update to the latest upstream version
status: patchset-created
Files placed on repository:
NOTE: Changeset is not merged, created temporary package repository.
repository URL: /centos-
OSCI Robot (oscirobot) wrote : | #22 |
RPM package python-eventlet has been built for project packages/
Package version == 0.16.1, package release == 1
Changeset: https:/
project: packages/
branch: 6.1
author: Roman Podoliaka
committer: Ivan Udovichenko
subject: Update to the latest upstream version
status: patchset-created
Files placed on repository:
python-
python-
NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://
OSCI Robot (oscirobot) wrote : | #23 |
DEB package python-eventlet has been built for project packages/
Package version == 0.16.1, package release == ubuntu3
Changeset: https:/
project: packages/
branch: 6.1
author: Roman Podoliaka
committer: Ivan Udovichenko
subject: Update to the latest upstream version
status: patchset-created
Files placed on repository:
python-
NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://
OSCI Robot (oscirobot) wrote : | #24 |
package python-eventlet has been built for project packages/
Package version == 0.16.1, package release == ubuntu1
Changeset: https:/
project: packages/
branch: 6.1
author: Roman Podoliaka
committer: Ivan Udovichenko
subject: Update to the latest upstream version
status: patchset-created
Files placed on repository:
NOTE: Changeset is not merged, created temporary package repository.
repository URL: /ubuntu-
OSCI Robot (oscirobot) wrote : | #26 |
DEB package python-eventlet has been built for project packages/
Package version == 0.16.1, package release == ubuntu3
Changeset: https:/
project: packages/
branch: 6.1
author: Roman Podoliaka
committer: Ivan Udovichenko
subject: Update to the latest upstream version
status: patchset-created
Files placed on repository:
python-
NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://
OSCI Robot (oscirobot) wrote : | #27 |
package python-eventlet has been built for project packages/
Package version == 0.16.1, package release == ubuntu1
Changeset: https:/
project: packages/
branch: 6.1
author: Roman Podoliaka
committer: Ivan Udovichenko
subject: Update to the latest upstream version
status: patchset-created
Files placed on repository:
NOTE: Changeset is not merged, created temporary package repository.
repository URL: /ubuntu-
OSCI Robot (oscirobot) wrote : | #29 |
DEB package python-eventlet has been built for project packages/
Package version == 0.16.1, package release == ubuntu3
Changeset: https:/
project: packages/
branch: 6.1
author: Roman Podoliaka
committer: Ivan Udovichenko
subject: Update to the latest upstream version
status: patchset-created
Files placed on repository:
python-
NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://
Roman Podoliaka (rpodolyaka) wrote : | #30 |
This seems to be fixed in the lastest eventlet release (https:/
We are not going to update eventlet in 6.0.1, though
Dmitry Mescheryakov (dmitrymex) wrote : | #31 |
We consider fixing this bug by bumping version of eventlet to be too risky in 6.0.1 timeframe. Hence moving to 6.0.2.
OSCI Robot (oscirobot) wrote : | #32 |
DEB package python-eventlet has been built for project packages/
Package version == 0.15.2~mos6.1, package release == ubuntu1
Changeset: https:/
project: packages/
branch: 6.1
author: Ivan Udovichenko
committer: Ivan Udovichenko
subject: Add patch to fix known issues
status: patchset-created
Files placed on repository:
python-
NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://
Roman Podoliaka (rpodolyaka) wrote : | #33 |
Ok, so we had a brief conversation with Sergey Kraynev on this and it seems that I added a lot of confusion to this thread.
Our current understanding is:
1) this is most likely an issue caused by old python-requests package. Sergey tried to reproduce the issue on his env and python-
2) there is a similar issue in the eventlet bug tracker (https:/
I suggest we try to update the python-requests package to a newer versions (>=2.1.0) for CentOS first, and should we reproduce the issue again, proceed with upgrading of eventlet.
According to https:/
Sorry for all the confusion I've added again :(
Roman Podoliaka (rpodolyaka) wrote : | #34 |
Please disregard all the comments on eventlet above - it seems to be red herring here.
description: | updated |
OSCI Robot (oscirobot) wrote : | #35 |
package python-urllib3 has been built for project packages/
Package version == 1.8.2, package release == 3
Changeset: https:/
project: packages/
branch: 6.1
author: Sergey Kolekonov
committer: Sergey Kolekonov
subject: Update urllib3 to version 1.8.2
status: patchset-created
Files placed on repository:
NOTE: Changeset is not merged, created temporary package repository.
repository URL: /centos-
OSCI Robot (oscirobot) wrote : | #36 |
package python-urllib3 has been built for project packages/
Package version == 1.7.1, package release == 2
Changeset: https:/
project: packages/
branch: 6.1
author: Sergey Kolekonov
committer: Sergey Kolekonov
subject: Update urllib3 to version 1.7.1
status: patchset-created
Files placed on repository:
NOTE: Changeset is not merged, created temporary package repository.
repository URL: /centos-
OSCI Robot (oscirobot) wrote : | #40 |
package python-urllib3 has been built for project packages/
Package version == 1.7.1, package release == 2
Changeset: https:/
project: packages/
branch: 6.1
author: Sergey Kolekonov
committer: Sergey Kolekonov
subject: Update urllib3 to version 1.7.1
status: change-merged
Files placed on repository:
Changeset merged. Package placed on primary repository
repository URL: /centos-
OSCI Robot (oscirobot) wrote : | #41 |
package python-requests has been built for project packages/
Package version == 2.2.1, package release == 2
Changeset: https:/
project: packages/
branch: 6.1
author: Sergey Kolekonov
committer: Sergey Kolekonov
subject: Updated python-requests to 2.2.1 version
status: patchset-created
Files placed on repository:
NOTE: Changeset is not merged, created temporary package repository.
repository URL: /centos-
OSCI Robot (oscirobot) wrote : | #44 |
package python-requests has been built for project packages/
Package version == 2.2.1, package release == 1
Changeset: https:/
project: packages/
branch: 6.1
author: Sergey Kolekonov
committer: Sergey Kolekonov
subject: Updated python-requests to 2.2.1 version
status: patchset-created
Files placed on repository:
NOTE: Changeset is not merged, created temporary package repository.
repository URL: /centos-
OSCI Robot (oscirobot) wrote : | #45 |
package python-requests has been built for project packages/
Package version == 2.2.1, package release == 1
Changeset: https:/
project: packages/
branch: 6.1
author: Sergey Kolekonov
committer: Sergey Kolekonov
subject: Updated python-requests to 2.2.1 version
status: change-merged
Files placed on repository:
Changeset merged. Package placed on primary repository
repository URL: /centos-
OSCI Robot (oscirobot) wrote : | #46 |
RPM package python-urllib3 has been built for project packages/
Package version == 1.7.1, package release == 2
Changeset: https:/
project: packages/
branch: 6.0.1
author: Ivan Berezovskiy
committer: Ivan Berezovskiy
subject: Update urllib3 to version 1.7.1
status: patchset-created
Files placed on repository:
python-
NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://
OSCI Robot (oscirobot) wrote : | #47 |
RPM package python-urllib3 has been built for project packages/
Package version == 1.7.1, package release == 2
Changeset: https:/
project: packages/
branch: 6.0.1
author: Ivan Berezovskiy
committer: Ivan Berezovskiy
subject: Update urllib3 to version 1.7.1
status: change-merged
Files placed on repository:
python-
Changeset merged. Package placed on primary repository
RPM repository URL: http://
OSCI Robot (oscirobot) wrote : | #48 |
RPM package python-requests has been built for project packages/
Package version == 2.2.1, package release == 1
Changeset: https:/
project: packages/
branch: 6.0.1
author: Ivan Berezovskiy
committer: Ivan Berezovskiy
subject: Updated python-requests to 2.2.1 version
status: patchset-created
Files placed on repository:
python-
NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://
OSCI Robot (oscirobot) wrote : | #49 |
RPM package python-requests has been built for project packages/
Package version == 2.2.1, package release == 1
Changeset: https:/
project: packages/
branch: 6.0.1
author: Ivan Berezovskiy
committer: Ivan Berezovskiy
subject: Updated python-requests to 2.2.1 version
status: change-merged
Files placed on repository:
python-
Changeset merged. Package placed on primary repository
RPM repository URL: http://
Yaroslav Lobankov (ylobankov) wrote : | #50 |
The fix for 6.1 is verified on
VERSION:
feature_groups:
- mirantis
production: "docker"
release: "6.1"
api: "1.0"
build_number: "159"
build_id: "2015-03-
nailgun_sha: "b18bd21c2a777c
python-
astute_sha: "81e7ca74722a46
fuellib_sha: "08cebf0b514b7f
ostf_sha: "b47bb2ad09e60a
fuelmain_sha: "baf24424a4e056
Yaroslav Lobankov (ylobankov) wrote : | #51 |
The fix for 6.0.1 is verified on
VERSION:
feature_groups:
- mirantis
production: "docker"
release: "6.0.1"
api: "1.0"
build_number: "117"
build_id: "2015-03-
astute_sha: "f7cda2171b0b67
fuellib_sha: "1c31c3f76c7ca7
ostf_sha: "3b57985d4d2155
nailgun_sha: "583e8331b622c8
fuelmain_sha: "bc1a1279509a87
Fuel Devops McRobotson (fuel-devops-robot) wrote : Related fix proposed to packages/centos6/python-requests (6.0-updates) | #52 |
Related fix proposed to branch: 6.0-updates
Change author: Denis V. Meltsaykin <email address hidden>
Review: https:/
Fuel Devops McRobotson (fuel-devops-robot) wrote : Related fix proposed to packages/centos6/python-urllib3 (6.0-updates) | #53 |
Related fix proposed to branch: 6.0-updates
Change author: Denis V. Meltsaykin <email address hidden>
Review: https:/
Fuel Devops McRobotson (fuel-devops-robot) wrote : Related fix merged to packages/centos6/python-urllib3 (6.0-updates) | #54 |
Reviewed: https:/
Submitter: Alex Ermolov <email address hidden>
Branch: 6.0-updates
Commit: cd880a9f2165ca4
Author: Denis V. Meltsaykin <email address hidden>
Date: Tue Jun 2 16:38:00 2015
Update urllib3 to version 1.7.1
Urllib3 has to be updated as a dependency for updated python-requests package
Change-Id: Ied5d7d9fc1e22b
Related-Bug: #1404983
Fuel Devops McRobotson (fuel-devops-robot) wrote : Related fix merged to packages/centos6/python-requests (6.0-updates) | #55 |
Reviewed: https:/
Submitter: Vitaly Sedelnik <email address hidden>
Branch: 6.0-updates
Commit: 2b78072c0914da6
Author: Denis V. Meltsaykin <email address hidden>
Date: Wed Jun 3 17:08:06 2015
Updated python-requests to 2.2.1 version
python-requests has to be updated due to bug with Heat
Change-Id: I953b167dff6e93
Related-bug: #1404983
tags: | added: customer-found support |
VERSION: 18_01-32- 01" a73030b8100cf8c 5ca6a970a91" 02c40a39885c502 ad1ebdf3eda" 570460c29d6c329 3219d3624d4" ff522ca9f6d34e7 e135f150a90" fbd9d60e134d722 29698b469c9"
feature_groups:
- mirantis
production: "docker"
release: "6.0"
api: "1.0"
build_number: "56"
build_id: "2014-12-
astute_sha: "16b252d93be6aa
fuellib_sha: "73332192a257ea
ostf_sha: "a9afb68710d809
nailgun_sha: "5f91157daa6798
fuelmain_sha: "45caacadb878ab