glance qpid notifier can hang under heavy load
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Glance |
Fix Released
|
Medium
|
Attila Fazekas | ||
Grizzly |
Fix Released
|
Medium
|
Flavio Percoco |
Bug Description
Glance qpid notifier can hang under heavy image creation load.
The issue happens because of two issse:
- The qpid notifier instance can be called by multiple green thread concurrently, one thread may recreate the connection object, while other thread working.
The connection object should be local variable instead of object variable, in order to avoid unwanted modification, or replacement.
The second issue the python-qpid uses PipeWaiter with a select.select. The select is not monkey patched to the be green thread friendly, is can causes hang issue.
All other AMQP user openstack competent monkey patches the thread and select modules, but glance not.
Usually good practice to make everything what is possible green thread/eventlet safe and make the application more preemptive.
cinder, neutron makes everything evenetlet friend, nova excludes the 'os' module.
I would recommend to make everything greenlet friendly.
Changed in glance: | |
status: | New → Triaged |
importance: | Undecided → Medium |
milestone: | none → havana-rc1 |
Changed in glance: | |
assignee: | nobody → Attila Fazekas (afazekas) |
status: | Triaged → In Progress |
tags: | added: grizzly-backport-potential |
Changed in glance: | |
status: | In Progress → Fix Committed |
Changed in glance: | |
status: | Fix Committed → Fix Released |
Changed in glance: | |
milestone: | havana-rc1 → 2013.2 |
tags: | removed: grizzly-backport-potential in-stable-grizzly |
Fix proposed to branch: master /review. openstack. org/47786
Review: https:/