no config for multiple conductor workers

Bug #1213080 reported by Peter Feiner
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Wishlist
Peter Feiner

Bug Description

Nova-conductor processes access the database directly. Since the database connection driver is typically implemented in a library beyond the purview of eventlet’s monkeypatching (i.e., a native python extension like _mysql.so), blocking database calls will block all eventlet coroutines. Since much of what nova-conductor does is access the database, a nova-conductor process’s handling of requests is effectively serial.

To mitigate this problem, you can simply run more nova-conductor processes. Deploying multiple conductor workers per host avoids serialization on database accesses caused by libmysqlclient.so blocking eventlet's single thread. In an experiment on a 24-core machine, when creating 20 VMs in parallel, maximum creation time was reduced by approx. 10s when using 20 conductor processes vis-a-vis a single conductor process. Profiling showed that all of the savings came from faster calls into nova.db.sqlalchemy.api.

To make running multiple nova-conductor processes straightforward, there should be a workers=N option in the [conductor] section of nova.conf -- just like the osapi_compute_workers=N flag in the [DEFAULT] section.

Tags: conductor db
Revision history for this message
David Ripton (dripton) wrote :

Makes sense. Are you planning to submit a patch?

Changed in nova:
importance: Undecided → Wishlist
status: New → Confirmed
assignee: nobody → David Ripton (dripton)
assignee: David Ripton (dripton) → nobody
Revision history for this message
Peter Feiner (pete5) wrote :

Ah yes. I thought gerrit would add a comment automatically.

https://review.openstack.org/#/c/42342/

Changed in nova:
assignee: nobody → Peter Feiner (pete5)
Revision history for this message
Davanum Srinivas (DIMS) (dims-v) wrote :
Changed in nova:
status: Confirmed → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.