c-api times out devstack waiting to start
Bug #1722326 reported by
Matt Riedemann
This bug affects 1 person
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Cinder |
Fix Released
|
Medium
|
Matt Riedemann |
Bug Description
We're seeing CI failures where c-api takes >60 seconds to start:
The devstack timeout is 60 seconds, and in this case it took 62 seconds for c-api to start serving requests:
Oct 09 15:10:14.061521 ubuntu-
Compare to n-api which takes 8 seconds to start:
Oct 09 15:05:30.335892 ubuntu-
So what is cinder-api doing on startup that takes so long? Can any of it be run in the background?
tags: | added: api performance |
Changed in cinder: | |
assignee: | Matt Riedemann (mriedem) → Sean McGinnis (sean-mcginnis) |
Changed in cinder: | |
assignee: | Sean McGinnis (sean-mcginnis) → Matt Riedemann (mriedem) |
Changed in cinder: | |
importance: | Undecided → Medium |
To post a comment you must log in.
I see "Initializing extension manager." logged 6 times on the same process.
There are two API workers:
osapi_volume_ workers = 2
So that's loading extensions for v1, v2 and v3 on 2 workers, so that's why extensions are loaded 6 times.
--
I'm seeing this in the logs:
Oct 09 15:10:13.364929 ubuntu- xenial- inap-mtl01- 11290241 <email address hidden>[21361]: result['domain'] = self.domain xenial- inap-mtl01- 11290241 <email address hidden>[21361]: INFO cinder.rpc [None req-2447e5c0- eea2-42c6- 9b11-958e3dbc9f a2 None None] Automatically selected cinder-scheduler objects version 1.27 as minimum service version. xenial- inap-mtl01- 11290241 <email address hidden>[21361]: INFO cinder.rpc [None req-2447e5c0- eea2-42c6- 9b11-958e3dbc9f a2 None None] Automatically selected cinder-scheduler RPC version 3.7 as minimum service version. xenial- inap-mtl01- 11290241 <email address hidden>[21361]: INFO cinder.rpc [None req-2447e5c0- eea2-42c6- 9b11-958e3dbc9f a2 None None] Automatically selected cinder-volume objects version 1.27 as minimum service version. xenial- inap-mtl01- 11290241 <email address hidden>[21361]: INFO cinder.rpc [None req-2447e5c0- eea2-42c6- 9b11-958e3dbc9f a2 None None] Automatically selected cinder-volume RPC version 3.15 as minimum service version.
Oct 09 15:10:13.529425 ubuntu-
Oct 09 15:10:13.533670 ubuntu-
Oct 09 15:10:13.538091 ubuntu-
Oct 09 15:10:13.542528 ubuntu-
Which looks related to a "work_cleanup" blocking RPC call from the API to the scheduler, which is also going to hit the database. That looks bad.