Leases stuck in PENDING indefinitely
Bug #1845531 reported by
Jason Anderson
This bug affects 2 people
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| Blazar |
Fix Released
|
Critical
|
Jacob Colleran | ||
Bug Description
We've encountered a situation several times wherein leases are stuck in the PENDING state even though their start_time has passed. There are no errors or logs in Blazar to indicate a problem; it seems that the events processor simply stops running or it is silently erroring out.
This seems to happen when the connection to the MySQL/MariaDB database is interrupted for some reason; in our case, it happens whenever we have to restart the DB for maintenance or if connectivity temporarily drops.
Restarting blazar-manager will fix it.
| Changed in blazar: | |
| importance: | Undecided → Critical |
| assignee: | nobody → Jacob Colleran (jcolleran) |
| milestone: | none → ussuri-3 |
Revision history for this message
| OpenStack Infra (hudson-openstack) wrote Fix merged to blazar (master) | #1 |
| Changed in blazar: | |
| status: | In Progress → Fix Released |
To post a comment you must log in.
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: master
commit e34923843f5c888
Author: jakecoll <email address hidden>
Date: Thu Mar 19 15:26:35 2020 -0500
Keep service threadgroups alive on exception
If the db connection is severed, blazar-manager service fails to
re-establish the connection because the timer for worker threadgroups
are set to fail on any exception. If we set the timer to continue
running after an exception then blazar can attempt to re-establish
connection to a live database.
TODO: Find a way to test this.
Change-Id: I019d10b22c471e
Closes-Bug: #1845531