Handle database disconnections gracefully
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Storm |
Fix Released
|
High
|
James Henstridge |
Bug Description
For any application that keeps database connections open for extended periods of time, there is a risk that the connection will be broken (e.g. database server reboots, network issues, etc). It would be nice if Storm could help applications respond appropriately to such situations.
In the case of a database connection failure, I'd expect storm to act the same as if the transaction has been doomed. That is, all requests fail until we abort() the transaction. On the start of the next transaction, there are two possibilities:
1. retry connection to the db until we can connect (possibly with exponential back off), and then proceed with the new transaction.
2. try and connect to the db again. If we fail, act as if this transaction has also been doomed.
I think there is value in providing both these modes: (1) is good for scripts where we want to get the job done eventually, while (2) is better for interactive uses like web applications where leaving the user hanging for minutes is as bad or worse than erroring out early.
Related branches
- No reviews requested
Changed in storm: | |
assignee: | nobody → jamesh |
importance: | Undecided → High |
status: | New → Confirmed |
Changed in storm: | |
status: | Confirmed → In Progress |
For web application requests, we already raise a Retry exception in the case of serialization or conflict errors. This causes the web application publisher to retry the entire request up to five times. On the sixth time, an error is presented to the user. We should do the same for database disconnections: doom the transaction, and make it raise a Retry to the publisher.