RFE: Introduce Error handling for ConnectionError

[Andreas Scheuring]

A ConnectionError can occur every time when interacting with the HMC. A concept on how to deal with such errors is needed.

The following is thinkable

- Retry mechanism - retries a certain command up to x times. x is configurable via config option
- Timeout mechanism - during this time (configureable), the error will be caught
  * The agent will be reported as down, but continues running until HMC is back or the timeout hit. Then it terminates
  * resources will be reported as up until the timeout hits. After that the agent gets terminated
  * resources will be reported as down until the timeout hits. After that the agent terminates

We need to carefully think which options make sense! Maybe all, maybe just one of them, maybe something totally different.

[Markus Zoeller (markus_z) (mzoeller)]

Doesn't block the release, but we might want to backport it when it's done in Pike.

[Markus Zoeller (markus_z) (mzoeller)]

That's a design we haven't talked about at the beginning. We need to do that early in Pike.

[Andreas Maier (maiera)]

zhmcclient v0.11.0 introduced connection retries. So if the zhmcclient now raises a ConnectionError, that means these retries are exhausted. The connect retries and connect timeouts can be configured using a new RetryTimeoutConfig class. See http://python-zhmcclient.readthedocs.io/en/latest/general.html#session

[Andreas Maier (maiera)]

Also, the zhmcclient has always had support for automatic initial logon and re-logon upon expiration of an HMC session. We did not test whether the re-logon works across an HMC reboot, though.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/480097

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/480158