Message listener error in nova-powervc causes PowerVC qpid broker hung

[Yi Ming Yin]

Current message listener code in powervc driver cannot parse JSON format event. It will cause listener thread died without any log recorded. The more severe problem is such dead listener thread will cause no message fetched from qpid broker in PowerVC. As time goes, the listener queue backlog will grow longer and longer. And finally, cause the whole qpid broker failed to response. It only allows a certain amount of messages to be backed up.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/109911

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/109911
Committed: https://git.openstack.org/cgit/stackforge/powervc-driver/commit/?id=f1fa4ba5ea2fa432b31a2c8c9f10e286d5f0607e
Submitter: Jenkins
Branch: master

commit f1fa4ba5ea2fa432b31a2c8c9f10e286d5f0607e
Author: yinym <email address hidden>
Date: Mon Jul 28 14:05:20 2014 +0800

    Fix message listening issue in common component

    Current message listening thread will die when it received unsupported
    message format (e.g. application/json). The issue will cause remote qpid
    broker backup all messages for this listener forever. When such backlog
    comes to a limit, qpid broker will refuse to put through new messages.

    Fix changes this behavior with following changes:
    1. Add logic to parse application/json format messages.
    2. Add extra path to ignore unsupported message format
    3. Catch unexpected exceptions in listener loop to avoid unexpected
    listener termination.

    Change-Id: I666c596f5725df25b548d82c733aa853fce7147c
    Closes-Bug: #1349227