check_rabbitmq.py ValueError: can't have unbuffered text I/O

[Xav Paice]

On upgrade to cs:rabbitmq-server-83 (from -82, and from 18.08), the nrpe check_rabbitmq.py reports first "CRITICAL: amqplib not found". I installed python3-amqplib and that got solved, note that python-amqplib is installed.

Once that was fixed, then I got:

/usr/local/lib/nagios/plugins/check_rabbitmq.py --user nagios-rabbitmq-server-0 --password somepassword --vhost nagios-rabbitmq-server-0
Traceback (most recent call last):
  File "/usr/local/lib/nagios/plugins/check_rabbitmq.py", line 248, in <module>
    options.ssl_ca)
  File "/usr/local/lib/nagios/plugins/check_rabbitmq.py", line 184, in main
    sys.stdout = os.fdopen(os.dup(1), "w", 0)
  File "/usr/lib/python3.5/os.py", line 1072, in fdopen
    return io.open(fd, *args, **kwargs)
ValueError: can't have unbuffered text I/O

[Xav Paice (xavpaice)]

the lack of python3-amqp is because in the config-changed hook:

    if rabbit.archive_upgrade_available():
        # Avoid packge upgrade collissions
        # Stopping and attempting to start rabbitmqs at the same time leads to
        # failed restarts
        rabbit.cluster_wait()
        rabbit.install_or_upgrade_packages()

This unfortunately means if we change the var PACKAGES in rabbit_utils.py in a charm upgrade, nothing triggers the installs.

[Alex Kavanagh (ajkavanagh)]

Yes, so the first issue is caused by a python2 to python3 upgrade, where the script check_rabbitmq.py needs the PY3 package, but it doesn't get installed as only the charm is being upgraded. We might need to special-case this condition.

[Alex Kavanagh (ajkavanagh)]

Hi @xavprice,

I've done some updates to the charm, but before committing them, I was wondering if you could perhaps test the charm in an upgrade from PY2 to PY3 to make sure it fixes all of your issues?

I've pushed a candidate to: cs:~ajkavanagh/rabbitmq-server-0

Many thanks
Alex.

[Xav Paice (xavpaice)]

@ajkavanagh thanks - I've tested the candidate and it looks good - alerts are all clear now. Thanks for working on that!

At some stage we should consider how on earth to add tests for things like this, there's a bunch of scripts in a bunch of charms that don't hit the usual charm unit tests.

[Alex Kavanagh (ajkavanagh)]

@xavprice, excellent news; thanks for doing that. I'll get the changes through gerrit and then backport to stable/18.11. Yes, I agree, we do need to get a test (or tests) created to actually catch these errors. These got through all of our CI and released without us knowing. I'll raise a separate bug (against rabbitmq-server charm) to track that.

Thanks again.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/619227
Committed: https://git.openstack.org/cgit/openstack/charm-rabbitmq-server/commit/?id=065914d696ffb39c77696925b55b9795147cae00
Submitter: Zuul
Branch: master

commit 065914d696ffb39c77696925b55b9795147cae00
Author: Alex Kavanagh <email address hidden>
Date: Wed Nov 21 12:32:36 2018 +0000

    Fix more PY3 issues in check_rabbitmq_queues.py

    Note that, for PY3, the unbuffered IO in the check_rabbitmq.py script
    has to be switched off as PY3 doesn't support unbuffered IO for text
    (think str) rather than binary byte streams.

    Change-Id: I79af54cdbd7381f88732b24f65e94451350b9ab9
    Closes-Bug: #1804349
    Closes-Bug: #1804348

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/18.11
Review: https://review.openstack.org/619564

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/619564
Committed: https://git.openstack.org/cgit/openstack/charm-rabbitmq-server/commit/?id=0a8455b0dfd07c017ed2d196c2ac73d68262c5df
Submitter: Zuul
Branch: stable/18.11

commit 0a8455b0dfd07c017ed2d196c2ac73d68262c5df
Author: Alex Kavanagh <email address hidden>
Date: Wed Nov 21 12:32:36 2018 +0000

    Fix more PY3 issues in check_rabbitmq_queues.py

    Note that, for PY3, the unbuffered IO in the check_rabbitmq.py script
    has to be switched off as PY3 doesn't support unbuffered IO for text
    (think str) rather than binary byte streams.

    Closes-Bug: #1804349
    Closes-Bug: #1804348
    (cherry picked from commit 065914d696ffb39c77696925b55b9795147cae00)

    Change-Id: I928c2e30d0025d0d1901dd85051e8e72dcd21a07