TypeError in oslo.messaging logs during failover

[Ilya Shakhat]

During failover oslo.messaging reports TypeError with the following traceback:
2015-08-20 07:42:10.456 26589 ERROR oslo_messaging._drivers.impl_rabbit [-] Declaring queue failed with ((0, 0): (541) INTERNAL_ERROR), retrying
2015-08-20 07:42:10.464 26589 ERROR oslo_messaging._drivers.impl_rabbit [-] Failed to consume message from queue: connection already closed
2015-08-20 07:42:10.469 26589 ERROR root [-] Unexpected exception occurred 1 time(s)... retrying.
2015-08-20 07:42:10.469 26589 TRACE root Traceback (most recent call last):
2015-08-20 07:42:10.469 26589 TRACE root File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 95, in inner_func
2015-08-20 07:42:10.469 26589 TRACE root return infunc(*args, **kwargs)
2015-08-20 07:42:10.469 26589 TRACE root File "/usr/lib/python2.7/dist-packages/oslo_messaging/_executors/impl_eventlet.py", line 96, in _executor_thread
2015-08-20 07:42:10.469 26589 TRACE root incoming = self.listener.poll()
2015-08-20 07:42:10.469 26589 TRACE root File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 122, in poll
2015-08-20 07:42:10.469 26589 TRACE root self.conn.consume(limit=1, timeout=timeout)
2015-08-20 07:42:10.469 26589 TRACE root File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1201, in consume
2015-08-20 07:42:10.469 26589 TRACE root six.next(it)
2015-08-20 07:42:10.469 26589 TRACE root File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1112, in iterconsume
2015-08-20 07:42:10.469 26589 TRACE root error_callback=_error_callback)
2015-08-20 07:42:10.469 26589 TRACE root File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 915, in ensure
2015-08-20 07:42:10.469 26589 TRACE root 'retry': retry}
2015-08-20 07:42:10.469 26589 TRACE root TypeError: %d format: a number is required, not NoneType
2015-08-20 07:42:10.469 26589 TRACE root

From the code it looks like the error doesn't affect functionality, however it indicates that parameter is in unexpected state.

[Ilya Shakhat (shakhat)]

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "7.0"
  openstack_version: "2015.1.0-7.0"
  api: "1.0"
  build_number: "189"
  build_id: "2015-08-18_17-24-26"
  nailgun_sha: "d0b727cdd0d8e7ce5e17e6ea1306d835bfdfb5e7"
  python-fuelclient_sha: "6ad5e0eb4dbbca6cade3444554606927ecd3f16f"
  fuel-agent_sha: "57145b1d8804389304cd04322ba0fb3dc9d30327"
  fuel-nailgun-agent_sha: "e01693992d7a0304d926b922b43f3b747c35964c"
  astute_sha: "e24ca066bf6160bc1e419aaa5d486cad1aaa937d"
  fuel-library_sha: "7c80eed2119260cc15a700068b9eb20ccc773926"
  fuel-ostf_sha: "235f21b230fea15724d625b2dc44ade0464527e2"
  fuelmain_sha: "c9dad194e82a60bf33060eae635fff867116a9ce"

[Ilya Shakhat (shakhat)]

The bug affects supportability of the product. Thus setting priority to High.

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: Victor Sergeyev <email address hidden>
Review: https://review.fuel-infra.org/10604

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/10604
Submitter: mos-infra-ci <>
Branch: openstack-ci/fuel-7.0/2015.1.0

Commit: e81cbe0e73d38773317f74f55c580ce9fac19187
Author: Victor Sergeyev <email address hidden>
Date: Thu Aug 20 13:00:49 2015

Fix error message placeholders

The error message in Connection.ensure() expects to get integer for
string formatting in values `retry` and `port`. But it seems to be, that
during failover these variables can get None values, so TypeError will
be raised in this case.

Replaced "%d" to "%s" to avoid of the TypeError.

Closes-Bug: #1486995

Change-Id: I401c8a458041893794b4a839d502055eb5f40490

[Oleksiy Butenko (obutenko)]

verified on MOS 7.0 ISO 265
{"build_id": "265", "build_number": "265", "release_versions": {"2015.1.0-7.0": {"VERSION": {"build_id": "265", "build_number": "265", "api": "1.0", "fuel-library_sha": "4fdf3d6b070204366593012428395d173698678a", "nailgun_sha": "0dfcf73deb8ae99654f3da2ea95b7b68b9ee7273", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "d7027952870a35db8dc52f185bb1158cdd3d1ebd", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "082a47bf014002e515001be05f99040437281a2d", "production": "docker", "python-fuelclient_sha": "9643fa07f1290071511066804f962f62fe27b512", "astute_sha": "e63709d16bd4c1949bef820ac336c9393c040d25", "fuel-ostf_sha": "582a81ccaa1e439a3aec4b8b8f6994735de840f4", "release": "7.0", "fuelmain_sha": "9ab01caf960013dc882825dc9b0e11ccf0b81cb0"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "4fdf3d6b070204366593012428395d173698678a", "nailgun_sha": "0dfcf73deb8ae99654f3da2ea95b7b68b9ee7273", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "d7027952870a35db8dc52f185bb1158cdd3d1ebd", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "082a47bf014002e515001be05f99040437281a2d", "production": "docker", "python-fuelclient_sha": "9643fa07f1290071511066804f962f62fe27b512", "astute_sha": "e63709d16bd4c1949bef820ac336c9393c040d25", "fuel-ostf_sha": "582a81ccaa1e439a3aec4b8b8f6994735de840f4", "release": "7.0", "fuelmain_sha": "9ab01caf960013dc882825dc9b0e11ccf0b81cb0"}

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-8.0/liberty
Change author: Victor Sergeyev <email address hidden>
Review: https://review.fuel-infra.org/13462

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Dmitry Mescheryakov <email address hidden> on branch: openstack-ci/fuel-8.0/liberty
Review: https://review.fuel-infra.org/13462
Reason: Fixed in stable/liberty by https://review.openstack.org/#/c/180453/