test_notifications fails with (403) ACCESS_REFUSED

Bug #1721202 reported by Rabi Mishra
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Heat
Triaged
High
Unassigned

Bug Description

traceback:

2017-10-04 05:50:11.595244 | 2017-10-04 05:50:11.594 | heat_integrationtests.functional.test_notifications.NotificationTest.test_asg_notifications
2017-10-04 05:50:11.596469 | 2017-10-04 05:50:11.596 | -------------------------------------------------------------------------------------------
2017-10-04 05:50:11.597613 | 2017-10-04 05:50:11.597 |
2017-10-04 05:50:11.598701 | 2017-10-04 05:50:11.598 | Captured traceback:
2017-10-04 05:50:11.599842 | 2017-10-04 05:50:11.599 | ~~~~~~~~~~~~~~~~~~~
2017-10-04 05:50:11.601100 | 2017-10-04 05:50:11.600 | b'Traceback (most recent call last):'
2017-10-04 05:50:11.602252 | 2017-10-04 05:50:11.601 | b' File "/opt/stack/new/heat/heat_integrationtests/functional/test_notifications.py", line 133, in setUp'
2017-10-04 05:50:11.603368 | 2017-10-04 05:50:11.603 | b' self.ch = self.conn.channel()'
2017-10-04 05:50:11.604490 | 2017-10-04 05:50:11.604 | b' File "/usr/local/lib/python3.5/dist-packages/kombu/connection.py", line 266, in channel'
2017-10-04 05:50:11.605668 | 2017-10-04 05:50:11.605 | b' chan = self.transport.create_channel(self.connection)'
2017-10-04 05:50:11.606883 | 2017-10-04 05:50:11.606 | b' File "/usr/local/lib/python3.5/dist-packages/kombu/connection.py", line 802, in connection'
2017-10-04 05:50:11.608019 | 2017-10-04 05:50:11.607 | b' self._connection = self._establish_connection()'
2017-10-04 05:50:11.609214 | 2017-10-04 05:50:11.608 | b' File "/usr/local/lib/python3.5/dist-packages/kombu/connection.py", line 757, in _establish_connection'
2017-10-04 05:50:11.610297 | 2017-10-04 05:50:11.610 | b' conn = self.transport.establish_connection()'
2017-10-04 05:50:11.611467 | 2017-10-04 05:50:11.611 | b' File "/usr/local/lib/python3.5/dist-packages/kombu/transport/pyamqp.py", line 130, in establish_connection'
2017-10-04 05:50:11.612619 | 2017-10-04 05:50:11.612 | b' conn.connect()'
2017-10-04 05:50:11.613744 | 2017-10-04 05:50:11.613 | b' File "/usr/local/lib/python3.5/dist-packages/amqp/connection.py", line 288, in connect'
2017-10-04 05:50:11.614858 | 2017-10-04 05:50:11.614 | b' self.drain_events(timeout=self.connect_timeout)'
2017-10-04 05:50:11.616016 | 2017-10-04 05:50:11.615 | b' File "/usr/local/lib/python3.5/dist-packages/amqp/connection.py", line 471, in drain_events'
2017-10-04 05:50:11.617048 | 2017-10-04 05:50:11.616 | b' while not self.blocking_read(timeout):'
2017-10-04 05:50:11.618154 | 2017-10-04 05:50:11.617 | b' File "/usr/local/lib/python3.5/dist-packages/amqp/connection.py", line 477, in blocking_read'
2017-10-04 05:50:11.619532 | 2017-10-04 05:50:11.619 | b' return self.on_inbound_frame(frame)'
2017-10-04 05:50:11.620611 | 2017-10-04 05:50:11.620 | b' File "/usr/local/lib/python3.5/dist-packages/amqp/method_framing.py", line 55, in on_frame'
2017-10-04 05:50:11.621626 | 2017-10-04 05:50:11.621 | b' callback(channel, method_sig, buf, None)'
2017-10-04 05:50:11.622648 | 2017-10-04 05:50:11.622 | b' File "/usr/local/lib/python3.5/dist-packages/amqp/connection.py", line 481, in on_inbound_method'
2017-10-04 05:50:11.623670 | 2017-10-04 05:50:11.623 | b' method_sig, payload, content,'
2017-10-04 05:50:11.624744 | 2017-10-04 05:50:11.624 | b' File "/usr/local/lib/python3.5/dist-packages/amqp/abstract_channel.py", line 128, in dispatch_method'
2017-10-04 05:50:11.625770 | 2017-10-04 05:50:11.625 | b' listener(*args)'
2017-10-04 05:50:11.626910 | 2017-10-04 05:50:11.626 | b' File "/usr/local/lib/python3.5/dist-packages/amqp/connection.py", line 603, in _on_close'
2017-10-04 05:50:11.627908 | 2017-10-04 05:50:11.627 | b' (class_id, method_id), ConnectionError)'
2017-10-04 05:50:11.629199 | 2017-10-04 05:50:11.628 | b'amqp.exceptions.AccessRefused: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile.'
2017-10-04 05:50:11.630836 | 2017-10-04 05:50:11.630 | b''
2017-10-04 05:50:11.632716 | 2017-10-04 05:50:11.632 |

Noticed at:

http://logs.openstack.org/32/509132/1/check/gate-heat-dsvm-functional-convg-mysql-lbaasv2-py35-ubuntu-xenial/6dba619/console.html#_2017-10-04_05_50_11_629199

rabbit log:
=ERROR REPORT==== 4-Oct-2017::05:51:51 ===
closing AMQP connection <0.3728.0> (127.0.0.1:45128 -> 127.0.0.1:5672):
{handshake_error,starting,0,
                 {amqp_error,access_refused,
                             "AMQPLAIN login refused: user 'guest' - invalid credentials",
                             'connection.start_ok'}}

http://logs.openstack.org/92/507792/1/check/gate-heat-dsvm-functional<email address hidden>

It's using the oslo_config defaults and it seems recent changes don't allow that.

Tags: gate-failure
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to heat (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/509367

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to heat (master)

Reviewed: https://review.openstack.org/509367
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=9762cc6ac44ed04e98eb2514775813c777404236
Submitter: Jenkins
Branch: master

commit 9762cc6ac44ed04e98eb2514775813c777404236
Author: rabi <email address hidden>
Date: Wed Oct 4 13:23:51 2017 +0530

    Skip test_notifications.NotificationTest.* functional tests

    Change-Id: I4cb4b03adef132c11ca664676b6e03ad3efe60d2
    Related-Bug: #1721202

Zane Bitter (zaneb)
Changed in heat:
milestone: none → queens-rc1
status: New → Triaged
Rabi Mishra (rabi)
Changed in heat:
importance: High → Medium
Rabi Mishra (rabi)
Changed in heat:
milestone: queens-rc1 → rocky-1
Zane Bitter (zaneb)
Changed in heat:
importance: Medium → High
tags: added: gate-failure
Rico Lin (rico-lin)
Changed in heat:
milestone: rocky-1 → rocky-2
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.