[ostf] Ostf tests which use Aodh fail because ostf api proxy does not have access to query aodh endpoint

Bug #1566371 reported by Artem Minasyan
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
Critical
Ivan Berezovskiy
Mitaka
Fix Released
Critical
Ivan Berezovskiy

Bug Description

All OSTF tests which have connection to the Aodh API during the testing failed. The general error for these tests is *ConnectionError: HTTPSConnectionPool(host='public.fuel.local', port=8042): Max retries exceeded with url: /v2/alarms (Caused by ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 403 Proxy Error',)))*

We think it is caused by lack of AODH port on the Allow ports for the apache proxy which is used for the OSTF. This parameters is configured in /etc/apache2/sites-available/25-apache_api_proxy.conf. Also lack of iptables rules for this port on master node could cause an error.

Proposed fix: to add port "8042" (it's an Aodh port) in Allow ports and add rules for 8042 using into iptables on the master node if it is needed.

Traces:
fuel_health.common.test_mixins: INFO: STEP:2, verify action: 'getting list of alarms'
fuel_health.common.test_mixins: DEBUG: Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/fuel_health/common/test_mixins.py", line 177, in verify
    result = func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/ceilometerclient/v2/alarms.py", line 83, in list
    return self._list(options.build_url(self._path(), q))
  File "/usr/lib/python2.7/site-packages/ceilometerclient/common/base.py", line 63, in _list
    resp = self.api.get(url)
  File "/usr/lib/python2.7/site-packages/ceilometerclient/openstack/common/apiclient/client.py", line 355, in get
    return self.client_request("GET", url, **kwargs)
  File "/usr/lib/python2.7/site-packages/ceilometerclient/openstack/common/apiclient/client.py", line 345, in client_request
    self, method, url, **kwargs)
  File "/usr/lib/python2.7/site-packages/ceilometerclient/openstack/common/apiclient/client.py", line 261, in client_request
    method, self.concat_url(endpoint, url), **kwargs)
  File "/usr/lib/python2.7/site-packages/ceilometerclient/openstack/common/apiclient/client.py", line 189, in request
    resp = self.http.request(method, url, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 468, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 597, in send
    history = [resp for resp in gen] if allow_redirects else []
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 195, in resolve_redirects
    **adapter_kwargs
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 423, in send
    raise ConnectionError(e, request=request)
ConnectionError: HTTPSConnectionPool(host='public.fuel.local', port=8042): Max retries exceeded with url: /v2/alarms (Caused by ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 403 Proxy Error',)))

Changed in fuel:
assignee: nobody → Fuel Library Team (fuel-library)
importance: Undecided → Critical
description: updated
Ilya Tyaptin (ityaptin)
summary: - [ceilometer aodh fuel-library]Ostf tests which use Aodh are failed
+ [ostf] Ostf tests which use Aodh are failed
tags: added: aodh ceilometer fuel-library
description: updated
Revision history for this message
Ilya Tyaptin (ityaptin) wrote : Re: [ostf] Ostf tests which use Aodh are failed

This issue ruined SWARM tests which uses deployed Ceilometer and Aodh.

description: updated
Changed in fuel:
milestone: none → 9.0
Revision history for this message
Bug Checker Bot (bug-checker) wrote : Autochecker

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

actual result

version

expected result

steps to reproduce

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

tags: added: need-info
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/301769

Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Ivan Berezovskiy (iberezovskiy)
status: New → In Progress
Revision history for this message
Andrey Maximov (maximov) wrote : Re: [ostf] Ostf tests which use Aodh are failed

Removing fuel-library and adding mos-area since it affects deployment openstack components

tags: added: area-mos
removed: fuel-library
tags: added: swarm-blocker
summary: - [ostf] Ostf tests which use Aodh are failed
+ [ostf] Ostf tests which use Aodh fail because ostf api proxy does not
+ have access to query aodh endpoint
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/301769
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=66323c18e6394b0aff330d11e985d6dae24d625f
Submitter: Jenkins
Branch: master

commit 66323c18e6394b0aff330d11e985d6dae24d625f
Author: iberezovskiy <email address hidden>
Date: Tue Apr 5 18:36:31 2016 +0300

    Allow connections to Aodh API

    * Add Aodh API port to list of opened for connections
      from master node.
    * Add ability to override this list from hiera.

    Closes-bug: #1566371

    Change-Id: Ia9c0e141c340e63147f5f3123eaca2a487361cbc

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/302668

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/mitaka)

Reviewed: https://review.openstack.org/302668
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=7171603f6bacdd8435c6632865c2daf5d161d96e
Submitter: Jenkins
Branch: stable/mitaka

commit 7171603f6bacdd8435c6632865c2daf5d161d96e
Author: iberezovskiy <email address hidden>
Date: Tue Apr 5 18:36:31 2016 +0300

    Allow connections to Aodh API

    * Add Aodh API port to list of opened for connections
      from master node.
    * Add ability to override this list from hiera.

    Closes-bug: #1566371

    Change-Id: Ia9c0e141c340e63147f5f3123eaca2a487361cbc
    (cherry picked from commit 66323c18e6394b0aff330d11e985d6dae24d625f)

Revision history for this message
Volodymyr Shypyguzov (vshypyguzov) wrote :

Reproduced again on iso #188

Revision history for this message
Ivan Berezovskiy (iberezovskiy) wrote :

188 ISO doesn't contain this fix

Revision history for this message
Artem Minasyan (aminasyan) wrote :
Dmitry Pyzhov (dpyzhov)
no longer affects: fuel/newton
Changed in fuel:
status: In Progress → Fix Committed
milestone: 9.0 → 10.0
tags: added: on-verification
Revision history for this message
Ekaterina Shutova (eshutova) wrote :

Verified on on 10.0 snapshot #1435.
https://product-ci.infra.mirantis.net/view/10.0_swarm/job/10.0.system_test.ubuntu.services_ha.ceilometer/201/
Most OSFTs are passed only 1 OSTF test is failed with another error. So original issue is fixed.

tags: removed: on-verification
Changed in fuel:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.