OSTF fails trying to authenticate to HTTPS horizon

[Anastasia Palkina]

1. Create new environment (Ubuntu)
2. Choose Neutron, VLAN
3. Uncheck "Cinder LVM over iSCSI for volumes" on Settings tab
4. Add 1 controller, 1 compute
5. Deploy the environment. It was successful
6. But there is CRITICAL error on controller (node-1)

2015-08-18 12:06:28 CRIT (Scope(Concat::Fragment[horizon_balancermember_horizon])) No content, source or symlink specified

7. Run OSTF tests. Test "Create user and authenticate with it to Horizon" has failed on the last step "Confirm that response status is 200." with error: Can not authenticate in Horizon

From HealthCheck logs:

2015-08-18 13:46:00 ERROR (nose_storage_plugin) fuel_health.tests.smoke.test_user_create.TestUserTenantRole.test_create_user
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/unittest2/case.py", line 340, in run
    testMethod()
  File "/usr/lib/python2.6/site-packages/fuel_health/tests/smoke/test_user_create.py", line 136, in test_create_user
    self.fail("Step 10 failed: Can not authenticate in Horizon. "
  File "/usr/lib/python2.6/site-packages/unittest2/case.py", line 415, in fail
    raise self.failureException(msg)
AssertionError: Step 10 failed: Can not authenticate in Horizon. Please refer to OpenStack logs for more details.

I successfully login to Horizon manually

"build_id": "2015-08-18_03-05-20", "build_number": "187", "release_versions": {"2015.1.0-7.0": {"VERSION": {"build_id": "2015-08-18_03-05-20", "build_number": "187", "api": "1.0", "fuel-library_sha": "0062e69db17f8a63f85996039bdefa87aea498e1", "nailgun_sha": "4710801a2f4a6d61d652f8f1e64215d9dde37d2e", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "e01693992d7a0304d926b922b43f3b747c35964c", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "57145b1d8804389304cd04322ba0fb3dc9d30327", "production": "docker", "python-fuelclient_sha": "4c74a60aa60c06c136d9197c7d09fa4f8c8e2863", "astute_sha": "e24ca066bf6160bc1e419aaa5d486cad1aaa937d", "fuel-ostf_sha": "17786b86b78e5b66d2b1c15500186648df10c63d", "release": "7.0", "fuelmain_sha": "c9dad194e82a60bf33060eae635fff867116a9ce"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "0062e69db17f8a63f85996039bdefa87aea498e1", "nailgun_sha": "4710801a2f4a6d61d652f8f1e64215d9dde37d2e", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "e01693992d7a0304d926b922b43f3b747c35964c", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "57145b1d8804389304cd04322ba0fb3dc9d30327", "production": "docker", "python-fuelclient_sha": "4c74a60aa60c06c136d9197c7d09fa4f8c8e2863", "astute_sha": "e24ca066bf6160bc1e419aaa5d486cad1aaa937d", "fuel-ostf_sha": "17786b86b78e5b66d2b1c15500186648df10c63d", "release": "7.0", "fuelmain_sha": "c9dad194e82a60bf33060eae635fff867116a9ce"

[Alex Schultz (alex-schultz)]

Error from OSTF is related to ssl validation,

urllib3.connectionpool: INFO: Starting new HTTP connection (1): 10.20.0.3
urllib3.connectionpool: DEBUG: Setting read timeout to None
urllib3.connectionpool: DEBUG: "POST http://192.168.0.2:35357/v2.0/tokens HTTP/1.1" 200 4505
urllib3.connectionpool: INFO: Starting new HTTP connection (1): 10.20.0.3
urllib3.connectionpool: DEBUG: Setting read timeout to None
urllib3.connectionpool: DEBUG: "GET http://172.16.0.3/ HTTP/1.1" 302 0
urllib3.connectionpool: INFO: Starting new HTTPS connection (1): 172.16.0.3
fuel_health.tests.smoke.test_user_create: DEBUG: Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/fuel_health/tests/smoke/test_user_create.py", line 107, in test_create_user
    client.get(url) # sets cookie
  File "/usr/lib/python2.6/site-packages/requests/sessions.py", line 395, in get
    return self.request('GET', url, **kwargs)
  File "/usr/lib/python2.6/site-packages/requests/sessions.py", line 383, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.6/site-packages/requests/sessions.py", line 506, in send
    history = [resp for resp in gen] if allow_redirects else []
  File "/usr/lib/python2.6/site-packages/requests/sessions.py", line 168, in resolve_redirects
    allow_redirects=False,
  File "/usr/lib/python2.6/site-packages/requests/sessions.py", line 486, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.6/site-packages/requests/adapters.py", line 385, in send
    raise SSLError(e)
SSLError: [Errno 1] _ssl.c:492: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

[Alex Schultz (alex-schultz)]

Split out the haproxy log message to bug 1486076

[Alexander Kurenyshev (akurenyshev)]

This bug is not a duplicate of 1485633.

The 1485633 bug is a small trick which is not working without a support from OSTF side.

Does this error exist when SSL is switched off on the cluster?

[Anastasia Palkina (apalkina)]

Reproduced on ISO #219 for environment with 3 controllers and access to environment is https://10.20.0.2:8443/

"build_id": "2015-08-23_15-01-12", "build_number": "219", "release_versions": {"2015.1.0-7.0": {"VERSION": {"build_id": "2015-08-23_15-01-12", "build_number": "219", "api": "1.0", "fuel-library_sha": "3a3ea6d9849bc1ba35c1bd882f0a0678b20d2e51", "nailgun_sha": "7790ce872512ecdf21689e6a5f970dd7119febdb", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "e01693992d7a0304d926b922b43f3b747c35964c", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "4c2ab9d6c623d345086c6e2874d1df81fd96a942", "production": "docker", "python-fuelclient_sha": "fc7b63aa6900fe3b2c183108ba6a13e868bc0472", "astute_sha": "53c86cba593ddbac776ce5a3360240274c20738c", "fuel-ostf_sha": "16839cbf471b7142b04c0d2c2d94786bc486fefe", "release": "7.0", "fuelmain_sha": "a494e6628319abfef57e1754f6453cf8f1a4bc65"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "3a3ea6d9849bc1ba35c1bd882f0a0678b20d2e51", "nailgun_sha": "7790ce872512ecdf21689e6a5f970dd7119febdb", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "e01693992d7a0304d926b922b43f3b747c35964c", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "4c2ab9d6c623d345086c6e2874d1df81fd96a942", "production": "docker", "python-fuelclient_sha": "fc7b63aa6900fe3b2c183108ba6a13e868bc0472", "astute_sha": "53c86cba593ddbac776ce5a3360240274c20738c", "fuel-ostf_sha": "16839cbf471b7142b04c0d2c2d94786bc486fefe", "release": "7.0", "fuelmain_sha": "a494e6628319abfef57e1754f6453cf8f1a4bc65"

[Anastasia Palkina (apalkina)]

Logs are here https://drive.google.com/a/mirantis.com/file/d/0B6SjzarTGFxaWkRVbmVXVS1YSjQ/view?usp=sharing

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/216382

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/218952

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/218952
Committed: https://git.openstack.org/cgit/stackforge/fuel-ostf/commit/?id=582a81ccaa1e439a3aec4b8b8f6994735de840f4
Submitter: Jenkins
Branch: master

commit 582a81ccaa1e439a3aec4b8b8f6994735de840f4
Author: Tatyana Leontovich <email address hidden>
Date: Mon Aug 31 20:05:57 2015 +0300

    Disable step auth to horizon over https

    In case if horizon runs over https, step:
    login in horizon will not be run until
    https://bugs.launchpad.net/fuel/+bug/1486056 in progress

    Change-Id: Ia9997ae2b7180c4b3a52d9c6f501a6cbef27ba5b
    Related-Bug: #1486056

[Alex Schultz (alex-schultz)]

We have committed a fix to disable the tests if ssl is enabled for 7.0 as it is too late in the cycle to fix the underlying issue.
There are two issues here, 1) the ability of the fuel master to talk to the public vip and 2) leveraging the proxy to talk to the public vip is failing with a connection refused error. The proxy issue is caused by the underlying urllib3 version that is causing the https proxy connection to try and be reused for https. The connection cannot be reused and we would need to either backport a fix or upgrade to a newer version of urllib3. The change(s) in urllib3 that would address the proxy issue is:

 https://github.com/shazow/urllib3/pull/365
 https://github.com/shazow/urllib3/pull/369

As we are close to HCF, we can't risk the backport for 7.0. So lowering the issue as the test is now disabled and adding 8.0 to the bug.

[Alexander Kurenyshev (akurenyshev)]

Just for clarify:

we merged a code which disables the last step 10 from the test when SSL for Horizon is used.
Link to the review with trick: https://review.openstack.org/#/c/218952/
For 7.0 we'll not merge another fix, so I change status to Won't fix.

For 8.0 we need to choose a right solution as Alex S said.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/223643

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/223643
Committed: https://git.openstack.org/cgit/stackforge/fuel-docs/commit/?id=5146569e5f6c802d15482b9b630e37876be3cc24
Submitter: Jenkins
Branch: master

commit 5146569e5f6c802d15482b9b630e37876be3cc24
Author: Maria Zlatkova <email address hidden>
Date: Tue Sep 15 18:07:54 2015 +0300

    [RN 7.0] OSTF test resolved isuue

    This patch adds the following resolved issue to the Release
    Notes 7.0:

    - OSTF fails trying to authenticate to HTTPS horizon.

    Can be added to the Test and Verification Issues subsection.

    Change-Id: I7970dde1d5296a6144c4293f0ef1c6ca17d06292
    Related-Bug: #1486056

[OpenStack Infra (hudson-openstack)]

Change abandoned by Fuel DevOps Robot (<email address hidden>) on branch: master
Review: https://review.openstack.org/216382
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[Dmitry Teselkin (teselkin-d)]

Those fixes were included in 1.8.3, and it matches with global requirements for liberty (urllib3>=1.8.3) [1]
Target platform - CentOS

[1] https://github.com/openstack/requirements/blob/stable/liberty/global-requirements.txt#L206

[Dmitry Teselkin (teselkin-d)]

Please confirm this issue reproduced on CentOS 7 - there is python-urllib3 even newer:
---
python-urllib3-1.10.4-6.el7~mos8.0.2.noarch.rpm 02-Nov-2015 19:48 107720
---

[Alexander Kurenyshev (akurenyshev)]

I've checked this issue on Centos7 and for now test could successfully authenticate via proxy connection.
Updated and re-worked this review https://review.openstack.org/#/c/218342/2

[Alexander Kurenyshev (akurenyshev)]

I've checked this issue on Centos7 and for now test could successfully authenticate via proxy connection.
Updated and re-worked this review https://review.openstack.org/#/c/218342

[OpenStack Infra (hudson-openstack)]

Change abandoned by tatyana-leontovich (<email address hidden>) on branch: master
Review: https://review.openstack.org/218342
Reason: Create new one https://review.openstack.org/#/c/258697/

[OpenStack Infra (hudson-openstack)]

Change abandoned by tatyana-leontovich (<email address hidden>) on branch: master
Review: https://review.openstack.org/216382
Reason: Consolidate the changes in https://review.openstack.org/#/c/258697/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/261350

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/258697
Committed: https://git.openstack.org/cgit/openstack/fuel-ostf/commit/?id=05bc6366efd9739a992d084c093d5831595e52ec
Submitter: Jenkins
Branch: master

commit 05bc6366efd9739a992d084c093d5831595e52ec
Author: Tatyana Leontovich <email address hidden>
Date: Thu Dec 17 00:46:01 2015 +0200

    Switch OSTF to work with public endpoint

    To be sure that OSTF works fine with SSL,
    switch to use public endpoints without
    validation of certificate

    Change-Id: I87a625899c1eb4b80376c8bdb5aca4fce3d8ad66
    Closes-Bug: #1486056
    Related-Bug: #1524850

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/261350
Committed: https://git.openstack.org/cgit/openstack/fuel-ostf/commit/?id=889ddb0f1a4fa5f839fd4ea0c0017a3c181aa0c1
Submitter: Jenkins
Branch: stable/8.0

commit 889ddb0f1a4fa5f839fd4ea0c0017a3c181aa0c1
Author: Tatyana Leontovich <email address hidden>
Date: Thu Dec 17 00:46:01 2015 +0200

    Switch OSTF to work with public endpoint

    To be sure that OSTF works fine with SSL,
    switch to use public endpoints without
    validation of certificate

    Change-Id: I87a625899c1eb4b80376c8bdb5aca4fce3d8ad66
    Closes-Bug: #1486056
    Related-Bug: #1524850

[Tatyanka (tatyana-leontovich)]

verified 361 iso