Artifact creation in GLARE using public endpoint that is under SSL failed with CommunicationError: [SSL: UNKNOWN_PROTOCOL] unknown protocol (_ssl.c:765)

[Victor Ryzhenkin]

Detailed bug description:
 During the test, which introduced in https://review.openstack.org/#/c/305951/ , the murano package should be uploaded into Glance Artifact repository. All requests to GLARE works fine, except uploading archive via PUT http method.
Steps to reproduce:
 1. Deploy build 470+ with SSL, Murano and GLARE backend.
 2. Run Murano OSTF platform test.
Expected results:
 OSTF passed
Actual result:
 Failed on step 2: Uploading package
Reproducibility:
 Always
Workaround:
 Replace 'httpclose' option in haproxy glare configuration to 'http-server-close' or remove it.
Impact:
 Failed 1 OSTF test
Description of the environment:
 Operation system: Ubuntu
 Versions of components: mitaka
 Network model: VXLAN
 Related projects installed: Murano, GlARe, OSTF
Additional information:
 - Part of OSTF log with sequence of requests: http://paste.openstack.org/show/509049/
 - This bug reproduced on SSL environments only (Without SSL enabled all works fine)
 - Strange unicode string in apache proxy log after request sequence to GLARE and code 400: http://paste.openstack.org/show/509051/
 - Last request wasn't appeared in HaProxy and Glare logs.
 - Directly, without proxy, all works fine (with https endpoint and --insecure flag)

upd1:
 - We found, that if we remove 'httpclose' flag in glare haproxy configuration (or replace it with http-server-close), SSL cases with GLARE will pass.
 - We found, that direct curl requests via proxy to GLARE works correctly.
 - We also found, that if we using CURL, in api-proxy log (proxy for OSTF of port 8888), there is HTTP/1.1 protocol used:
10.20.0.2 - - [10/Jun/2016:20:17:13 +0000] "CONNECT public.fuel.local:9494 HTTP/1.1" 200 - "-" "python-glanceclient"
But if we using OSTF tests, there is HTTP/1.0 in logs:
10.20.0.2 - - [10/Jun/2016:22:29:53 +0000] "CONNECT public.fuel.local:9494 HTTP/1.0" 200 - "-" "-"

Finally, it may be an api proxy (proxy on port 8888) problem.

It looks, that this is the same problem as in https://bugs.launchpad.net/mos/+bug/1527224

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/327621

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/327622

[Victor Ryzhenkin (vryzhenkin)]

Update:
We are found, that GLARE can't handle SSL termination.

Proof:

root@node-1:~# curl -k https://public.fuel.local:9494

{"versions": [{"status": "EXPERIMENTAL", "id": "v0.1", "links": [{"href": "http://public.fuel.local:9494/v0.1/", "rel": "self"}]}]}

In response over https should be a valid https endpoint, not http.

[Kirill Zaitsev (kzaitsev)]

relevant patches for other services:
https://review.openstack.org/#/c/64142/ heat
https://review.openstack.org/#/c/234134/ murano

[Timur Nurlygayanov (tnurlygayanov)]

Glare team, could you please prepare the fix for the issue based on fixes for Murano and Heat?

[Victor Ryzhenkin (vryzhenkin)]

Folks, we have new update.
After digging an environment, we found, that SSL termination - this is not exactly true problem.

Finally, we found, that following option in 081-glance-glare.cfg of HAPROXY 'option httpclose' leads to errors. After commenting it, we are able to see passed OSTF for Murano/Glare.

[Timur Nurlygayanov (tnurlygayanov)]

What does this option do?

[Dina Belova (dbelova)]

@Timur, we were investigating what influence will be in case of this option usage / non usage. In fact this option leads to session closing in case of timeouts, etc. - so it's important. The thing is that this issue cannot be easily reproduced, and exactly the same scenario run manually is working perfectly. Probably there is an issue in the either in openstack clients or their usage - this is something Victor is trying to find this out.

[OpenStack Infra (hudson-openstack)]

Change abandoned by tatyana-leontovich (<email address hidden>) on branch: master
Review: https://review.openstack.org/327621

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/329641

[OpenStack Infra (hudson-openstack)]

Change abandoned by Victor Ryzhenkin (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/327622
Reason: Incorrect root cause for a bug

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/329645

[Victor Ryzhenkin (vryzhenkin)]

During debugging, we are found, that not only GLARE affected here.
If we are change 'dummy_data' string with generated 'data' variable (put into image smth significant) in this line https://github.com/openstack/fuel-ostf/blob/master/fuel_health/glancemanager.py#L77 , we catch the same error.

Also, it looks like that only glance using PUT method in these tests. So, we are can't verify, that this is global problem.

Possible solution with HTTP_PROXY variable will not work (we are tried to verify it, but bvt was failed. Without HTTPS_PROXY it will not work).

I still think, that this happens due httpclose parameter in haproxy, but I've talked with Alex Shultz about it, and he voted against this change.

I can't find, what exactly component of the chain lead to this problem.

[Timur Nurlygayanov (tnurlygayanov)]

Release notes:
Murano OSTF tests fail for deployments with enabled TLS feature. It doesn't affect Murano functionality, it affects only OSTF tests because these tests use proxy service on OpenStack controller nodes to work with OpenStack API.

[Dmitry Klenov (dklenov)]

Sustaining team, can you please take a look on this issue?

Summary:
* problem is not in murano and murano-tests. Not in glance.
* possible problem in haproxy, ostf-framework or glance client.

Feel free to reassign to other teams if reason not in your area.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Victor Ryzhenkin (<email address hidden>) on branch: master
Review: https://review.openstack.org/329641
Reason: Incorrect root cause

[OpenStack Infra (hudson-openstack)]

Change abandoned by Victor Ryzhenkin (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/329645
Reason: Incorrect root cause

[Maksim Malchuk (mmalchuk)]

Victor, could you please update the status of the issue.

[Victor Ryzhenkin (vryzhenkin)]

Max, status if the issue wasn't changed.

[Oleksiy Molchanov (omolchanov)]

Request that fails:

glanceclient.common.http: DEBUG: curl -g -i -X PUT -H 'Accept-Encoding: gzip, deflate' -H 'Accept: ⁠⁠⁠/⁠⁠⁠' -H 'User-Agent: python-glanceclient' -H 'Connection: keep-alive' -H 'X-Auth-Token: {SHA1}8741e14bac7d38329763df01c0b20a2b23a4542c' -H 'Content-Type: application/octet-stream' -k --cert None --key None https://public.fuel.local:9494/v0.1/artifacts/murano/v1/e5f00dbc-3cd6-41da-90ea-ca114b30c1e6/archive

CommunicationError: Error finding address for https://public.fuel.local:9494/v0.1/artifacts/murano/v1/e5f00dbc-3cd6-41da-90ea-ca114b30c1e6/archive: [SSL: UNKNOWN_PROTOCOL] unknown protocol (_ssl.c:765)

I have tried to catch this request using tcpdump on controller, but it never comes to port 9494.

Glance team, can you check?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/378475

[OpenStack Infra (hudson-openstack)]

Change abandoned by Oleksiy Molchanov (<email address hidden>) on branch: master
Review: https://review.openstack.org/378475

[Oleksiy Molchanov (omolchanov)]

SSL tunings for haproxy did not help. The only solution seems is to set option server-http-close for this backend.

[Stanislaw Bogatkin (sbogatkin)]

Blocked by https://bugs.launchpad.net/fuel/+bug/1634550

[Stanislaw Bogatkin (sbogatkin)]

*Probably blocked.
I'll prepare a patch without murano envolved.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/408735

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/409149

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/409150

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/408735
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=0331b59169b773bca6f184119f02e08b89515c11
Submitter: Jenkins
Branch: master

commit 0331b59169b773bca6f184119f02e08b89515c11
Author: Stanislaw Bogatkin <email address hidden>
Date: Thu Dec 8 20:31:57 2016 +0300

    Use http-server-close for glance glare service

    Change-Id: Ic74d78ffb948f15e6a7297a05ab92b67b79da962
    Closes-Bug: #1590633

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/409149
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=9f85768baeb4c64bcd22fa8b522324030baa1892
Submitter: Jenkins
Branch: stable/newton

commit 9f85768baeb4c64bcd22fa8b522324030baa1892
Author: Stanislaw Bogatkin <email address hidden>
Date: Thu Dec 8 20:31:57 2016 +0300

    Use http-server-close for glance glare service

    Change-Id: Ic74d78ffb948f15e6a7297a05ab92b67b79da962
    Closes-Bug: #1590633
    (cherry-picked from 0331b59169b773bca6f184119f02e08b89515c11)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/409150
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=047e301a5c9b3239b3679207289bb105cd8e71c8
Submitter: Jenkins
Branch: stable/mitaka

commit 047e301a5c9b3239b3679207289bb105cd8e71c8
Author: Stanislaw Bogatkin <email address hidden>
Date: Thu Dec 8 20:31:57 2016 +0300

    Use http-server-close for glance glare service

    Change-Id: Ic74d78ffb948f15e6a7297a05ab92b67b79da962
    Closes-Bug: #1590633
    (cherry-picked from 0331b59169b773bca6f184119f02e08b89515c11)

[Ekaterina Shutova (eshutova)]

All OSTFs are passed. Verified on:
cat /etc/fuel_build_id:
 495
cat /etc/fuel_build_number:
 495
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 mitaka-9.0
rpm -qa | egrep 'fuel|astute|network-checker|nailgun|packetary|shotgun':
 fuel-nailgun-9.0.0-1.mos8924.noarch
 network-checker-9.0.0-1.mos77.x86_64
 fuel-ostf-9.0.0-1.mos947.noarch
 fuel-notify-9.0.0-1.mos8680.noarch
 fuel-agent-9.0.0-1.mos291.noarch
 python-packetary-9.0.0-1.mos160.noarch
 nailgun-mcagents-9.0.0-1.mos784.noarch
 fuel-setup-9.0.0-1.mos6359.noarch
 shotgun-9.0.0-1.mos90.noarch
 fuel-utils-9.0.0-1.mos8680.noarch
 fuelmenu-9.0.0-1.mos276.noarch
 fuel-provisioning-scripts-9.0.0-1.mos8924.noarch
 fuel-mirror-9.0.0-1.mos160.noarch
 fuel-openstack-metadata-9.0.0-1.mos8924.noarch
 rubygem-astute-9.0.0-1.mos784.noarch
 fuel-release-9.0.0-1.mos6359.noarch
 fuel-misc-9.0.0-1.mos8680.noarch
 fuel-ui-9.0.0-1.mos2854.noarch
 fuel-library9.0-9.0.0-1.mos8680.noarch
 fuel-bootstrap-cli-9.0.0-1.mos291.noarch
 fuel-migrate-9.0.0-1.mos8680.noarch
 python-fuelclient-9.0.0-1.mos364.noarch
 fuel-9.0.0-1.mos6359.noarch

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: master
Change author: Mariia Zlatkova <email address hidden>
Review: https://review.fuel-infra.org/30303

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/30303
Submitter: Olena Logvinova <email address hidden>
Branch: master

Commit: dc4cfe1141c0237b04d32ddea8c33b09ac0f854d
Author: Mariia Zlatkova <email address hidden>
Date: Thu Feb 2 13:50:37 2017

[RN-9.2] Fuel resolved and known issues

Change-Id: Idb919f92b981eee0f2cb48618dde243e4582ee5b
Related-Bug: #1590633
Related-Bug: #1625293
Related-Bug: #1561092
Related-Bug: #1619341
Related-Bug: #1563465
Related-Bug: #1628500
Related-Bug: #1593277
Related-Bug: #1628940
Related-Bug: #1658952

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: stable/9.2
Change author: Mariia Zlatkova <email address hidden>
Review: https://review.fuel-infra.org/30423

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/30423
Submitter: Mariia Zlatkova <email address hidden>
Branch: stable/9.2

Commit: c040581a57fac1dfeaed44952359f21963216d62
Author: Mariia Zlatkova <email address hidden>
Date: Thu Feb 2 14:03:50 2017

[RN-9.2] Fuel resolved and known issues

Change-Id: Idb919f92b981eee0f2cb48618dde243e4582ee5b
Related-Bug: #1590633
Related-Bug: #1625293
Related-Bug: #1561092
Related-Bug: #1619341
Related-Bug: #1563465
Related-Bug: #1628500
Related-Bug: #1593277
Related-Bug: #1628940
Related-Bug: #1658952
(cherry picked from commit dc4cfe1141c0237b04d32ddea8c33b09ac0f854d)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/fuel-library 11.0.0.0rc1 release candidate.

[Ilya Bumarskov (ibumarskov)]

Can't verify due to https://bugs.launchpad.net/fuel/+bug/1653073