Artifact creation in GLARE using public endpoint that is under SSL failed with CommunicationError: [SSL: UNKNOWN_PROTOCOL] unknown protocol (_ssl.c:765)

Bug #1590633 reported by Victor Ryzhenkin on 2016-06-09
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
High
Stanislaw Bogatkin
Mitaka
High
Stanislaw Bogatkin
Newton
High
Stanislaw Bogatkin
Ocata
High
Stanislaw Bogatkin

Bug Description

Detailed bug description:
 During the test, which introduced in https://review.openstack.org/#/c/305951/ , the murano package should be uploaded into Glance Artifact repository. All requests to GLARE works fine, except uploading archive via PUT http method.
Steps to reproduce:
 1. Deploy build 470+ with SSL, Murano and GLARE backend.
 2. Run Murano OSTF platform test.
Expected results:
 OSTF passed
Actual result:
 Failed on step 2: Uploading package
Reproducibility:
 Always
Workaround:
 Replace 'httpclose' option in haproxy glare configuration to 'http-server-close' or remove it.
Impact:
 Failed 1 OSTF test
Description of the environment:
 Operation system: Ubuntu
 Versions of components: mitaka
 Network model: VXLAN
 Related projects installed: Murano, GlARe, OSTF
Additional information:
 - Part of OSTF log with sequence of requests: http://paste.openstack.org/show/509049/
 - This bug reproduced on SSL environments only (Without SSL enabled all works fine)
 - Strange unicode string in apache proxy log after request sequence to GLARE and code 400: http://paste.openstack.org/show/509051/
 - Last request wasn't appeared in HaProxy and Glare logs.
 - Directly, without proxy, all works fine (with https endpoint and --insecure flag)

upd1:
 - We found, that if we remove 'httpclose' flag in glare haproxy configuration (or replace it with http-server-close), SSL cases with GLARE will pass.
 - We found, that direct curl requests via proxy to GLARE works correctly.
 - We also found, that if we using CURL, in api-proxy log (proxy for OSTF of port 8888), there is HTTP/1.1 protocol used:
10.20.0.2 - - [10/Jun/2016:20:17:13 +0000] "CONNECT public.fuel.local:9494 HTTP/1.1" 200 - "-" "python-glanceclient"
But if we using OSTF tests, there is HTTP/1.0 in logs:
10.20.0.2 - - [10/Jun/2016:22:29:53 +0000] "CONNECT public.fuel.local:9494 HTTP/1.0" 200 - "-" "-"

Finally, it may be an api proxy (proxy on port 8888) problem.

It looks, that this is the same problem as in https://bugs.launchpad.net/mos/+bug/1527224

description: updated
description: updated
Changed in fuel:
assignee: nobody → Fuel Sustaining (fuel-sustaining-team)
Dmitry Pyzhov (dpyzhov) on 2016-06-09
Changed in fuel:
assignee: Fuel Sustaining (fuel-sustaining-team) → MOS Murano (mos-murano)
Dina Belova (dbelova) on 2016-06-09
Changed in fuel:
assignee: MOS Murano (mos-murano) → Stan Lagun (slagun)
assignee: Stan Lagun (slagun) → MOS Murano (mos-murano)
Changed in fuel:
assignee: MOS Murano (mos-murano) → Victor Ryzhenkin (vryzhenkin)

Update:
We are found, that GLARE can't handle SSL termination.

Proof:

root@node-1:~# curl -k https://public.fuel.local:9494

{"versions": [{"status": "EXPERIMENTAL", "id": "v0.1", "links": [{"href": "http://public.fuel.local:9494/v0.1/", "rel": "self"}]}]}

In response over https should be a valid https endpoint, not http.

summary: - [Murano][Glare][OSTF] Glare client can't upload a blob with SSL over
- HTTP proxy
+ [Murano][Glare][OSTF] Glare can not handle SSL termination proxies
Kirill Zaitsev (kzaitsev) wrote :
Changed in fuel:
assignee: Victor Ryzhenkin (vryzhenkin) → MOS Glance (mos-glance)

Glare team, could you please prepare the fix for the issue based on fixes for Murano and Heat?

Changed in fuel:
assignee: MOS Glance (mos-glance) → Kairat Kushaev (kkushaev)
Victor Ryzhenkin (vryzhenkin) wrote :

Folks, we have new update.
After digging an environment, we found, that SSL termination - this is not exactly true problem.

Finally, we found, that following option in 081-glance-glare.cfg of HAPROXY 'option httpclose' leads to errors. After commenting it, we are able to see passed OSTF for Murano/Glare.

Changed in fuel:
assignee: Kairat Kushaev (kkushaev) → MOS Puppet Team (mos-puppet)

What does this option do?

Dina Belova (dbelova) wrote :

@Timur, we were investigating what influence will be in case of this option usage / non usage. In fact this option leads to session closing in case of timeouts, etc. - so it's important. The thing is that this issue cannot be easily reproduced, and exactly the same scenario run manually is working perfectly. Probably there is an issue in the either in openstack clients or their usage - this is something Victor is trying to find this out.

Change abandoned by tatyana-leontovich (<email address hidden>) on branch: master
Review: https://review.openstack.org/327621

no longer affects: fuel
no longer affects: fuel/newton
Changed in fuel:
status: New → Confirmed
importance: Undecided → High
assignee: nobody → Victor Ryzhenkin (vryzhenkin)
milestone: none → 10.0
description: updated
description: updated
summary: - [Murano][Glare][OSTF] Glare can not handle SSL termination proxies
+ Artifact creation in GLARE using public endpoint that is under SSL
+ failed with CommunicationError: [SSL: UNKNOWN_PROTOCOL] unknown
+ protocol (_ssl.c:765)
tags: added: area-glance
description: updated

Fix proposed to branch: master
Review: https://review.openstack.org/329641

Changed in fuel:
status: Confirmed → In Progress

Change abandoned by Victor Ryzhenkin (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/327622
Reason: Incorrect root cause for a bug

Changed in fuel:
status: In Progress → Confirmed
assignee: Victor Ryzhenkin (vryzhenkin) → nobody
Victor Ryzhenkin (vryzhenkin) wrote :

During debugging, we are found, that not only GLARE affected here.
If we are change 'dummy_data' string with generated 'data' variable (put into image smth significant) in this line https://github.com/openstack/fuel-ostf/blob/master/fuel_health/glancemanager.py#L77 , we catch the same error.

Also, it looks like that only glance using PUT method in these tests. So, we are can't verify, that this is global problem.

Possible solution with HTTP_PROXY variable will not work (we are tried to verify it, but bvt was failed. Without HTTPS_PROXY it will not work).

I still think, that this happens due httpclose parameter in haproxy, but I've talked with Alex Shultz about it, and he voted against this change.

I can't find, what exactly component of the chain lead to this problem.

tags: added: release-notes

Release notes:
Murano OSTF tests fail for deployments with enabled TLS feature. It doesn't affect Murano functionality, it affects only OSTF tests because these tests use proxy service on OpenStack controller nodes to work with OpenStack API.

Dmitry Klenov (dklenov) wrote :

Sustaining team, can you please take a look on this issue?

Summary:
* problem is not in murano and murano-tests. Not in glance.
* possible problem in haproxy, ostf-framework or glance client.

Feel free to reassign to other teams if reason not in your area.

Changed in fuel:
assignee: nobody → Fuel Sustaining (fuel-sustaining-team)

Change abandoned by Victor Ryzhenkin (<email address hidden>) on branch: master
Review: https://review.openstack.org/329641
Reason: Incorrect root cause

Change abandoned by Victor Ryzhenkin (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/329645
Reason: Incorrect root cause

Changed in fuel:
assignee: Fuel Sustaining (fuel-sustaining-team) → Maksim Malchuk (mmalchuk)
Changed in fuel:
status: Confirmed → Triaged
Dmitry Pyzhov (dpyzhov) on 2016-07-20
tags: added: 9.1-proposed
Dmitry Pyzhov (dpyzhov) on 2016-07-21
tags: added: 9.1-opposed
removed: 9.1-proposed
Dmitry Pyzhov (dpyzhov) on 2016-08-04
tags: added: 9.1-proposed
removed: 9.1-opposed
Maksim Malchuk (mmalchuk) wrote :

Victor, could you please update the status of the issue.

Changed in fuel:
assignee: Maksim Malchuk (mmalchuk) → Victor Ryzhenkin (vryzhenkin)
Victor Ryzhenkin (vryzhenkin) wrote :

Max, status if the issue wasn't changed.

Changed in fuel:
assignee: Victor Ryzhenkin (vryzhenkin) → Maksim Malchuk (mmalchuk)
Changed in fuel:
assignee: Maksim Malchuk (mmalchuk) → Oleksiy Molchanov (omolchanov)
tags: added: release-notes-done
removed: release-notes
Dmitry Pyzhov (dpyzhov) on 2016-09-22
Changed in fuel:
status: Triaged → Confirmed
Oleksiy Molchanov (omolchanov) wrote :

Request that fails:

glanceclient.common.http: DEBUG: curl -g -i -X PUT -H 'Accept-Encoding: gzip, deflate' -H 'Accept: ⁠⁠⁠/⁠⁠⁠' -H 'User-Agent: python-glanceclient' -H 'Connection: keep-alive' -H 'X-Auth-Token: {SHA1}8741e14bac7d38329763df01c0b20a2b23a4542c' -H 'Content-Type: application/octet-stream' -k --cert None --key None https://public.fuel.local:9494/v0.1/artifacts/murano/v1/e5f00dbc-3cd6-41da-90ea-ca114b30c1e6/archive

CommunicationError: Error finding address for https://public.fuel.local:9494/v0.1/artifacts/murano/v1/e5f00dbc-3cd6-41da-90ea-ca114b30c1e6/archive: [SSL: UNKNOWN_PROTOCOL] unknown protocol (_ssl.c:765)

I have tried to catch this request using tcpdump on controller, but it never comes to port 9494.

Glance team, can you check?

Changed in fuel:
assignee: Oleksiy Molchanov (omolchanov) → MOS Glance (mos-glance)
Changed in fuel:
assignee: MOS Glance (mos-glance) → Oleksiy Molchanov (omolchanov)

Fix proposed to branch: master
Review: https://review.openstack.org/378475

Changed in fuel:
status: Confirmed → In Progress

Change abandoned by Oleksiy Molchanov (<email address hidden>) on branch: master
Review: https://review.openstack.org/378475

Oleksiy Molchanov (omolchanov) wrote :

SSL tunings for haproxy did not help. The only solution seems is to set option server-http-close for this backend.

Changed in fuel:
assignee: Oleksiy Molchanov (omolchanov) → Fuel Sustaining (fuel-sustaining-team)
status: In Progress → Confirmed
tags: added: customer-found
Changed in fuel:
milestone: 10.0 → 11.0
assignee: Fuel Sustaining (fuel-sustaining-team) → Stanislaw Bogatkin (sbogatkin)
Stanislaw Bogatkin (sbogatkin) wrote :

*Probably blocked.
I'll prepare a patch without murano envolved.

Fix proposed to branch: master
Review: https://review.openstack.org/408735

Changed in fuel:
status: Confirmed → In Progress

Reviewed: https://review.openstack.org/408735
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=0331b59169b773bca6f184119f02e08b89515c11
Submitter: Jenkins
Branch: master

commit 0331b59169b773bca6f184119f02e08b89515c11
Author: Stanislaw Bogatkin <email address hidden>
Date: Thu Dec 8 20:31:57 2016 +0300

    Use http-server-close for glance glare service

    Change-Id: Ic74d78ffb948f15e6a7297a05ab92b67b79da962
    Closes-Bug: #1590633

Changed in fuel:
status: In Progress → Fix Committed

Reviewed: https://review.openstack.org/409149
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=9f85768baeb4c64bcd22fa8b522324030baa1892
Submitter: Jenkins
Branch: stable/newton

commit 9f85768baeb4c64bcd22fa8b522324030baa1892
Author: Stanislaw Bogatkin <email address hidden>
Date: Thu Dec 8 20:31:57 2016 +0300

    Use http-server-close for glance glare service

    Change-Id: Ic74d78ffb948f15e6a7297a05ab92b67b79da962
    Closes-Bug: #1590633
    (cherry-picked from 0331b59169b773bca6f184119f02e08b89515c11)

Reviewed: https://review.openstack.org/409150
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=047e301a5c9b3239b3679207289bb105cd8e71c8
Submitter: Jenkins
Branch: stable/mitaka

commit 047e301a5c9b3239b3679207289bb105cd8e71c8
Author: Stanislaw Bogatkin <email address hidden>
Date: Thu Dec 8 20:31:57 2016 +0300

    Use http-server-close for glance glare service

    Change-Id: Ic74d78ffb948f15e6a7297a05ab92b67b79da962
    Closes-Bug: #1590633
    (cherry-picked from 0331b59169b773bca6f184119f02e08b89515c11)

tags: added: on-verification
Ekaterina Shutova (eshutova) wrote :

All OSTFs are passed. Verified on:
cat /etc/fuel_build_id:
 495
cat /etc/fuel_build_number:
 495
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 mitaka-9.0
rpm -qa | egrep 'fuel|astute|network-checker|nailgun|packetary|shotgun':
 fuel-nailgun-9.0.0-1.mos8924.noarch
 network-checker-9.0.0-1.mos77.x86_64
 fuel-ostf-9.0.0-1.mos947.noarch
 fuel-notify-9.0.0-1.mos8680.noarch
 fuel-agent-9.0.0-1.mos291.noarch
 python-packetary-9.0.0-1.mos160.noarch
 nailgun-mcagents-9.0.0-1.mos784.noarch
 fuel-setup-9.0.0-1.mos6359.noarch
 shotgun-9.0.0-1.mos90.noarch
 fuel-utils-9.0.0-1.mos8680.noarch
 fuelmenu-9.0.0-1.mos276.noarch
 fuel-provisioning-scripts-9.0.0-1.mos8924.noarch
 fuel-mirror-9.0.0-1.mos160.noarch
 fuel-openstack-metadata-9.0.0-1.mos8924.noarch
 rubygem-astute-9.0.0-1.mos784.noarch
 fuel-release-9.0.0-1.mos6359.noarch
 fuel-misc-9.0.0-1.mos8680.noarch
 fuel-ui-9.0.0-1.mos2854.noarch
 fuel-library9.0-9.0.0-1.mos8680.noarch
 fuel-bootstrap-cli-9.0.0-1.mos291.noarch
 fuel-migrate-9.0.0-1.mos8680.noarch
 python-fuelclient-9.0.0-1.mos364.noarch
 fuel-9.0.0-1.mos6359.noarch

tags: removed: on-verification

Related fix proposed to branch: master
Change author: Mariia Zlatkova <email address hidden>
Review: https://review.fuel-infra.org/30303

Reviewed: https://review.fuel-infra.org/30303
Submitter: Olena Logvinova <email address hidden>
Branch: master

Commit: dc4cfe1141c0237b04d32ddea8c33b09ac0f854d
Author: Mariia Zlatkova <email address hidden>
Date: Thu Feb 2 13:50:37 2017

[RN-9.2] Fuel resolved and known issues

Change-Id: Idb919f92b981eee0f2cb48618dde243e4582ee5b
Related-Bug: #1590633
Related-Bug: #1625293
Related-Bug: #1561092
Related-Bug: #1619341
Related-Bug: #1563465
Related-Bug: #1628500
Related-Bug: #1593277
Related-Bug: #1628940
Related-Bug: #1658952

Related fix proposed to branch: stable/9.2
Change author: Mariia Zlatkova <email address hidden>
Review: https://review.fuel-infra.org/30423

Reviewed: https://review.fuel-infra.org/30423
Submitter: Mariia Zlatkova <email address hidden>
Branch: stable/9.2

Commit: c040581a57fac1dfeaed44952359f21963216d62
Author: Mariia Zlatkova <email address hidden>
Date: Thu Feb 2 14:03:50 2017

[RN-9.2] Fuel resolved and known issues

Change-Id: Idb919f92b981eee0f2cb48618dde243e4582ee5b
Related-Bug: #1590633
Related-Bug: #1625293
Related-Bug: #1561092
Related-Bug: #1619341
Related-Bug: #1563465
Related-Bug: #1628500
Related-Bug: #1593277
Related-Bug: #1628940
Related-Bug: #1658952
(cherry picked from commit dc4cfe1141c0237b04d32ddea8c33b09ac0f854d)

This issue was fixed in the openstack/fuel-library 11.0.0.0rc1 release candidate.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers