reproducer queens undercloud upgrade job fails trying to setup the queens repo

[Alex Schultz]

While trying to reproduce the failure from http://logs.openstack.org/75/620975/1/check/tripleo-ci-centos-7-undercloud-upgrades/a6589b1/

I ran the reproducer script and the deployment fails on the fetching of the dlrn hash during the repo-setup-queens.sh execution

2018-11-30 16:17:01 | + rdo_dlrn=https://trunk.rdoproject.org/centos7-queens/93/b3/93b3d6841f99d9bdb20c6eed921600bb91897bd5_97a17ccc
2018-11-30 16:17:01 | ++ curl --silent http://mirror.regionone.rdo-cloud.rdoproject.org:8080/rdo/centos7-queens/d7d80c59146ce502ae82ad9ee93ab6409a3353f2_15fc9723/delorean.repo -S
2018-11-30 16:17:01 | ++ grep baseurl
2018-11-30 16:17:01 | ++ cut -d= -f2
2018-11-30 16:17:02 | + tripleo_dlrn=
2018-11-30 16:17:02 | + [[ -z https://trunk.rdoproject.org/centos7-queens/93/b3/93b3d6841f99d9bdb20c6eed921600bb91897bd5_97a17ccc ]]
2018-11-30 16:17:02 | + [[ -z '' ]]
2018-11-30 16:17:02 | + echo 'Failed to parse dlrn hash'
2018-11-30 16:17:02 | Failed to parse dlrn hash
2018-11-30 16:17:02 | + exit 1

When I tried to fetch this url from the host, I get:
curl: (35) SSL received a record that exceeded the maximum permissible length.

[Sorin Sbarnea (ssbarnea)]

I seen this SSL error when miss-matching HTTP(S) -- aka wrong port being used.

[Sorin Sbarnea (ssbarnea)]

run: curl -s -D- https://rdoproject.org/ | grep Strict

This will explain why even if yuo make a request to http://mirror.regionone.rdo-cloud.rdoproject.org:8080 you may get a failure.

SSL is enforced at root domain level and applies to all subdomain, mainly this means we cannot use HTTP on any rdoproject.org domains.

Caching is configured to 1Y and there is no practical way to reset it, by design.

I think that support for this was added relatively recently for curl.

[Sorin Sbarnea (ssbarnea)]

I also found the bug causing this, check http://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/tree/roles/nodepool-setup/templates/mirror_info.sh.j2

What we do with protocol and port does not really work.

[Sorin Sbarnea (ssbarnea)]

It seems that the default mirror to be used is mirror.regionone.rdo-cloud.rdoproject.org which means tha the entire configuration is invalid as at the moment I am writing this SSL is not configured for it.

[David Moreau Simard (dmsimard)]

Enabling SSL on the mirrors is not trivial right now.
We need to land the two following patches:
- https://review.openstack.org/#/c/529376
- https://review.openstack.org/#/c/528739

As well as a third patch that isn't written yet to enable SSL.

We can disable HSTS on rdoproject.org which propagates to require SSL on every subdomain under rdoproject.org.

www.rdoproject.org is managed by OSAS and I've sent a pull request to the role that manages the website apache configuration so that we may toggle HSTS off: https://gitlab.com/osas/ansible-role-ah-httpd/merge_requests/58/diffs?commit_id=eb50ce36bb8f015d3bd0de0c5035247bcf97af72

[David Moreau Simard (dmsimard)]

The review to toggle HSTS off for rdoproject.org is here: https://review.rdoproject.org/r/#/c/17629/

[Alan Pevec (apevec)]

> run: curl -s -D- https://rdoproject.org/ | grep Strict

this returns nothing now, I've manually added
<If "%{HTTP_HOST} = 'www.rdoproject.org'">
in /etc/httpd/conf.d/www.rdoproject.org_ssl.conf

How to do it properly will be discussed in OSAS gitlab merge request.

[Alan Pevec (apevec)]

osas/ansible-role-ah-httpd gitlab merge request was not accepted, final workaround implemented as described in comment 7