[queens] [Periodic][check] OVB jobs failing in OC deploy when trying to ssh to nodes

[Alfredo Moralejo]

All Periodic/Third party check OVB jobs are failing in overcloud deployment phase with an error similar to:

http://logs.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-queens/5996bf7/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

2019-09-24 06:39:36 | 2019-09-24 06:39:31Z [overcloud.AllNodesDeploySteps.BlockStoragePostConfig]: CREAssh-keygen has been run successfully
2019-09-24 06:39:36 | Warning: Permanently added '192.168.24.8' (ECDSA) to the list of known hosts.
2019-09-24 06:41:36 | Authentication failed.
2019-09-24 06:41:36 | Command '['ssh', '-o', 'ConnectionAttempts=6', '-o', 'ConnectTimeout=30', '-o', 'StrictHostKeyChecking=no', '-o', 'PasswordAuthentication=no', '-o', 'UserKnownHostsFile=/dev/null', '-o', 'StrictHostKeyChecking=no', '-i', '/home/zuul/.ssh/id_rsa', '-l', 'heat-admin', u'192.168.24.8', "echo -e '\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCuO6Q/cDvh6uM0uCiiBz9zyr//O+OtoDnLVc8ZQIy4Z9kKjoGdowvHcMJ3mOkmQErWF3mLEJWjgODwpc8mXgep7iHJeR9Il3Yvfv01OcbwczUj1iF6bHkpNhI+p6GlXvAohMpQt1p8iEv7EoK/f2pBB9OISOeDuXGAUgeUnshONd1xdgvfeVqn4IdZQWV19owi0KofyTLCdS1NfVeMIVsJn5iua5NzBVsO/Sn5sL5pikO/Pljyi/yWugd2ISkr5cySll6v+z/ZuB/kkQaoSk/GADDAowTfxjQklfC8f0fS63cETJs4rPfjAHDrZe1VLpC7Zr/TKWRMszPHhcNGK+p547mkA1LYaKeHq5YlN66BFG4k4JcOqMWwImfNOjjT7e1/uz9uXQV070MZ3Zpdv/+C7j/BgHWpTHS66l+z3zNP+gxWrOa3sGwZ+XppVf6cTp7+bimCtldiuays1moNdCUYgzkJYub615ColrVHpbIpMqciKddj7GhoKfULPknoooyWSaT22NUxQaOPm7E4pyYlHr0kGtqkVK18luxhJlMp9+SHrxzlD8WjW6iw7MiGX7aMnOIjbrgXeZMs5Hkf1UQr5cNSQIvArPZ6eSx+eqMvNlr4EXq5FGIgTZaUBzMP+q40JtQPoNQAa4NEgM+64PwDT9jfrAwsK2eh7eirxdYotQ== TripleO split stack short term key\n' >> $HOME/.ssh/authorized_keys"]' returned non-zero exit status 255
2019-09-24 06:41:36 | END return value: 1
2019-09-24 06:41:36 | TE_COMPLETE state changed

Note all these jobs are also failing to collect logs from OVB nodes because of ssh authentication, so it may be related?:

http://logs.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-queens/5996bf7/job-output.txt.gz

[wes hayutin (weshayutin)]

Still happening
http://logs.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens/9f915d9/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

[wes hayutin (weshayutin)]

http://logs.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens/9f915d9/logs/

http://logs.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens/9f915d9/logs/baremetal_1-console.log

http://logs.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens/9f915d9/logs/bmc-console.log

Looks like a networking issue that we need to debug a bit.

FYI...bmc console shows the box is up.logs do not have the overcloud nodes indicating they are unreachable

PLAY RECAP *********************************************************************localhost : ok=41 changed=32 unreachable=0 failed=0 skipped=78 rescued=0 ignored=9 overcloud-controller-0 : ok=0 changed=0 unreachable=1 failed=0 skipped=0 rescued=0 ignored=0 overcloud-controller-1 : ok=0 changed=0 unreachable=1 failed=0 skipped=0 rescued=0 ignored=0 overcloud-controller-2 : ok=0 changed=0 unreachable=1 failed=0 skipped=0 rescued=0 ignored=0 overcloud-novacompute-0 : ok=0 changed=0 unreachable=1 failed=0 skipped=0 rescued=0 ignored=0 undercloud : ok=70 changed=56 unreachable=0 failed=0 skipped=57 rescued=0 ignored=7

[wes hayutin (weshayutin)]

In a master job... looks like overcloud nodes are not getting ip addresses

### IPv6 addresses
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 state UNKNOWN qlen 1000
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 state UP qlen 1000
    inet6 fe80::f816:3eff:fe3e:41da/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000
    inet6 fe80::f816:3eff:fefe:b33a/64 scope link
       valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000
    inet6 fe80::f816:3eff:fe4d:4f5e/64 scope link
       valid_lft forever preferred_lft forever
5: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000
    inet6 fe80::f816:3eff:fe18:9bf7/64 scope link
       valid_lft forever preferred_lft forever
6: eth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000
    inet6 fe80::f816:3eff:fe0d:9a59/64 scope link
       valid_lft forever preferred_lft forever
7: eth5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000
    inet6 fe80::f816:3eff:fe14:5c9c/64 scope link
       valid_lft forever preferred_lft forever
### IPv6 routing

http://logs.rdoproject.org/11/684411/5/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/54e01f5/logs/overcloud-controller-0/var/log/extra/network.txt.gz

[wes hayutin (weshayutin)]

so David Alvarez, Yatin, Alfredo and Arx are looking into this atm..

Looks like the kernel on the undercloud is suspect w/ the latest OVS in 7.7.
When 7.7 was tested previously here: https://review.opendev.org/#/c/618832/ queens passed, but the undercloud is a nodepool node and not using the latest kernel.

Now in the promotion jobs.. we have the latest 7.7 kernel across all the nodes.. and network traffic is messed up, ssh failing etc. David reports seeing mtu packet sizes above 1350 which could cause errors w/ ssh.

[Alfredo Moralejo (amoralej)]

I suspect from the mix ovb + centos7.7 kernel + ovs-2.9. That'd explain why it's only impacting queens (newer releases are using ovs-2.11) and match observations with relation to kernel versions in different jobs.

[Alfredo Moralejo (amoralej)]

According to the new tests in reproducer, we've check that setting useDNS=no in /etc/ssh/sshd_config in overcloud nodes should fix the issue.

It's still not clear what changed in 7.7 and why ssh works for other networks that 192.168.24.1 but that's what we've found so far.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/queens
Review: https://review.opendev.org/685342

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/685343

[Alfredo Moralejo (amoralej)]

Apparently, overcloud nodes have undercloud configured as default gateway but it seems it's not working properly and overcloud nodes can't reach anything out of the tenant. That's probably why dns resolution is taking very long to fail and ssh failure may be related to timeouts. Note that removing one of the three nameservers configured in /etc/resolv.conf also fixed ssh.

Probably, even when useDNS=no fixes ssh connectivity and it's in newer releases so it's probably fine to add, jobs will fail with other issues if they try to contact outside to install packages or anything. Let's see how ci jobs looks like on the proposed patches.

[wes hayutin (weshayutin)]

So.. not sure what we should do here..

Every time I try this outside of the openstack-nodepool tenant in rdo-cloud it gets past the ssh error

2019-09-27 23:24:58 | 2019-09-27 23:24:56Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step5]: CREATE_IN_PROGRESS state changed
2019-09-27 23:24:58 | 2019-09-27 23:24:56Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step5]: CREATE_COMPLETE state changed
2019-09-27 23:25:07 | 2019-09-27 23:24:56Z [overcloud.AllNodesDeploySteps.Blockssh-keygen has been run successfully
2019-09-27 23:25:07 | Warning: Permanently added '10.9.121.10' (ECDSA) to the list of known hosts.
2019-09-27 23:25:18 | Warning: Permanently added '10.9.121.6' (ECDSA) to the list of known hosts.
2019-09-27 23:25:24 | Warning: Permanently added '10.9.121.16' (ECDSA) to the list of known hosts.
2019-09-27 23:25:35 | Warning: Permanently added '10.9.121.7' (ECDSA) to the list of known hosts.

https://code.engineering.redhat.com/gerrit/#/c/182350/

Should we change the tripleo to work in rdo-cloud openstack-nodepool?

[wes hayutin (weshayutin)]

Queens works out side rdo-cloud, I'm fine w/ turning off dns lookups for dns patches.

Just for the record this is an issue w/ the network environment in openstack-nodepool.
Ping me for access to the environment.

2019-09-28 00:20:17 | +----------------+------------------------------------------------------+
2019-09-28 00:20:17 | | Name | EntryPoint |
2019-09-28 00:20:17 | +----------------+------------------------------------------------------+
2019-09-28 00:20:17 | | keystone_tests | keystone_tempest_plugin.plugin:KeystoneTempestPlugin |
2019-09-28 00:20:17 | | neutron_tests | neutron_tempest_plugin.plugin:NeutronTempestPlugin |
2019-09-28 00:20:17 | +----------------+------------------------------------------------------+
2019-09-28 00:20:17 | + /usr/bin/tempest cleanup --init-saved-state
2019-09-28 00:20:38 | Initializing saved state.
2019-09-28 00:20:38 | + /usr/bin/tempest run --whitelist_file=/home/zuul/whitelist_file.conf --blacklist_file=/home/zuul/skip_file --concurrency 4
2019-09-28 00:23:18 | /usr/lib/python2.7/site-packages/paramiko/rsakey.py:119: DeprecationWarning: signer and verifier have been deprecated. Please use sign and verify instead.
2019-09-28 00:23:18 | algorithm=hashes.SHA1(),
2019-09-28 00:25:21 | {0} tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops [263.856512s] ... ok

2019-09-28 00:12:20 | PLAY RECAP *********************************************************************
2019-09-28 00:12:20 | overcloud-controller-0 : ok=210 changed=110 unreachable=0 failed=0
2019-09-28 00:12:20 | overcloud-controller-1 : ok=207 changed=109 unreachable=0 failed=0
2019-09-28 00:12:20 | overcloud-controller-2 : ok=208 changed=109 unreachable=0 failed=0
2019-09-28 00:12:20 | overcloud-novacompute-0 : ok=180 changed=87 unreachable=0 failed=0
2019-09-28 00:12:20 | undercloud : ok=2 changed=0 unreachable=0 failed=0
2019-09-28 00:12:20 |
2019-09-28 00:12:20 | Saturday 28 September 2019 00:12:06 +0000 (0:00:00.395) 0:44:19.305 ****
2019-09-28 00:12:20 | ===============================================================================
2019-09-28 00:12:20 |
2019-09-28 00:12:20 | Ansible passed.
2019-09-28 00:12:20 | Overcloud configuration completed.
2019-09-28 00:12:20 | Started Mistral Workflow tripleo.deployment.v1.get_horizon_url. Execution ID: 6fec148f-92a0-4dc3-9262-a2e9410680db
2019-09-28 00:12:20 | Overcloud Endpoint: https://10.9.121.96:13000/
2019-09-28 00:12:20 | Overcloud Horizon Dashboard URL: https://10.9.121.96:443/dashboard
2019-09-28 00:12:20 | Overcloud rc file: /home/zuul/overcloudrc
2019-09-28 00:12:20 | Overcloud Deployed
2019-09-28 00:12:21 | + status_code=0
2019-09-28 00:12:21 | + openstack stack list

[heat-admin@overcloud-controller-1 ~]$
[heat-admin@overcloud-controller-1 ~]$ nslookup www.redhat.com
Server: 10.5.30.160
Address: 10.5.30.160#53

Non-authoritative answer:
www.redhat.com canonical name = ds-www.redhat.com.edgekey.net.
ds-www.redhat.com.edgekey.net canonical name = ds-www.redhat.com.edgekey.net.globalredir.akadns.net.
ds-www.r...

[wes hayutin (weshayutin)]

https://code.engineering.redhat.com/gerrit/#/c/182350/ is what kicked the job

[wes hayutin (weshayutin)]

bah... even though the job is marked periodic, it picked up current-tripleo. Have to retest w/ the appropriate delorean hash.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/685342
Committed: https://git.openstack.org/cgit/openstack/tripleo-puppet-elements/commit/?id=ada8f50d604464dba64be5eaef60f59a270a1327
Submitter: Zuul
Branch: stable/queens

commit ada8f50d604464dba64be5eaef60f59a270a1327
Author: Emilien Macchi <email address hidden>
Date: Fri Jun 1 09:14:48 2018 -0700

    Add element for openssh and override UseDNS

    We need to disable UseDNS in default sshd_config, which is something we
    already do when deploying SSHD service in puppet/services/sshd.yaml but
    the parameter is still enabled before Puppet has run, which causes
    problems when Ansible has to execute remote tasks via ssh, it timeouts.

    This patch adds an element that will allow us to override the default
    openssh configuration and it'll disable UseDNS.

    Change-Id: I2df6c45936ee88e0eb048998ae23c9334f295fdf
    Related-Bug: #1774557
    Related-Bug: #1845166
    (cherry picked from commit 939b5861e1ff2f5311a16ccf4c63922493159f56)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/685343
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=08dd04a3763e3658b6454aee2103b7837f8b4970
Submitter: Zuul
Branch: stable/queens

commit 08dd04a3763e3658b6454aee2103b7837f8b4970
Author: Emilien Macchi <email address hidden>
Date: Fri Jun 1 09:17:50 2018 -0700

    overcloud: include openssh element

    This new element is required on all overcloud images.

    Depends-On: https://review.opendev.org/#/c/685342/
    Change-Id: I93a60febf5b75dad2f92d103f1dcd141dc418f92
    Related-Bug: #1774557
    Closes-Bug: #1845166
    (cherry picked from commit 027868b4e7bd5668e17ac06c1936ae6d1408e467)

[Alfredo Moralejo (amoralej)]

Here are our findings after setting a new reproducer:

- The actual problem is that masquerading for overcloud nodes in undercloud is not working properly.

- "Something" is seting policy to DROP for FORWARD policy in iptables (as seen with iptables -S):

-P FORWARD DROP

That is preventing any traffic forwarding in undercloud.

- The "something" setting iptables wrong is docker:

https://github.com/moby/moby/pull/28257

So the behavior is:

if net.ipv4.ip_forward=0 when docker is about to be started the first time, it will add that DROP policy in FORWARD chain.

According to one of the reproducer logs:

Oct 1 09:24:09 tripleo-ovb-centos-7-rdo-cloud-tripleo-0002179662 systemd: Starting Docker Application Container Engine...
2019-10-01 09:38:02 | 2019-10-01 09:38:02,726 INFO: ^[[mNotice: /Stage[main] Main/Sysctl::Value[net.ipv4.ip_forward]/Sysctl[net.ipv4.ip_forward]/ensure: created

Still investigating what change introduced this early docker start.

[yatin (yatinkarel)]

<< Still investigating what change introduced this early docker start.

So this is introduced by docker login enabled by default in all jobs with https://review.rdoproject.org/r/#/c/22304/ which runs container-registry role, https://github.com/openstack/ansible-role-container-registry/blob/master/tasks/install-engine.yml#L14-L17, and hence by that time ip_forward rule is not set, and starting docker introduces FORWARD DROP rule

[Alfredo Moralejo (amoralej)]

I'm not sure what's the proper fix is, in puppet-tripleo, or something or ansible-role-container-registry.

[Alex Schultz (alex-schultz)]

We already addressed this in puppet-tripleo for the normal deployment. This is a side effect of the CI items. We shouldn't be logging in before the undercloud is installed for queens.

[Alex Schultz (alex-schultz)]

https://github.com/openstack/ansible-role-container-registry/blob/master/tasks/docker.yml#L20-L28

[Alex Schultz (alex-schultz)]

There is a docker setting to disable this iptables management that you can add to /etc/sysconfig/docker as well. But this is an issue with extra stuff that needs to not be done via the CI framework. We support authentication in the deployment so that should be used.

[Alex Schultz (alex-schultz)]

the option is '--iptables=false' which will prevent docker from adding the default rules if you don't enable ip_forward before you start docker

[wes hayutin (weshayutin)]

OK.. I have a passing deployment in my rdo-cloud tenant, but failing tempest on baseurl=https://trunk.rdoproject.org/centos7-queens/e2/20/e2208117a31eea79c50330333501b6581f9faafe_15a9b671

Issue was w/ OOM's in the infra.

(undercloud) [zuul@undercloud ~]$ openstack server list
+--------------------------------------+-------------------------+--------+------------------------+----------------+-----------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+-------------------------+--------+------------------------+----------------+-----------+
| 2d3e3c6a-1025-452d-832c-6e209ef554ea | overcloud-controller-2 | ACTIVE | ctlplane=192.168.24.8 | overcloud-full | baremetal |
| c303f8d6-3c1b-4154-80c8-8fad570ff967 | overcloud-controller-0 | ACTIVE | ctlplane=192.168.24.12 | overcloud-full | baremetal |
| eb5fd595-7446-4012-be9c-3cd160f55057 | overcloud-controller-1 | ACTIVE | ctlplane=192.168.24.20 | overcloud-full | baremetal |
| fed9b36a-725e-4111-86a8-e545c57ca7ba | overcloud-novacompute-0 | ACTIVE | ctlplane=192.168.24.14 | overcloud-full | baremetal |
+--------------------------------------+-------------------------+--------+------------------------+----------------+-----------+
(undercloud) [zuul@undercloud ~]$ ssh heat-admin@192.168.24.8
The authenticity of host '192.168.24.8 (192.168.24.8)' can't be established.
ECDSA key fingerprint is SHA256:mekrS/mVlWYAcvAgSy4lTP6szxV+XewNGMYmhoE7pY8.
ECDSA key fingerprint is MD5:d8:9e:02:4f:e6:ee:76:cc:9b:1e:bd:cf:e6:db:04:4f.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.24.8' (ECDSA) to the list of known hosts.
Last login: Tue Oct 1 19:44:44 2019 from 192.168.24.1
[heat-admin@overcloud-controller-2 ~]$ ls
[heat-admin@overcloud-controller-2 ~]$ nslookup www.ibm.com
Server: 8.8.8.8
Address: 8.8.8.8#53

Non-authoritative answer:
www.ibm.com canonical name = www.ibm.com.cs186.net.
www.ibm.com.cs186.net canonical name = outer-ccdn-dual.ibmcom.edgekey.net.
outer-ccdn-dual.ibmcom.edgekey.net canonical name = outer-ccdn-dual.ibmcom.edgekey.net.globalredir.akadns.net.
outer-ccdn-dual.ibmcom.edgekey.net.globalredir.akadns.net canonical name = e2874.dscx.akamaiedge.net.
Name: e2874.dscx.akamaiedge.net
Address: 96.16.19.195
Name: e2874.dscx.akamaiedge.net
Address: 2600:1404:1800:683::b3a
Name: e2874.dscx.akamaiedge.net
Address: 2600:1404:1800:68d::b3a

=================================================================================
I have another local install on my owner hardware.. same hash..
Overcloud deployed

2019-10-01 19:48:45 | PLAY [External deployment Post Deploy tasks] ***********************************
2019-10-01 19:48:45 | META: ran handlers
2019-10-01 19:48:45 | META: ran handlers
2019-10-01 19:48:45 | META: ran handlers
2019-10-01 19:48:45 |
2019-10-01 19:48:45 | PLAY RECAP *********************************************************************
2019-10-01 19:48:45 | overcloud-controller-0 : ok=216 changed=112 unreachable=0 failed=0
2019-10-01 19:48:45 | overcloud-novacompute-0 : ok=180 changed=88 unreachabl...

[Sorin Sbarnea (ssbarnea)]

Is https://review.opendev.org/#/c/686173/ what we need?

[Marios Andreou (marios-b)]

latest queens jobs are now hitting a NTP related issue like [1]

2019-10-23 08:45:45 | fatal: [overcloud-controller-0]: FAILED! => {"changed": true, "cmd": ["ntpdate", "-u", "0.pool.ntp.org", "1.pool.ntp.org", "2.pool.ntp.org", "3.pool.ntp.org"], "delta": "0:03:44.223481", "end": "2019-10-23 08:45:44.059100", "msg": "non-zero return code", "rc": 1, "start": "2019-10-23 08:41:59.835619", "stderr": "Error resolving 0.pool.ntp.org: Name or service not known (-2)\n23 Oct 08:42:55 ntpdate[14373]: Can't find host 0.pool.ntp.org: Name or service not known (-2)\nError resolving 1.pool.ntp.org: Name or service not known (-2)\n23 Oct 08:43:51 ntpdate[14373]: ...

I filed that in https://bugs.launchpad.net/tripleo/+bug/1849099 but ykarel pointed me here and I've marked it as duplicate suspecting it is to do with comment #17 and docker/iptables.

[1] http://logs.rdoproject.org/openstack-periodic-wednesday-weekend/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-queens/16ad4e3/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz "

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/692223

[Ronelle Landy (rlandy)]

periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-queens openstack/tripleo-ci master openstack-periodic-wednesday-weekend master 15138 2019-11-06T07:05:17 SUCCESS

We had a successful run w/o testproject yesterday - closing this out

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sorin Sbarnea (<email address hidden>) on branch: master
Review: https://review.opendev.org/692196

[OpenStack Infra (hudson-openstack)]

Change abandoned by wes hayutin (<email address hidden>) on branch: master
Review: https://review.opendev.org/692223
Reason: too_old

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-common queens-eol release.