Placeholder for RDO Cloud issues

[Gabriele Cerami]

This is a tracker bug for the failing periodic job since the 25th rdocloud shortage.

rdocloud experienced problems with one of the nova nodes. As a result some servers were shut down.
In the night OVB jobs in the promotion failed to get nodes because there was not enough capacity.
After solving the capacity problems some other server needed for the jobs to work are experiencing problems.

This one was solved

https://bugs.launchpad.net/tripleo/+bug/1728033

but now the jobs are failing to contact the proxy.
The proxy server is up but doesn't respond.

Trying to access, but none of my keys are working.

Do not close this bug until we get the periodic jobs to work again.

[Gabriele Cerami (gcerami)]

Correction, the proxy server is up, running and accepting requests.

This is the error we've seen on the job:

14:06:52 Exception:
14:06:52 Traceback (most recent call last):
14:06:52 File "/usr/lib/python2.7/site-packages/virtualenv_support/pip-9.0.1-py2.py3-none-any.whl/pip/basecommand.py", line 215, in main
14:06:52 status = self.run(options, args)
14:06:52 File "/usr/lib/python2.7/site-packages/virtualenv_support/pip-9.0.1-py2.py3-none-any.whl/pip/commands/install.py", line 324, in run
14:06:52 requirement_set.prepare_files(finder)
14:06:52 File "/usr/lib/python2.7/site-packages/virtualenv_support/pip-9.0.1-py2.py3-none-any.whl/pip/req/req_set.py", line 380, in prepare_files
14:06:52 ignore_dependencies=self.ignore_dependencies))
14:06:52 File "/usr/lib/python2.7/site-packages/virtualenv_support/pip-9.0.1-py2.py3-none-any.whl/pip/req/req_set.py", line 554, in _prepare_file
14:06:52 require_hashes
14:06:52 File "/usr/lib/python2.7/site-packages/virtualenv_support/pip-9.0.1-py2.py3-none-any.whl/pip/req/req_install.py", line 278, in populate_link
14:06:52 self.link = finder.find_requirement(self, upgrade)
14:06:52 File "/usr/lib/python2.7/site-packages/virtualenv_support/pip-9.0.1-py2.py3-none-any.whl/pip/index.py", line 465, in find_requirement
14:06:52 all_candidates = self.find_all_candidates(req.name)
14:06:52 File "/usr/lib/python2.7/site-packages/virtualenv_support/pip-9.0.1-py2.py3-none-any.whl/pip/index.py", line 423, in find_all_candidates
14:06:52 for page in self._get_pages(url_locations, project_name):
14:06:52 File "/usr/lib/python2.7/site-packages/virtualenv_support/pip-9.0.1-py2.py3-none-any.whl/pip/index.py", line 568, in _get_pages
14:06:52 page = self._get_page(location)
14:06:52 File "/usr/lib/python2.7/site-packages/virtualenv_support/pip-9.0.1-py2.py3-none-any.whl/pip/index.py", line 683, in _get_page
14:06:52 return HTMLPage.get_page(link, session=self.session)
14:06:52 File "/usr/lib/python2.7/site-packages/virtualenv_support/pip-9.0.1-py2.py3-none-any.whl/pip/index.py", line 792, in get_page
14:06:52 "Cache-Control": "max-age=600",
14:06:52 File "/usr/lib/python2.7/site-packages/virtualenv_support/pip-9.0.1-py2.py3-none-any.whl/pip/_vendor/requests/sessions.py", line 488, in get
14:06:52 return self.request('GET', url, **kwargs)
14:06:52 File "/usr/lib/python2.7/site-packages/virtualenv_support/pip-9.0.1-py2.py3-none-any.whl/pip/download.py", line 386, in request
14:06:52 return super(PipSession, self).request(method, url, *args, **kwargs)
14:06:52 File "/usr/lib/python2.7/site-packages/virtualenv_support/pip-9.0.1-py2.py3-none-any.whl/pip/_vendor/requests/sessions.py", line 475, in request
14:06:52 resp = self.send(prep, **send_kwargs)
14:06:52 File "/usr/lib/python2.7/site-packages/virtualenv_support/pip-9.0.1-py2.py3-none-any.whl/pip/_vendor/requests/sessions.py", line 596, in send
14:06:52 r = adapter.send(request, **kwargs)
14:06:52 File "/usr/lib/python2.7/site-packages/virtualenv_support/pip-9.0.1-py2.py3-none-any.whl/pip/_vendor/requests/adapters.py", line 482, in send
14:06:52 raise RetryError(e, request=request)
14:06:5...

[Gabriele Cerami (gcerami)]

The error appeared again. The proxy server was using the mirror server as dns, and for some reason the mirror server was shut down again.
I was able to enter the proxy server and change the dns there too.

[Attila Darazs (adarazs)]

Pike started promoting and the master errors are not infra related. Closing this bug for now.