setup-infrastructure in 19.0.0.0rc1 fails with 'File not found'

[Andreas Florath]

Running

openstack-ansible -v -v setup-infrastructure.yml

fails with the attached error.

Version: openstack-ansible 19.0.0.0rc1
Running on Ubuntu 18.04

Using three controller, three storage (ceph) and four compute nodes.

A re-run of setup-hosts and setup-infrastructure does not fix this.

===

fatal: [controller0_utility_container-01221bee]: FAILED! => {"attempts": 5, "changed": false, "cmd": ["/openstack/venvs/utility-19.0.0.0rc1/bin/pip2", "install", "-U", "--log", "/var/log/python_ve
nv_build.log", "--find-links", "http://192.168.72.29:8181/os-releases/19.0.0.0rc1/ubuntu-18.04-x86_64", "--trusted-host", "192.168.72.29", "pip", "setuptools", "wheel"], "msg": "\n:stderr: Excepti
on:\nTraceback (most recent call last):\n File \"/openstack/venvs/utility-19.0.0.0rc1/lib/python2.7/site-packages/pip/basecommand.py\", line 215, in main\n status = self.run(options, args)\n
File \"/openstack/venvs/utility-19.0.0.0rc1/lib/python2.7/site-packages/pip/commands/install.py\", line 353, in run\n wb.build(autobuilding=True)\n File \"/openstack/venvs/utility-19.0.0.0rc1/
lib/python2.7/site-packages/pip/wheel.py\", line 749, in build\n self.requirement_set.prepare_files(self.finder)\n File \"/openstack/venvs/utility-19.0.0.0rc1/lib/python2.7/site-packages/pip/r
eq/req_set.py\", line 380, in prepare_files\n ignore_dependencies=self.ignore_dependencies))\n File \"/openstack/venvs/utility-19.0.0.0rc1/lib/python2.7/site-packages/pip/req/req_set.py\", lin
e 487, in _prepare_file\n req_to_install, finder)\n File \"/openstack/venvs/utility-19.0.0.0rc1/lib/python2.7/site-packages/pip/req/req_set.py\", line 428, in _check_skip_installed\n req_to
_install, upgrade_allowed)\n File \"/openstack/venvs/utility-19.0.0.0rc1/lib/python2.7/site-packages/pip/index.py\", line 465, in find_requirement\n all_candidates = self.find_all_candidates(r
eq.name)\n File \"/openstack/venvs/utility-19.0.0.0rc1/lib/python2.7/site-packages/pip/index.py\", line 423, in find_all_candidates\n for page in self._get_pages(url_locations, project_name):\
n File \"/openstack/venvs/utility-19.0.0.0rc1/lib/python2.7/site-packages/pip/index.py\", line 568, in _get_pages\n page = self._get_page(location)\n File \"/openstack/venvs/utility-19.0.0.0r
c1/lib/python2.7/site-packages/pip/index.py\", line 683, in _get_page\n return HTMLPage.get_page(link, session=self.session)\n File \"/openstack/venvs/utility-19.0.0.0rc1/lib/python2.7/site-pa
ckages/pip/index.py\", line 795, in get_page\n resp.raise_for_status()\n File \"/openstack/venvs/utility-19.0.0.0rc1/share/python-wheels/requests-2.18.4-py2.py3-none-any.whl/requests/models.py
\", line 935, in raise_for_status\n raise HTTPError(http_error_msg, response=self)\nHTTPError: 404 Client Error: Not Found for url: http://192.168.72.29:8181/os-releases/19.0.0.0rc1/ubuntu-18.0
4-x86_64\n"}

[Andreas Florath (ansreas)]

Looks that I found the problem.

This

  http://192.168.72.29:8181/os-releases/19.0.0.0rc1/ubuntu-18.04-x86_64

is pointing towards the haproxy - which in turn passes the request on towards one repo-container. There are three repo-containers, one on each controller node, but only one (controller 0) contains the needed files.

Looks that they are not distributed correctly.

[Andreas Florath (ansreas)]

Yes: Copied over manually the files from the controller0 repo-container to the others, and re-run the setup-infrastructure: the problem vanished.

[Andreas Florath (ansreas)]

It looks that this is a more generic problem. It also happens multiple-times during setup-openstack.

TASK [python_venv_build : pip] **************************************************************************************************************************************************************************************************************
task path: /etc/ansible/roles/python_venv_build/tasks/python_venv_install.yml:114
<controller2_keystone_container-15ca0a5b> The "physical_host" variable of "controller2" has been found to have a corresponding host entry in inventory.
<controller2_keystone_container-15ca0a5b> The "physical_host" variable of "controller2" terminates at "192.168.72.20" using the host variable "ansible_host".
FAILED - RETRYING: python_venv_build : pip (5 retries left).
FAILED - RETRYING: python_venv_build : pip (4 retries left).
FAILED - RETRYING: python_venv_build : pip (3 retries left).
FAILED - RETRYING: python_venv_build : pip (2 retries left).
FAILED - RETRYING: python_venv_build : pip (1 retries left).
fatal: [controller2_keystone_container-15ca0a5b]: FAILED! => {"attempts": 5, "changed": false, "cmd": ["/openstack/venvs/keystone-19.0.0.0rc1/bin/pip2", "install", "-U", "--constraint", "/openstack/venvs/keystone-19.0.0.0rc1/constraints.txt", "--pre", "--log", "/var/log/python_venv_build.log", "--find-links", "http://192.168.72.29:8181/os-releases/19.0.0.0rc1/ubuntu-18.04-x86_64", "--trusted-host", "192.168.72.29", "keystone", "ldappool", "osprofiler", "PyMySQL", "pyngus", "python-memcached", "python-openstackclient", "systemd-python", "uWSGI"], "msg": "stdout: Looking in links: http://192.168.72.29:8181/os-releases/19.0.0.0rc1/ubuntu-18.04-x86_64\nCollecting systemd_python==234 (from -c /openstack/venvs/keystone-19.0.0.0rc1/constraints.txt (line 8))\n\n:stderr: DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.\n HTTP error 404 while getting http://192.168.72.29:8181/os-releases/19.0.0.0rc1/ubuntu-18.04-x86_64/systemd_python-234-cp27-cp27mu-linux_x86_64.whl (from http://192.168.72.29:8181/os-releases/19.0.0.0rc1/ubuntu-18.04-x86_64/)\n Could not install requirement systemd_python==234 from http://192.168.72.29:8181/os-releases/19.0.0.0rc1/ubuntu-18.04-x86_64/systemd_python-234-cp27-cp27mu-linux_x86_64.whl (from -c /openstack/venvs/keystone-19.0.0.0rc1/constraints.txt (line 8)) because of error 404 Client Error: Not Found for url: http://192.168.72.29:8181/os-releases/19.0.0.0rc1/ubuntu-18.04-x86_64/systemd_python-234-cp27-cp27mu-linux_x86_64.whl\nCould not install requirement systemd_python==234 from http://192.168.72.29:8181/os-releases/19.0.0.0rc1/ubuntu-18.04-x86_64/systemd_python-234-cp27-cp27mu-linux_x86_64.whl (from -c /openstack/venvs/keystone-19.0.0.0rc1/constraints.txt (line 8)) because of HTTP error 404 Client Error: Not Found for url: http://192.168.72.29:8181/os-releases/19.0.0.0rc1/ubuntu-18.04-x86_64/systemd_python-234-cp27-cp27mu-linux_x86_64.whl for URL http://192.168.72.29:8181/os-releases/19.0.0.0rc1/ubuntu-18.04-x86_64/systemd_python-234-cp27-cp...