bootstrap-ansible.sh not installing ansible==2.4.3.0

[Matt Thompson]

I currently have an AIO deployed w/ master -- initial deployment went fine. Periodically, I pull down changes in /opt/openstack-ansible and re-run scripts/bootstrap-ansible.sh to grab latest roles. scripts/bootstrap-ansible.sh runs fine but then running any playbooks w/ openstack-ansible fails because the system has ansible 2.3.2.0 installed and not 2.4.3.0.

Here's some output from scripts/bootstrap-ansible.sh:

+ PIP_COMMAND=/opt/ansible-runtime/bin/pip
+ PIP_OPTS+=' --constraint global-requirement-pins.txt'
+ PIP_OPTS+=' --constraint https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt?id=83dbef854e3b2b699f9382a993997a94bba1f5cb'
+ /opt/ansible-runtime/bin/pip install --constraint global-requirement-pins.txt --constraint 'https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt?id=83dbef854e3b2b699f9382a993997a94bba1f5cb' -r requirements.txt ansible==2.4.3.0
Ignoring jsbeautifier: markers 'python_version == "3.4"' don't match your environment
Ignoring hyperframe: markers 'python_version == "3.4"' don't match your environment
Ignoring cssutils: markers 'python_version == "3.4"' don't match your environment
Ignoring kaitaistruct: markers 'python_version == "3.4"' don't match your environment
Ignoring unicodecsv: markers 'python_version == "2.7"' don't match your environment
Ignoring pathlib: markers 'python_version == "2.7"' don't match your environment
Ignoring tornado: markers 'python_version == "3.4"' don't match your environment
Ignoring backports.ssl-match-hostname: markers 'python_version == "2.7"' don't match your environment
Ignoring ruamel.yaml: markers 'python_version == "3.4"' don't match your environment
Ignoring html2text: markers 'python_version == "3.4"' don't match your environment
Ignoring funcsigs: markers 'python_version == "2.7"' don't match your environment
Ignoring dnspython3: markers 'python_version == "3.4"' don't match your environment
Ignoring hpack: markers 'python_version == "3.4"' don't match your environment
Ignoring brotlipy: markers 'python_version == "3.4"' don't match your environment
Ignoring mypy: markers 'python_version == "3.4"' don't match your environment
Ignoring urwid: markers 'python_version == "3.4"' don't match your environment
Ignoring singledispatch: markers 'python_version == "2.7"' don't match your environment
Ignoring thriftpy: markers 'python_version == "2.7"' don't match your environment
Ignoring argh: markers 'python_version == "3.4"' don't match your environment
Ignoring weakrefmethod: markers 'python_version == "2.7"' don't match your environment
Ignoring blinker: markers 'python_version == "3.4"' don't match your environment
Ignoring typed-ast: markers 'python_version == "3.4"' don't match your environment
Ignoring h2: markers 'python_version == "3.4"' don't match your environment
Ignoring mitmproxy: markers 'python_version == "3.4"' don't match your environment
Ignoring EditorConfig: markers 'python_version == "3.4"' don't match your environment
Ignoring ndg-httpsclient: markers 'python_version == "2.7"' don't match your environment
Ignoring functools32: markers 'python_version == "2.7"' don't match your environment
Ignoring watchdog: markers 'python_version == "3.4"' don't match your environment
Ignoring pathtools: markers 'python_version == "3.4"' don't match your environment
Ignoring qpid-python: markers 'python_version == "2.7"' don't match your environment
Ignoring scandir: markers 'python_version == "2.7"' don't match your environment
Ignoring futures: markers 'python_version == "2.7"' don't match your environment
Ignoring happybase: markers 'python_version == "2.7"' don't match your environment
Ignoring sortedcontainers: markers 'python_version == "3.4"' don't match your environment
Ignoring ndg-httpsclient: markers 'python_version < "3.0"' don't match your environment
Collecting pyopenssl==17.5.0 (from -c http://172.29.236.100:8181/os-releases/17.0.0/ubuntu-16.04-x86_64/requirements_absolute_requirements.txt (line 14))
  Downloading http://172.29.236.100:8181/packages/79/db/7c0cfe4aa8341a5fab4638952520d8db6ab85ff84505e12c00ea311c3516/pyOpenSSL-17.5.0-py2.py3-none-any.whl (53kB)
Collecting netaddr==0.7.19 (from -c http://172.29.236.100:8181/os-releases/17.0.0/ubuntu-16.04-x86_64/requirements_absolute_requirements.txt (line 16))
  Downloading http://172.29.236.100:8181/packages/ba/97/ce14451a9fd7bdb5a397abf99b24a1a6bb7a1a440b019bebd2e9a0dbec74/netaddr-0.7.19-py2.py3-none-any.whl (1.6MB)
Collecting prettytable==0.7.2 (from -c http://172.29.236.100:8181/os-releases/17.0.0/ubuntu-16.04-x86_64/requirements_absolute_requirements.txt (line 17))
Collecting pyyaml==3.12 (from -c http://172.29.236.100:8181/os-releases/17.0.0/ubuntu-16.04-x86_64/requirements_absolute_requirements.txt (line 20))
Collecting pyasn1==0.4.2 (from -c http://172.29.236.100:8181/os-releases/17.0.0/ubuntu-16.04-x86_64/requirements_absolute_requirements.txt (line 54))
  Downloading http://172.29.236.100:8181/packages/ba/fe/02e3e2ee243966b143657fb8bd6bc97595841163b6d8c26820944acaec4d/pyasn1-0.4.2-py2.py3-none-any.whl (71kB)
Collecting python_memcached==1.59 (from -c http://172.29.236.100:8181/os-releases/17.0.0/ubuntu-16.04-x86_64/requirements_absolute_requirements.txt (line 58))
  Downloading http://172.29.236.100:8181/packages/f5/90/19d3908048f70c120ec66a39e61b92c253e834e6e895cd104ce5e46cbe53/python_memcached-1.59-py2.py3-none-any.whl
Collecting six==1.11.0 (from -c http://172.29.236.100:8181/os-releases/17.0.0/ubuntu-16.04-x86_64/requirements_absolute_requirements.txt (line 60))
  Downloading http://172.29.236.100:8181/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
Collecting virtualenv==15.1.0 (from -c http://172.29.236.100:8181/os-releases/17.0.0/ubuntu-16.04-x86_64/requirements_absolute_requirements.txt (line 89))
  Downloading http://172.29.236.100:8181/packages/6f/86/3dc328ee7b1a6419ebfac7896d882fba83c48e3561d22ddddf38294d3e83/virtualenv-15.1.0-py2.py3-none-any.whl (1.8MB)
Collecting ansible==2.3.2.0 (from -c http://172.29.236.100:8181/os-releases/17.0.0/ubuntu-16.04-x86_64/requirements_absolute_requirements.txt (line 91))
Collecting paramiko==2.4.0 (from -c http://172.29.236.100:8181/os-releases/17.0.0/ubuntu-16.04-x86_64/requirements_absolute_requirements.txt (line 171))
  Downloading http://172.29.236.100:8181/packages/be/9f/2b899b028aec1f3973253c0cf8dda6fbff65f4930f7ebedc43033e9f1b18/paramiko-2.4.0-py2.py3-none-any.whl (192kB)
Collecting jinja2==2.10 (from -c http://172.29.236.100:8181/os-releases/17.0.0/ubuntu-16.04-x86_64/requirements_absolute_requirements.txt (line 172))
  Downloading http://172.29.236.100:8181/packages/7f/ff/ae64bacdfc95f27a016a7bed8e8686763ba4d277a78ca76f32659220a731/Jinja2-2.10-py2.py3-none-any.whl (126kB)
Collecting bcrypt==3.1.4 (from -c http://172.29.236.100:8181/os-releases/17.0.0/ubuntu-16.04-x86_64/requirements_absolute_requirements.txt (line 263))
  Downloading http://172.29.236.100:8181/packages/72/8d/57da727cbb4ddf0295b0665a93bd8f46fbbd2a48b76cbb0896ca7d27301f/bcrypt-3.1.4-cp35-cp35m-manylinux1_x86_64.whl (54kB)
Collecting pynacl==1.2.1 (from -c http://172.29.236.100:8181/os-releases/17.0.0/ubuntu-16.04-x86_64/requirements_absolute_requirements.txt (line 294))
  Downloading http://172.29.236.100:8181/packages/c9/9c/c8d4ae0f590faded4af55381131ad4db8729214eba6537c520622ba3548a/PyNaCl-1.2.1-cp35-cp35m-manylinux1_x86_64.whl (692kB)
Collecting markupsafe==1.0 (from -c http://172.29.236.100:8181/os-releases/17.0.0/ubuntu-16.04-x86_64/requirements_absolute_requirements.txt (line 295))
Collecting cryptography==2.1.4 (from -c http://172.29.236.100:8181/os-releases/17.0.0/ubuntu-16.04-x86_64/requirements_absolute_requirements.txt (line 326))
  Downloading http://172.29.236.100:8181/packages/15/4e/c0a0641dc0b4bba7c1dfcf66e30ef34effe7f7dc20a37459e9e052afc4cf/cryptography-2.1.4-cp35-cp35m-manylinux1_x86_64.whl (2.2MB)
Requirement already up-to-date: setuptools==38.5.1 in /opt/ansible-runtime/lib/python3.5/site-packages (from -c http://172.29.236.100:8181/os-releases/17.0.0/ubuntu-16.04-x86_64/requirements_absolute_requirements.txt (line 2))
Collecting pycrypto==2.6.1 (from -c http://172.29.236.100:8181/os-releases/17.0.0/ubuntu-16.04-x86_64/requirements_absolute_requirements.txt (line 29))
Collecting cffi==1.11.4 (from -c http://172.29.236.100:8181/os-releases/17.0.0/ubuntu-16.04-x86_64/requirements_absolute_requirements.txt (line 80))
  Downloading http://172.29.236.100:8181/packages/cd/e7/2e146fd7fe666727dd67d8a882c8692a40524541fb0f785300b011a12ed0/cffi-1.11.4-cp35-cp35m-manylinux1_x86_64.whl (419kB)
Collecting asn1crypto==0.24.0 (from -c http://172.29.236.100:8181/os-releases/17.0.0/ubuntu-16.04-x86_64/requirements_absolute_requirements.txt (line 277))
  Downloading http://172.29.236.100:8181/packages/ea/cd/35485615f45f30a510576f1a56d1e0a7ad7bd8ab5ed7cdc600ef7cd06222/asn1crypto-0.24.0-py2.py3-none-any.whl (101kB)
Collecting idna==2.6 (from -c http://172.29.236.100:8181/os-releases/17.0.0/ubuntu-16.04-x86_64/requirements_absolute_requirements.txt (line 41))
  Downloading http://172.29.236.100:8181/packages/27/cc/6dd9a3869f15c2edfab863b992838277279ce92663d334df9ecf5106f5c6/idna-2.6-py2.py3-none-any.whl (56kB)
Collecting pycparser==2.18 (from -c http://172.29.236.100:8181/os-releases/17.0.0/ubuntu-16.04-x86_64/requirements_absolute_requirements.txt (line 117))
  Downloading http://172.29.236.100:8181/os-releases/17.0.0/ubuntu-16.04-x86_64/pycparser-2.18-py2.py3-none-any.whl (206kB)
Installing collected packages: six, asn1crypto, idna, pycparser, cffi, cryptography, pyopenssl, netaddr, prettytable, pyyaml, pycrypto, pyasn1, python-memcached, virtualenv, markupsafe, jinja2, pynacl, bcrypt, paramiko, ansible
Successfully installed ansible-2.3.2.0 asn1crypto-0.24.0 bcrypt-3.1.4 cffi-1.11.4 cryptography-2.1.4 idna-2.6 jinja2-2.10 markupsafe-1.0 netaddr-0.7.19 paramiko-2.4.0 prettytable-0.7.2 pyasn1-0.4.2 pycparser-2.18 pycrypto-2.6.1 pynacl-1.2.1 pyopenssl-17.5.0 python-memcached-1.59 pyyaml-3.12 six-1.11.0 virtualenv-15.1.0

As you can see, we request ansible==2.4.3.0 but ansible-2.3.2.0 is getting installed. If I look at requirements_absolute_requirements.txt, I can see ansible==2.3.2.0 listed:

root@aio1:/opt/openstack-ansible# curl -s http://172.29.236.100:8181/os-releases/17.0.0/ubuntu-16.04-x86_64/requirements_absolute_requirements.txt | grep ansible
ansible==2.3.2.0
root@aio1:/opt/openstack-ansible#

A hacky workaround has been to move /root/.pip aside, run scripts/bootstrap-ansible.sh, and then continue on.

Please let me know if I can provide any further info.

Thanks!

--Matt

[Matt Thompson (mattt416)]

Additionally, here's my /root/.pip/pip.conf:

root@aio1:/opt/openstack-ansible# cat /root/.pip/pip.conf
# Ansible managed

[global]
timeout = 120
index-url = http://172.29.236.100:8181/simple
constraint =
        http://172.29.236.100:8181/os-releases/17.0.0/ubuntu-16.04-x86_64/requirements_absolute_requirements.txt
trusted-host =
        172.29.236.100

[install]
upgrade = True
pre = True
find-links =
        http://172.29.236.100:8181/os-releases/17.0.0/ubuntu-16.04-x86_64/
root@aio1:/opt/openstack-ansible#

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/547526

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/547526
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=6b94b21e18181eab1eddb1f3040c430051871a20
Submitter: Zuul
Branch: master

commit 6b94b21e18181eab1eddb1f3040c430051871a20
Author: Jesse Pretorius <email address hidden>
Date: Fri Feb 23 17:34:41 2018 +0000

    Isolate the Ansible bootstrap

    As we move forward with the python build/install simplification,
    more of the config is being applied in the pip conf, rather than
    being given as CLI options in each ansible task. This is good
    for providing a consistent experience, but has side effects for
    things like the Ansible bootstrap.

    When bootstrapping Ansible, we want to make use of a specific
    set of constraints which is not polluted by additional config
    on the host. We do not want the constraints from the repo
    server because they happen to include a constraint for Ansible
    which clashes with our requirement. As such, we only want the
    native OpenStack upper-constraints.

    In this patch we use the --isolated CLI option to ignore the
    host's pip configuration. We also introduce the ability to set
    extra pip install options by using the PIP_OPTS environment
    variable. This can be used to set the installation to use an
    alternative pypi mirror, or for extra links, etc.

    Change-Id: Ic966bafd04c4c01b3d93851a0e3ec2c1f3312f28
    Closes-Bug: #1751316

[Jun Go (gojun077)]

Request: Could you also apply this fix to "stable/pike" branch in addition to master?

Thanks

[Jesse Pretorius (jesse-pretorius)]

@Jun Can you please explain why it's necessary to back port it, given that the problem as reported does not exist in stable/pike?

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible 18.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible 18.0.0.0b2 development milestone.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/625529

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/625572

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/625572
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=e86956fec6c253307d0f974247477dd165a8f124
Submitter: Zuul
Branch: stable/pike

commit e86956fec6c253307d0f974247477dd165a8f124
Author: Jesse Pretorius <email address hidden>
Date: Fri Feb 23 17:34:41 2018 +0000

    Isolate the Ansible bootstrap

    As we move forward with the python build/install simplification,
    more of the config is being applied in the pip conf, rather than
    being given as CLI options in each ansible task. This is good
    for providing a consistent experience, but has side effects for
    things like the Ansible bootstrap.

    When bootstrapping Ansible, we want to make use of a specific
    set of constraints which is not polluted by additional config
    on the host. We do not want the constraints from the repo
    server because they happen to include a constraint for Ansible
    which clashes with our requirement. As such, we only want the
    native OpenStack upper-constraints.

    In this patch we use the --isolated CLI option to ignore the
    host's pip configuration. We also introduce the ability to set
    extra pip install options by using the PIP_OPTS environment
    variable. This can be used to set the installation to use an
    alternative pypi mirror, or for extra links, etc.

    Change-Id: Ic966bafd04c4c01b3d93851a0e3ec2c1f3312f28
    Closes-Bug: #1751316
    (cherry picked from commit 6b94b21e18181eab1eddb1f3040c430051871a20)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/625529
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=35e57ff59dd89d658cf577b244b544bc1d938b0a
Submitter: Zuul
Branch: stable/queens

commit 35e57ff59dd89d658cf577b244b544bc1d938b0a
Author: Jesse Pretorius <email address hidden>
Date: Fri Feb 23 17:34:41 2018 +0000

    Isolate the Ansible bootstrap

    As we move forward with the python build/install simplification,
    more of the config is being applied in the pip conf, rather than
    being given as CLI options in each ansible task. This is good
    for providing a consistent experience, but has side effects for
    things like the Ansible bootstrap.

    When bootstrapping Ansible, we want to make use of a specific
    set of constraints which is not polluted by additional config
    on the host. We do not want the constraints from the repo
    server because they happen to include a constraint for Ansible
    which clashes with our requirement. As such, we only want the
    native OpenStack upper-constraints.

    In this patch we use the --isolated CLI option to ignore the
    host's pip configuration. We also introduce the ability to set
    extra pip install options by using the PIP_OPTS environment
    variable. This can be used to set the installation to use an
    alternative pypi mirror, or for extra links, etc.

    Change-Id: Ic966bafd04c4c01b3d93851a0e3ec2c1f3312f28
    Closes-Bug: #1751316
    (cherry picked from commit 6b94b21e18181eab1eddb1f3040c430051871a20)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible 16.0.24 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible 17.1.6 release.