build-containers jobs failing on "no module named http"

[wes hayutin]

2020-02-10 17:41:14 | Requirement already up-to-date: decorator in ./venv_build/lib/python2.7/site-packages (4.4.1)
2020-02-10 17:41:18 | WARNING: Failed to import plugin baremetal.
2020-02-10 17:41:18 | Exception raised: No module named http
2020-02-10 17:41:18 | ================= COMPLETE ====================
2020-02-10 17:41:18 | find: ‘/tmp/kolla-*’: No such file or directory

https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_190/705323/2/gate/tripleo-build-containers-centos-7/1901532/logs/build.log

https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_284/705323/2/gate/tripleo-build-containers-centos-7-buildah/2848b37/logs/build.log

[wes hayutin (weshayutin)]

has to be one of the following

https://opendev.org/openstack/kolla/src/branch/master/requirements.txt

[wes hayutin (weshayutin)]

perhaps this change?

https://opendev.org/openstack/requirements/commit/4e86eea7c78e56b4ddd0dda647fd1c29a977b3c2

[wes hayutin (weshayutin)]

could just be a mirror issue?

[wes hayutin (weshayutin)]

hrm.. it's not the kolla requirements

ls /home/whayutin/virtualenv/python2/lib/python2.7/site-packages
babel funcsigs MarkupSafe-1.1.1.dist-info pip six.py
Babel-2.8.0.dist-info funcsigs-1.0.2.dist-info monotonic-1.5.dist-info pip-20.0.2.dist-info six.pyc
backports git monotonic.py pkg_resources smmap
backports.ssl_match_hostname-3.7.0.1.dist-info gitdb monotonic.pyc pyparsing-2.4.6.dist-info smmap2-2.0.5.dist-info
certifi gitdb2-2.0.6.dist-info netaddr pyparsing.py stevedore
certifi-2019.11.28.dist-info GitPython-2.1.11.dist-info netaddr-0.7.19.dist-info pyparsing.pyc stevedore-1.31.0.dist-info
chardet idna netifaces-0.10.9.dist-info pytz urllib3
chardet-3.0.4.dist-info idna-2.8.dist-info netifaces.so pytz-2019.3.dist-info urllib3-1.25.8.dist-info
debtcollector ipaddress-1.0.23.dist-info oslo_config PyYAML-5.3.dist-info websocket
debtcollector-1.22.0.dist-info ipaddress.py oslo.config-7.0.0.dist-info requests websocket_client-0.57.0.dist-info
docker ipaddress.pyc oslo_i18n requests-2.22.0.dist-info wheel
docker-4.2.0.dist-info iso8601 oslo.i18n-3.25.1.dist-info rfc3986 wheel-0.34.2.dist-info
easy_install.py iso8601-0.1.12.dist-info oslo_utils rfc3986-1.3.2.dist-info wrapt
easy_install.pyc jinja2 oslo.utils-3.42.1.dist-info setuptools wrapt-1.11.2.dist-info
enum Jinja2-2.11.1.dist-info pbr setuptools-44.0.0.dist-info yaml
enum34-1.1.6.dist-info markupsafe pbr-5.4.4.dist-info six-1.14.0.dist-info

[wes hayutin (weshayutin)]

Can't recreate this issue locally on centos-8, AH..
and it works on centos-8
https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_a11/706572/14/check/tripleo-build-containers-centos-8/a11954a/

OK.. trying on centos-7

[wes hayutin (weshayutin)]

hit's all centos-7 container builds.
https://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-7-train-containers-build-push/c7d1abb/logs/build.log

def.. a blocker localized to python2.7 and centos-7

[wes hayutin (weshayutin)]

ok.. have a centos-7 reproducer..

(venv_build) [zuul@wes-centos-7 workspace]$ openstack --debug overcloud container image build \
> --kolla-config-file /home/zuul/workspace/kolla-build.conf \
> --exclude opendaylight \
> --list-images
WARNING: Failed to import plugin baremetal.
START with options: --debug overcloud container image build --kolla-config-file /home/zuul/workspace/kolla-build.conf --exclude opendaylight --list-images
options: Namespace(access_token='***', access_token_endpoint='', access_token_type='', application_credential_id='', application_credential_name='', application_credential_secret='***', auth_methods='', auth_type='', auth_url='', cacert=None, cert='', client_id='', client_secret='***', cloud='', code='', debug=True, default_domain='default', default_domain_id='', default_domain_name='', deferred_help=False, discovery_endpoint='', domain_id='', domain_name='', endpoint='', identity_provider='', insecure=None, inspector_api_version='1', inspector_url=None, interface='public', key='', log_file=None, openid_scope='', os_beta_command=False, os_compute_api_version='', os_identity_api_version='', os_image_api_version='', os_network_api_version='', os_object_api_version='', os_orchestration_api_version='1', os_project_id=None, os_project_name=None, os_queues_api_version='2', os_tripleoclient_api_version='2', os_volume_api_version='', os_workflow_api_version='2', passcode='', password='***', project_domain_id='', project_domain_name='', project_id='', project_name='', protocol='', redirect_uri='', region_name='', remote_project_domain_id='', remote_project_domain_name='', remote_project_id='', remote_project_name='', service_provider='', system_scope='', timing=False, token='***', trust_id='', user_domain_id='', user_domain_name='', user_id='', username='', verbose_level=3, verify=None)
Auth plugin password selected
auth_config_hook(): {'auth_type': 'password', 'beta_command': False, u'image_status_code_retries': '5', 'tripleoclient_api_version': '2', 'orchestration_api_version': '1', 'timing': False, 'inspector_api_version': '1', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], 'queues_api_version': '2', 'verify': True, u'object_store_api_version': u'1', u'status': u'active', 'verbose_level': 3, 'region_name': '', u'baremetal_introspection_status_code_retries': '5', 'api_timeout': None, 'auth': {}, 'default_domain': 'default', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', 'key': None, u'interface': 'public', 'cacert': None, 'workflow_api_version': '2', u'baremetal_status_code_retries': '5', 'deferred_help': False, 'cert': None, u'secgroup_source': u'neutron', 'debug': True, u'disable_vendor_agent': {}}
defaults: {u'auth_type': 'password', u'status': u'active', u'image_status_code_retries': 5, u'baremetal_introspection_status_code_retries': 5, 'api_timeout': None, 'cacert': None, u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', 'key': None, u'interface': None, u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', u'baremetal_status_code_retries': 5, 'verify': True, 'cert': None, u'secgroup_source'...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/707054

[Marios Andreou (marios-b)]

need that one too please [1]:

ci-config master: Add ppc_manifests check and switch to manifest_push to exclude ppc https://review.rdoproject.org/r/24807

[1] also seen in the tripleo-ci-promotion-staging job in ci-config at https://logserver.rdoproject.org/07/24807/12/check/tripleo-ci-promotion-staging/9e1bc7b/logs/tripleo-container-image.prepare.log

[Marios Andreou (marios-b)]

sorry for comment #9 i mean to point to

ci-config master: Pin ironicclient under 4.0.0 for py27 ci-promotion-staging job
                   https://review.rdoproject.org/r/24913

the one on comment 9 comes on top and is not related to this bug thanks

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/707054
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=a05d173feeaac3abef505feae3de11a1a008b261
Submitter: Zuul
Branch: master

commit a05d173feeaac3abef505feae3de11a1a008b261
Author: Wes Hayutin <email address hidden>
Date: Mon Feb 10 20:59:48 2020 -0700

    pin ironicclient under 4.0.0 for py27

    Depends-On: https://review.opendev.org/#/c/707062
    Depends-On: https://review.opendev.org/707204

    Closes-Bug: #1862674
    Change-Id: I50b9acad43095577dafc665f80300f3a7dc38ee7

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/707927

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/707927
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=712d8d7a6103e22e8c4703929bccdc53653d2f04
Submitter: Zuul
Branch: master

commit 712d8d7a6103e22e8c4703929bccdc53653d2f04
Author: Wes Hayutin <email address hidden>
Date: Fri Feb 14 16:39:06 2020 -0700

    update ironclient requirements

    Partial-Bug: #1862674
    Change-Id: I325761643bd0c765e1759905e5cccd80dafa742f

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-common 12.1.0 release.