RAX yum mirror issues Error: python-libs conflicts, ethtool 404 etc

[wes hayutin]

gate job failures across the board here. This is not a tripleo issue, however documenting it here.

2018-05-10 01:57:17 | ---> Package python-ply.noarch 0:3.4-11.el7 will be installed
2018-05-10 01:57:17 | --> Processing Conflict: python-libs-2.7.5-68.el7.x86_64 conflicts python-virtualenv < 15.1.0-1
2018-05-10 01:57:17 | --> Finished Dependency Resolution
2018-05-10 01:57:17 | Error: python-libs conflicts with python-virtualenv-1.10.1-4.el7.noarch
2018-05-10 01:57:17 | You could try using --skip-broken to work around the problem
2018-05-10 01:57:17 | You could try running: rpm -Va --nofiles --nodigest

http://logs.openstack.org/84/564784/4/gate/tripleo-ci-centos-7-undercloud-containers/00d2e30/logs/undercloud/home/zuul/repo_setup.log.txt.gz

http://logs.openstack.org/84/564784/4/gate/tripleo-ci-centos-7-undercloud-oooq/8fc99f9/logs/undercloud/home/zuul/repo_setup.log.txt.gz

[wes hayutin (weshayutin)]

This happening on the rax cloud, not sure if other cloud providers are hitting the same issue yet.

[quickstart-centos-base]
name=quickstart-centos-base
baseurl=http://mirror.dfw.rax.openstack.org/centos/7/os/x86_64/
gpgcheck=0
enabled=1

[wes hayutin (weshayutin)]

also seen in the overcloud-deploy

http://logs.openstack.org/20/561920/6/gate/tripleo-ci-centos-7-scenario001-multinode-oooq-container/5647d61/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-05-10_00_58_02

"[2018-05-10 00:57:59,229] (heat-config) [DEBUG] + yum install -y jq python-ipaddress puppet-tripleo os-net-config openvswitch 'python-heat-agent*' openstack-selinux",
2018-05-10 00:58:02 | "http://mirror.ord.rax.openstack.org/centos/7/os/x86_64/Packages/ethtool-4.8-1.el7.x86_64.rpm: [Errno 14] HTTP Error 404 - Not Found",
2018-05-10 00:58:02 | "Trying other mirror.",

[Ian Wienand (iwienand)]

I think what has happened here is that we pin python-virtualenv, setuptools, and pip packages on our hosts so that packages don't overwrite the upstream versions we install (see [1])

I think the problem here is that upstream has released new versions, and so now we're getting conflicts on packages that want these newer versions.

The quickest way to deal with this is probably to rebuild teh centos-7 images so they have the newer packages pinned. I've started that build now (see [2])

[1] https://git.openstack.org/cgit/openstack/diskimage-builder/tree/diskimage_builder/elements/pip-and-virtualenv/install.d/pip-and-virtualenv-source-install/04-install-pip#n146
[2] http://nl01.openstack.org/dib-image-list

[Ian Wienand (iwienand)]

Images should have been updated ... LMN if this resolves it!

[Sagi (Sergey) Shnaidman (sshnaidm)]

Jobs passed successfully this step, so it's resolved now. Thanks, Ian!

[Sagi (Sergey) Shnaidman (sshnaidm)]

Just noticed it still appears in RDO cloud jobs: https://logs.rdoproject.org/71/530071/3/openstack-check/gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master/Z5b0135bce708475cbd0a2c86883fa625/undercloud/home/jenkins/repo_setup.log.txt.gz

[wes hayutin (weshayutin)]

Thanks Ian!

[Jeremy Stanley (fungi)]

This pinning of RPM package versions to avoid clobbering Python libraries which have been globally installed with pip seems like a very fragile workaround, as evidenced here. Ultimately we should avoid mixing different package managers within the same Python context (e.g. with `pip install --local ...` or deep symlinks info virtualenvs).

[Ian Wienand (iwienand)]

@fungi -- yes, but it's down to our usual problems of the whole world not using virtualenvs. We've always needed a very up-to-date pip as we've had problems in the past with packaged versions not handling the requirements.txt bits we use, and stuff like that which I'm sure you're aware of. This just ensures there's a late-enough pip when people do "pip install foo" that it doesn't blow up. You really have to have the packages pinned, as mixing the two creates a big old mess.