centos-7 ovb jobs failing in build-test-packages due to pip install pip failure

Bug #1868120 reported by wes hayutin
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Sorin Sbarnea

Bug Description

https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/build-test-packages/tasks/main.yml#L104-L110

'https://logserver.rdoproject.org/55/704655/1/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/a6f07e6/job-output.txt

020-03-19 14:40:01.661677 | primary | TASK [build-test-packages : Install and update pip] ****************************
2020-03-19 14:40:01.681876 | primary | Thursday 19 March 2020 14:40:01 +0000 (0:00:03.511) 0:04:03.065 ********
2020-03-19 14:40:03.091614 | primary | fatal: [undercloud]: FAILED! => {
2020-03-19 14:40:03.091723 | primary | "changed": false
2020-03-19 14:40:03.091752 | primary | }
2020-03-19 14:40:03.091770 | primary |
2020-03-19 14:40:03.091790 | primary | MSG:
2020-03-19 14:40:03.091806 | primary |
2020-03-19 14:40:03.091902 | primary | Could not get output from /usr/bin/virtualenv --help: Traceback (most recent call last):
2020-03-19 14:40:03.091986 | primary | File "/usr/bin/virtualenv", line 3, in <module>
2020-03-19 14:40:03.092034 | primary | virtualenv.main()
2020-03-19 14:40:03.092099 | primary | AttributeError: 'module' object has no attribute 'main'
2020-03-19 14:40:03.092115 | primary |

This is image related... tested locally on a centos-7 box

[root@localhost ~]# ansible-playbook /var/tmp/playbook.yml
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'

PLAY [recreate] *******************************************************************************************************************************************************************************

TASK [Gathering Facts] ************************************************************************************************************************************************************************
ok: [localhost]

TASK [Install and update pip] *****************************************************************************************************************************************************************
changed: [localhost]

PLAY RECAP ************************************************************************************************************************************************************************************
localhost : ok=2 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

[root@localhost ~]# cat /var/tmp/playbook.yml
---
- name: recreate
  hosts: localhost
  tasks:
    - name: Install and update pip
      # noqa 403
      pip:
        name: pip
        virtualenv: "/tmp/wes/dlrn-venv"
        state: latest
        extra_args: --upgrade
[root@localhost ~]# cat /etc/redhat-release
CentOS Linux release 7.7.1908 (Core)

Revision history for this message
wes hayutin (weshayutin) wrote :

Example of a failing patch
https://review.opendev.org/#/c/713731/

Revision history for this message
wes hayutin (weshayutin) wrote :

Jobs just started failing on this

https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001

tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 openstack/tripleo-common stable/rocky openstack-check 704655,1 1299 2020-03-19T14:24:56 FAILURE
tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 openstack/tripleo-common stable/train openstack-check 713195,3 1169 2020-03-19T13:40:29 FAILURE
tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 openstack/tripleo-common stable/train openstack-check 713880,1 1311 2020-03-19T13:37:06 FAILURE
tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 openstack/tripleo-ansible stable/train openstack-check 713731,1 1151 2020-03-19T13:03:17 FAILURE
tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 openstack/tripleo-common stable/stein openstack-check 713872,1 1127 2020-03-19T13:12:22 FAILURE
tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 openstack/tripleo-common stable/train openstack-check 713810,1 1122 2020-03-19T10:19:43 FAILURE
tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 openstack/tripleo-common stable/train openstack-check 713810,1 1167 2020-03-19T09:47:20 FAILURE
tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 openstack/tripleo-ansible stable/train openstack-check 713731,1 9660 2020-03-18T20:06:07 SUCCESS
tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 openstack/tripleo-common stable/train openstack-check 713722,1 10038 2020-03-18T19:36:10 SUCCESS
tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 openstack/tripleo-ansible stable/train openstack-check 707380,2 9774 2020-03-18T19:04:06 SUCCESS

tags: added: promotion-blocker
Revision history for this message
Sagi (Sergey) Shnaidman (sshnaidm) wrote :
Revision history for this message
wes hayutin (weshayutin) wrote :
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-quickstart-extras (master)

Fix proposed to branch: master
Review: https://review.opendev.org/713937

Changed in tripleo:
assignee: nobody → wes hayutin (weshayutin)
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.opendev.org/713949

Changed in tripleo:
assignee: wes hayutin (weshayutin) → Sorin Sbarnea (ssbarnea)
Revision history for this message
yatin (yatinkarel) wrote :

<< probable cause https://pypi.org/project/virtualenv/20.0.12/

I think issue is happening after new virtualenv package is udpated in CentOS mirrors http://mirror.centos.org/centos/7/updates/x86_64/Packages/.

The issue happened because in jobs virtualenv package got updated as new package is available in CentOS7 mirrors which updated /usr/bin/virtualenv(virtualenv.main()) and that is not compatible with new virtualenv installed from pypi sources.

Issue happened only in third party jobs as i see image used there is build on 2020-03-10 14:41, newly built images with latest CentOS7 packages shouldn't have this issue.

Changed in tripleo:
assignee: Sorin Sbarnea (ssbarnea) → wes hayutin (weshayutin)
Revision history for this message
yatin (yatinkarel) wrote :

<< Issue happened only in third party jobs as i see image used there is build on 2020-03-10 14:41, newly built images with latest CentOS7 packages shouldn't have this issue.

New nodepool images are built today[1] which contains latest virtualenv package and jobs getting green now with new images, image build had issues from some time due to which images were not build, build issue is fixed with https://softwarefactory-project.io/r/#/c/17854/

Changed in tripleo:
assignee: wes hayutin (weshayutin) → Sorin Sbarnea (ssbarnea)
wes hayutin (weshayutin)
Changed in tripleo:
milestone: ussuri-3 → ussuri-rc3
wes hayutin (weshayutin)
Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tripleo-quickstart-extras (master)

Change abandoned by Sorin Sbarnea (<email address hidden>) on branch: master
Review: https://review.opendev.org/713937
Reason: Fix released, no longer needed.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Change abandoned by zbr (<email address hidden>) on branch: master
Review: https://review.opendev.org/713949
Reason: too_old

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.