periodic-tripleo-ci-centos-8-scenario010-standalone-network-master deploy fails with package conflict - "openstack-neutron-common-1:16.0.0-0.20200318094753.8c7ca77.el8.noarch conflicts"

Bug #1867931 reported by Ronelle Landy
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Unassigned

Bug Description

periodic-tripleo-ci-centos-8-scenario010-standalone-network-master is failing to deploy standalone:

Error: Transaction check error:\n file /etc/neutron/plugins/networking-ovn/networking-ovn.ini from install of openstack-neutron-common-1:16.0.0-0.20200318094753.8c7ca77.el8.noarch conflicts with file from package python3-networking-ovn-7.0.0-0.20200109131747.eda5d7f.el8.noarch\n \n level=debug msg="error running [bash -x /tmp/yum_update.sh delorean-current,quickstart-centos-ceph-nautilus,network ] in container \\"centos-binary-octavia-api-working-container\\": error while running runtime: exit status 1"\n stderr_lines: <omitted>\n stdout: |-\n Last metadata expiration check: 0:00:36 ago on Wed 18 Mar 2020 12:29:13 PM UTC.\n Dependencies resolved.\n

The full deploy log is below:

https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-standalone-network-master/5127081/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

This error hit in the networking component pipeline after https://review.rdoproject.org/r/#/c/24462/ merged to upin neutron.

Ronelle Landy (rlandy)
Changed in tripleo:
milestone: none → ussuri-3
importance: Undecided → Critical
status: New → Triaged
Revision history for this message
wes hayutin (weshayutin) wrote :

  Obsoleting : python3-networking-ovn-7.0.0-0.20200109131747.eda5d7f.el8.noarch 42/49

when testing manually

Revision history for this message
wes hayutin (weshayutin) wrote :
Download full text (5.4 KiB)

I think the issue is that /etc/neutron/plugins/networking-ovn/networking-ovn.ini is configured in the container. When python3-networking-ovn is obsoleted the rpm clash on which rpm owns /etc/neutron/plugins/networking-ovn/networking-ovn.ini

I think the rpm should handle the change in rpm ownership of the file. I don't think a fresh install would hit this. So we could rebuild the containers as well however this may be masking an issue hit later on.

      Obsoleting : python3-networking-ovn-7.0.0-0.20200109131747.eda5 14/23
.....
      Verifying : python3-networking-ovn-7.0.0-0.20200109131747.eda5 3/23

    Upgraded:
      openstack-neutron-1:16.0.0-0.20200318094753.8c7ca77.el8.noarch
      openstack-neutron-common-1:16.0.0-0.20200318094753.8c7ca77.el8.noarch
      openstack-neutron-ml2-1:16.0.0-0.20200318094753.8c7ca77.el8.noarch
      python3-neutron-1:16.0.0-0.20200318094753.8c7ca77.el8.noarch
      python3-vmware-nsxlib-15.0.7-0.20200317013238.c2f0480.el8.noarch
      openstack-tripleo-common-container-base-12.1.1-0.20200317201257.6f4cdee.el8.noarch
      puppet-concat-6.2.1-0.20200316161547.596d8cc.el8.noarch
      puppet-glance-16.1.1-0.20200316163251.1e628f8.el8.noarch
      puppet-kafka-5.3.1-0.20200317212736.3cc9d4a.el8.noarch
      puppet-keystone-16.1.1-0.20200316155745.c9377a8.el8.noarch
      puppet-vcsrepo-3.1.1-0.20200316163851.e1b5e67.el8.noarch

....
2020-03-18 12:29:19,099 p=48123 u=root | fatal: [localhost]: FAILED! => changed=true
  cmd:
  - buildah
  - run
  - --volume
  - /tmp/ansible.jxv5rb9b:/tmp/yum_update.sh
  - --volume
  - /etc/yum.repos.d:/etc/yum.repos.d
  - --volume
  - /etc/pki:/etc/pki
  - --user
  - root
  - --net
  - host
  - centos-binary-octavia-api-working-container
  - /tmp/yum_update.sh
  - 'delorean-current,quickstart-centos-ceph-nautilus,network '
  delta: '0:00:19.722724'
  end: '2020-03-18 12:29:19.025894'
  msg: non-zero return code
  rc: 1
  start: '2020-03-18 12:28:59.303170'
  stderr: |-
    Error: Transaction check error:
      file /etc/neutron/plugins/networking-ovn/networking-ovn.ini from install of openstack-neutron-common-1:16.0.0-0.20200318094753.8c7ca77.el8.noarch conflicts with file from package python3-networking-ovn-7.0.0-0.20200109131747.eda5d7f.el8.noarch
  stderr_lines: <omitted>
  stdout: |-
    delorean-python-ironic-tests-tempest-2dd726ae88 163 kB/s | 47 kB 00:00
    delorean-openstack-cinder-fcbfa927a39add44d633b 62 kB/s | 26 kB 00:00
    delorean-python-os-vif-d5b61d10655b3b7a9d32378b 486 kB/s | 127 kB 00:00
    delorean-openstack-aodh-59a36b714623ceec6d8f73b 95 kB/s | 25 kB 00:00
    delorean-openstack-congress-85243abf63dfc7c086e 1.0 MB/s | 294 kB 00:00
    delorean-openstack-nova-e483ca1cd9a5db5856f87fc 151 kB/s | 43 kB 00:00
    delorean-python-glance-store-4f5b1f403b22f94f52 63 kB/s | 17 kB 00:00
    delorean-python-manila-tests-tempest-39ef895d11 87 kB/s | 23 kB 00:00
    delorean-openstack-designate-46dc595f21f758b31e 593 kB/s | 155 kB 00:00
    delorean-python-octavia-tests-tempest-b0eb0aa97 87 kB/s | 23 kB 00:00
    delorean-python-castellan-141e7e4209b96b83ad61e 183 kB/s | 50 kB 00...

Read more...

tags: added: promotion-blocker
Revision history for this message
Alfredo Moralejo (amoralej) wrote :
Download full text (43.2 KiB)

The new rpm is handling the file ownership change properly, see output of a manual rpm update. I think the problem is somewhere in yum_update.sh script or repos configuration, could i reproduce the issue using the same ansible module out of the CI?

# docker run -u root -it --rm docker.io/tripleomaster/centos-binary-octavia-api:46b71a6620e3372c998db7a694112fd2 /bin/bash
()[root@8975d7f6b8fb /]# cd /etc/yum.repos.d/
()[root@8975d7f6b8fb yum.repos.d]# curl -o delorean-network.repo https://trunk.rdoproject.org/centos8-master/component/network/consistent/delorean.repo
curl (https://trunk.rdoproject.org/centos8-master/component/network/consistent/delorean.repo): response: 200, time: 0.470607, size: 264
()[root@8975d7f6b8fb yum.repos.d]# cat delorean-network.repo
[delorean-component-network]
name=delorean-python-neutron-tests-tempest-7729b6afac9227fe78d0782d64daa71b4a87f50e
baseurl=https://trunk.rdoproject.org/centos8/component/network/77/29/7729b6afac9227fe78d0782d64daa71b4a87f50e_21e2b632
enabled=1
gpgcheck=0
priority=1
()[root@8975d7f6b8fb yum.repos.d]# sed -i 's/delorean-component-network/delorean-component-network-consistent/g' delorean-network.repo
()[root@8975d7f6b8fb yum.repos.d]#
()[root@8975d7f6b8fb yum.repos.d]#
()[root@8975d7f6b8fb yum.repos.d]#
()[root@8975d7f6b8fb yum.repos.d]# dnf update
delorean-python-neutron-tests-tempest-7729b6afac9227fe78d0782d64daa71b4a87f50e 127 kB/s | 158 kB 00:01
delorean-python-ironic-tests-tempest-2dd726ae8849e0f405390393989999d58d47c56d 46 kB/s | 47 kB 00:01
delorean-openstack-cinder-fcbfa927a39add44d633b7ef8e987a9ed109fc54 25 kB/s | 26 kB 00:01
delorean-python-os-vif-d5b61d10655b3b7a9d32378b5f7bcdb06479e15d 96 kB/s | 127 kB 00:01
delorean-openstack-aodh-59a36b714623ceec6d8f73ba3e1ce9309b714e29 25 kB/s | 25 kB 00:00
delorean-openstack-congress-85243abf63dfc7c086e28e9bdb3fb0b7c9d2ad94 76 kB/s | 294 kB 00:03
delorean-openstack-nova-e483ca1cd9a5db5856f87fc69ca07c42d2be5def 42 kB/s | 43 kB 00:01
delorean-python-glance-store-4f5b1f403b22f94f529bd249552c8e81373b81ab 19 kB/s | 17 kB 00:00
delorean-python-manila-tests-tempest-39ef895d112f947564d7ca52a73a5b2afb82f741 23 kB/s |...

Revision history for this message
chandan kumar (chkumar246) wrote :

Since we are not able to reproduce locally. So we are waiving off the component criteria temporally. https://review.rdoproject.org/r/25996 so that integration pipeline, container builds can build fresh packages

Revision history for this message
Alfredo Moralejo (amoralej) wrote :

New packages are covering the change of file ownership files and the packages upgrade works fine.

I've been able to reproduce the issue using tripleo-modify-image and the problem is actually in yum_update.sh script which is not covering the upgrades when there are new packages with obsoletes/provides. The yum command is trying to build is trying is:

dnf -y update openstack-neutron-common openstack-tripleo-common-container-base puppet-concat puppet-glance puppet-kafka puppet-keystone puppet-midonet puppet-vcsrepo python3-neutron

That command is wrong as in this particular case it needs to install openstack-neutron package which has the obsolets/provides to python3-networking-ovn.

I see yum_update.sh has logic to identify the packages to be updated but it just cover basic cases. IMO, trying to replicate in bash the logic that yum/dnf has to resolve deps/provides/obsoletes chains is wrong and a different approach would be better of just let package manager do it's work. I understant the reason for this is to avoid updating packages from deps or OS repos and limit to only delorean-current and component ones, but using current approach will be hard and easy to reach corner cases like this.

As workaround for this particular case, i think we can move provides/obsoletes to openstack-neutron-common or python3-network in packaging, i'll propose it and discuss with neutron team.

Revision history for this message
Alfredo Moralejo (amoralej) wrote :

Let's see if https://review.rdoproject.org/r/25997 helps to mitigate.

Revision history for this message
Alfredo Moralejo (amoralej) wrote :

Issue fixed by https://review.rdoproject.org/r/25997. As explained in comment #5 see that similar cases may be hit in future given the logic to discover packages to be updated in yum_update.sh.

wes hayutin (weshayutin)
Changed in tripleo:
milestone: ussuri-3 → ussuri-rc3
wes hayutin (weshayutin)
Changed in tripleo:
milestone: ussuri-rc3 → victoria-1
Changed in tripleo:
milestone: victoria-1 → victoria-3
wes hayutin (weshayutin)
Changed in tripleo:
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.