[master][standalone-upgrade] Fails while applying ovn controller puppet config with Error: Failed to apply catalog: Execution of '/usr/bin/ovs-vsctl list Open_vSwitch .' returned 1: ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)

[Bhagyashri Shewale]

Error log:

2020-09-30T02:32:51.382582920+00:00 stderr F + /usr/bin/puppet apply --summarize --detailed-exitcodes --color=false --modulepath=/etc/puppet/modules:/usr/share/openstack-puppet/modules --tags '"file,file_line,concat,augeas,cron,vs_config,exec"' /etc/config.pp
2020-09-30T02:32:55.185977218+00:00 stderr F <13>Sep 30 02:32:51 puppet-user: Warning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5
2020-09-30T02:32:55.185977218+00:00 stderr F <13>Sep 30 02:32:55 puppet-user: (file: /etc/puppet/hiera.yaml)
2020-09-30T02:32:55.186248907+00:00 stderr F <13>Sep 30 02:32:55 puppet-user: Warning: Undefined variable '::deploy_config_name';
2020-09-30T02:32:55.186248907+00:00 stderr F <13>Sep 30 02:32:55 puppet-user: (file & line not available)
2020-09-30T02:32:55.244780258+00:00 stderr F <13>Sep 30 02:32:55 puppet-user: Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://puppet.com/docs/puppet/6.14/deprecated_language.html
2020-09-30T02:32:55.244780258+00:00 stderr F <13>Sep 30 02:32:55 puppet-user: (file & line not available)
2020-09-30T02:32:55.465105657+00:00 stderr F <13>Sep 30 02:32:55 puppet-user: Notice: Compiled catalog for standalone.localdomain in environment production in 0.30 seconds
2020-09-30T02:32:55.540213621+00:00 stderr F <13>Sep 30 02:32:55 puppet-user: Error: Could not prefetch vs_config provider 'ovs': Execution of '/usr/bin/ovs-vsctl list Open_vSwitch .' returned 1: ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)
2020-09-30T02:32:55.547209885+00:00 stderr F <13>Sep 30 02:32:55 puppet-user: Error: Failed to apply catalog: Execution of '/usr/bin/ovs-vsctl list Open_vSwitch .' returned 1: ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)
2020-09-30T02:32:55.547209885+00:00 stderr F <13>Sep 30 02:32:55 puppet-user: Changes:
2020-09-30T02:32:55.547209885+00:00 stderr F <13>Sep 30 02:32:55 puppet-user: Events:
2020-09-30T02:32:55.547209885+00:00 stderr F <13>Sep 30 02:32:55 puppet-user: Resources:
2020-09-30T02:32:55.547209885+00:00 stderr F <13>Sep 30 02:32:55 puppet-user: Skipped: 3
2020-09-30T02:32:55.547209885+00:00 stderr F <13>Sep 30 02:32:55 puppet-user: Total: 3
2020-09-30T02:32:55.547209885+00:00 stderr F <13>Sep 30 02:32:55 puppet-user: Time:
2020-09-30T02:32:55.547230968+00:00 stderr F <13>Sep 30 02:32:55 puppet-user: Config retrieval: 0.38
2020-09-30T02:32:55.547230968+00:00 stderr F <13>Sep 30 02:32:55 puppet-user: Last run: 1601433175
2020-09-30T02:32:55.547230968+00:00 stderr F <13>Sep 30 02:32:55 puppet-user: Total: 0.03
2020-09-30T02:32:55.547230968+00:00 stderr F <13>Sep 30 02:32:55 puppet-user: Version:
2020-09-30T02:32:55.547239458+00:00 stderr F <13>Sep 30 02:32:55 puppet-user: Config: 1601433175
2020-09-30T02:32:55.547245572+00:00 stderr F <13>Sep 30 02:32:55 puppet-user: Puppet: 6.14.0
2020-09-30T02:32:55.617656987+00:00 stderr F + rc=1

Also impacting check job:- https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-standalone-upgrade

Logs link:

- https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-upgrade-master/90bb089/job-output.txt

- https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-upgrade-master/90bb089/logs/undercloud/home/zuul/standalone_upgrade.log.txt.gz

- https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-upgrade-master/90bb089/logs/undercloud/var/log/containers/stdouts/container-puppet-ovn_controller.log.txt.gz

This is happening because openvswitch service is in stopped state. And it went to stopped state after upgrade to openvswitch2.13-2.13. openvswitch needs to be in started state before upgrade get's triggered.

[Alfredo Moralejo (amoralej)]

So, what is happening in the upgrade job is:

1. After the repos for master are configured in the job, a " sudo dnf update -y" is executed which updates all packages in the host. This includes upgrading from old "non-layered" openvswitch-2.12 to new layered openvswitch2.13 via rdo-openvswitch-2.13. After this upgrade, openvswitch service is stopped, this is expected and a design decision.

https://logserver.rdoproject.org/51/29351/15/check/periodic-tripleo-ci-centos-8-standalone-upgrade-master/70ef433/logs/undercloud/home/zuul/repo_setup_upgrade.log.txt.gz

2. In the upgrade part, when the logic for ovs is executed, no ovs upgrade is detected (it has been updated previously) so the tast to start openvswitch is skipped:

2020-09-30 04:47:33 | 2020-09-30 04:47:33.428539 | fa163eea-bbcd-be32-770f-00000000082a | TASK | Special treatment for OpenvSwitch
2020-09-30 04:47:34 | 2020-09-30 04:47:34.478348 | fa163eea-bbcd-be32-770f-00000000082a | OK | Special treatment for OpenvSwitch | standalone
2020-09-30 04:47:34 | 2020-09-30 04:47:34.479476 | fa163eea-bbcd-be32-770f-00000000082a | TIMING | Special treatment for OpenvSwitch | standalone | 0:04:06.525630 | 1.05s
2020-09-30 04:47:34 | 2020-09-30 04:47:34.525550 | fa163eea-bbcd-be32-770f-00000000082b | TASK | Always ensure the openvswitch service is enabled and running after upgrades
2020-09-30 04:47:34 | 2020-09-30 04:47:34.550044 | fa163eea-bbcd-be32-770f-00000000082b | SKIPPED | Always ensure the openvswitch service is enabled and running after upgrades | standalone
2020-09-30 04:47:34 | 2020-09-30 04:47:34.551015 | fa163eea-bbcd-be32-770f-00000000082b | TIMING | Always ensure the openvswitch service is enabled and running after upgrades | standalone | 0:04:06.597170 | 0.02s

https://logserver.rdoproject.org/51/29351/15/check/periodic-tripleo-ci-centos-8-standalone-upgrade-master/70ef433/logs/undercloud/home/zuul/standalone_upgrade.log.txt.gz

So openvswitch is stopped in the host and upgrade process fails.

Note that currently the tripleo_ovs_upgrade.py module does not work with upstream wrapper as the package name is different, downstream is rhosp-openvswitch while upstream is rdo-openvswitch and also it seems it doesn't cover the upgrade from non-layered to layered.

https://opendev.org/openstack/tripleo-ansible/raw/branch/master/tripleo_ansible/ansible_plugins/modules/tripleo_ovs_upgrade.py

Possible solutions I can see:

1. Remove the "dnf update -y" from repo_setup_upgrade.log (iiuc there is some task later that would take care of updating host packages?) and fixing the tripleo_ovs_upgrade.py to cover this upstream case.

2. Keep the dnf update -y in repo_setup_upgrade.sh and add some task in the script or at a later stage to restart it before undercloud-upgrade tasks.

I think we need upgrades team here.

[Sofer Athlan-Guyot (sofer-athlan-guyot)]

Hi, thanks for the analysis, will look into it shortly.

[Marios Andreou (marios-b)]

also saw this at https://review.opendev.org/753403 just now

https://api.us-east.open-edge.io:8080/swift/v1/AUTH_e02c11e4e2c24efc98022353c88ab506/zuul_opendev_logs_ee3/753403/7/check/tripleo-ci-centos-8-standalone-upgrade/ee33c34/logs/undercloud/home/zuul/standalone_upgrade.log

 https://api.us-east.open-edge.io:8080/swift/v1/AUTH_e02c11e4e2c24efc98022353c88ab506/zuul_opendev_logs_ee3/753403/7/check/tripleo-ci-centos-8-standalone-upgrade/ee33c34/logs/undercloud/var/log/containers/stdouts/container-puppet-ovn_controller.log
        * 2020-09-29T13:31:47.643473262+00:00 stderr F <13>Sep 29 13:31:47 puppet-user: Error: Could not prefetch vs_config provider 'ovs': Execution of '/usr/bin/ovs-vsctl list Open_vSwitch .' returned 1: ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)
2020-09-29T13:31:47.649619522+00:00 stderr F <13>Sep 29 13:31:47 puppet-user: Error: Failed to apply catalog: Execution of '/usr/bin/ovs-vsctl list Open_vSwitch .' returned 1: ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)

[yatin (yatinkarel)]

Also additionally i checked why issue didn't hit in undercloud-upgrade job[1]. There i found that dnf update do not run as part of repo setup but as part of upgrade_tasks[2]. And at that stage openvswitch service stops which is similar to standalone upgrade. But openvswitch service get's started as part of tripleo_os_net_config[3] which is not happening in standalone-upgrade due to missing[4]. In that patch too it's not working as tripleo_network_config_with_ansible: false[5] likely because tht got updated at a later stage. /me don't know why openvswitch is getting started with tripleo_network_config_with_ansible: true and hide the issue in undercloud-upgrade job.

[1] https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-undercloud-upgrade
[2] https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/deployment/undercloud/undercloud-upgrade.yaml#L78-L83
[3] https://opendev.org/openstack/tripleo-ansible/src/branch/master/tripleo_ansible/roles/tripleo_network_config/tasks/os_net_config.yml#L42-L51
[4] https://review.opendev.org/#/c/754974/
[5] https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_2f8/754974/3/check/tripleo-ci-centos-8-standalone-upgrade/2f8b1f0/logs/undercloud/home/zuul/standalone-ansible-0c3wr89g/group_vars/Standalone

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/755238

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/755252

[Sofer Athlan-Guyot (sofer-athlan-guyot)]

Hi,

So the first thing to fix here is the ci job. It shouldn't run "yum
upgrade" outside of the upgrade/update procedure.

In tripleo-ovs-upgrade we should support the rdo-openvswtich packaging
but this won't solve this issue as we are moving from non-layered to
layered (I got that right, right ?). [0] should take care of this.

Now, here in tripleo-ovs-upgrade, I'm not sure we have anything to fix
as the non-layered to layered scenario should be covered. If not we
will fix it. We cannot reach conclusion until we know if the new
version of ovs has restart in postuninstall. We only know that the
previous had it :)

Removing the conditional for making sure ovs is started is an option,
but I'm not sure if I like it. In this context it would just hide an
issue (namely ci running yum upgrade before the upgrade/update
process). But ... maybe it's a sane thing to do in production context.

Full explanation and some other comments in my inline reply:

> 1. After the repos for master are configured in the job, a " sudo dnf
> update -y" is executed which updates all packages in the host. This
> includes upgrading from old "non-layered" openvswitch-2.12 to new
> layered openvswitch2.13 via rdo-openvswitch-2.13.

This has to be removed from upgrade/update jobs as it short cut any
mechanism we have in there. It's also ensure that we don't test the
upgrade/update the proper way as one if its main function has already
be done.

We need a fix there, I'm not really familiar with that part of the
code. Who could do it ?

> After this upgrade, openvswitch service is stopped, this is expected
> and a design decision.

I think at some point we should challenge that decision. This makes
the upgrade of that package unusable in production environment. You
upgrade your ovs, and, by design, it introduces a cut in your flows.

We have lost days and days and a created a good deal of unhappy
costumers because of that packaging decision.

> 2. In the upgrade part, when the logic for ovs is executed, no ovs
> upgrade is detected (it has been updated previously) so the tast to
> start openvswitch is skipped:

So this is expected and was a kind of optimization/validation. If we
don't special treat ovs, then no need to make sure it's running.

This issue challenge that view :)

It's easy to remove the conditional so that we always do the restart:
but then we could miss cases of ovs upgrade not being treated
specialy. Basically we would have never seen this issue. It's really
being convenient vs being right choice here (i think). I believe being
right should be favored, but maybe being conveniant/resiliant is
better.

I'm open to suggestion here.

> Note that currently the tripleo_ovs_upgrade.py module does not work
> with upstream wrapper as the package name is different, downstream is
> rhosp-openvswitch while upstream is rdo-openvswitch

So https://review.opendev.org/755252 should fix that in. Thanks for
point that out. Note however that this shouldn't fix that issue as we
are switching from non-layered to layered. So rdo-openvswitch
shouldn't be installed here (it is just because the ci run yum
upgrade, but in theory it shouldn't ... man this is so confusing)

> and als...

[Alfredo Moralejo (amoralej)]

Some notes wrt comment #7:

> Now, here in tripleo-ovs-upgrade, I'm not sure we have anything to fix
> as the non-layered to layered scenario should be covered. If not we
> will fix it. We cannot reach conclusion until we know if the new
> version of ovs has restart in postuninstall. We only know that the
> previous had it :)

I understand you mean the non-layered ovs, These are the uninstall scriptlets for the initially installed openvswitch-2.12.0:

preuninstall scriptlet (using /bin/sh):

if [ $1 -eq 0 ] ; then
        # Package removal, not upgrade
        systemctl --no-reload disable --now openvswitch.service &>/dev/null || :
fi
postuninstall program: /bin/sh

postuninstall is empty.

>> After this upgrade, openvswitch service is stopped, this is expected
>> and a design decision.

> I think at some point we should challenge that decision. This makes
> the upgrade of that package unusable in production environment. You
> upgrade your ovs, and, by design, it introduces a cut in your flows.
>
> We have lost days and days and a created a good deal of unhappy
> costumers because of that packaging decision.

I agree, I think this is related to the package renaming in FDP which was decided by that team, unrelated to RDO itself.

> It's easy to remove the conditional so that we always do the restart:
> but then we could miss cases of ovs upgrade not being treated
> specialy. Basically we would have never seen this issue. It's really
> being convenient vs being right choice here (i think). I believe being
> right should be favored, but maybe being conveniant/resiliant is
> better.

I think removing the conditional would not hurt as it's just make sure ovs is running but i understand we prefer to be aware if something is going in an unexpected path, so yes, i think it's good to keep it as is.

>> and also it seems it doesn't cover the upgrade from non-layered to
>> layered.

> So, we should support this case. I cannot reach any conclusion here
> because of the initial "ci yum upgrade", but it needs to be check.
>
> From reading the code, we go either in layered or not layered upgrade
> of ovs. We don't see the rdo-openvswitch layered product but we should
> see the non-layere product[1].
>
> Then we check if that version has restart in postuninstall to decide
> if we take action.

The upgrade should be:

openvswitch -> rdo-openvswitch + openvswitch2.13

I'd say this is not treated properly in the tripleo_ovs_upgrade.py. I've been doing some manual tests with a playbook which only runs that task and seems to not work (after replacing rhosp-openvswitch by rdo-openvswitch) in this case. I'd say it has some assumptions of upgrades non-layered to non-layered and layered to layered but not this case.

[Sofer Athlan-Guyot (sofer-athlan-guyot)]

For repo setup not running update we should have:

repo_setup_run_update: false

for upgrade/update jobs. It seems to be true in master[1].

Where is it defined for upstream ci jobs ?

[1] https://github.com/openstack/tripleo-quickstart/blob/master/roles/repo-setup/defaults/main.yml#L37..L37

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/755310

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/755312

[Sofer Athlan-Guyot (sofer-athlan-guyot)]

So currently we have 3 patches for this :)

1. https://review.opendev.org/#/c/755312/ : fix the multinode upgrade to prevent dnf upgrade from running

2. https://review.opendev.org/#/c/755252/ : add support for rdo layered package

3. https://review.opendev.org/#/c/755310/ : add support for the latest openvswitch pkg hack to ... stop openvswitch during a upgrade of the package (!?)

So now, we need to test that, a clever depends on 755310 that runs a multinode upgrade with 755312 included as well should do the trick. I look into it, but ping me if you know how this is done.

[Sofer Athlan-Guyot (sofer-athlan-guyot)]

Running a test there https://review.opendev.org/#/c/755336/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/755310
Committed: https://git.openstack.org/cgit/openstack/tripleo-ansible/commit/?id=9d712acb187cb36c45776a69da736e7f685648fb
Submitter: Zuul
Branch: master

commit 9d712acb187cb36c45776a69da736e7f685648fb
Author: Sofer Athlan-Guyot <email address hidden>
Date: Wed Sep 30 15:57:54 2020 +0200

    Ensure we catch new openvswitch disruption during upgrade/update.

    Openvswtich may require special treatment during upgrade in order to
    prevent disruption of the service provided.

    In recents ovs package the disruption was changed:

        preuninstall scriptlet (using /bin/sh):

        if [ $1 -eq 0 ] ; then
                # Package removal, not upgrade
                systemctl --no-reload disable --now openvswitch.service &>/dev/null || :
        fi
        postuninstall program: /bin/sh

        postuninstall is empty.

    We add this is new profile to the conditional that warrants a special
    treatment.

    Change-Id: Idc007c16da7b9d15bfbc35fcaed341e1c7aef5f8
    Closes-Bug: #1897863

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/755755

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/755756

[yatin (yatinkarel)]

Shoudn't have closed the bug, it's still failing, and it never passed since the reported bug, re opening:- https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-standalone-upgrade

[yatin (yatinkarel)]

<< Also additionally i checked why issue didn't hit in undercloud-upgrade job[1].
<< [1] https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-undercloud-upgrade

On checking further, found that issue existed in undercloud-upgrade too but didn't got detected as it do not run tempest, confirmed by enabling tempest in undercloud-upgrade with https://review.rdoproject.org/r/#/c/29867/.

<< /me don't know why openvswitch is getting started with tripleo_network_config_with_ansible: true and hide the issue in undercloud-upgrade job.
tripleo_network_config started openvswitch during undercloud upgrade as there was change in network-config, example:- https://cef4e2ae1ec05c2861e9-3ba74d52e8ef64a78bfc3a9422d30897.ssl.cf2.rackcdn.com/755994/1/check/tripleo-ci-centos-8-undercloud-upgrade/c2cc65e/logs/undercloud/etc/os-net-config/

Now with respect to the issue:-
<< 1. https://review.opendev.org/#/c/755312/ : fix the multinode upgrade to prevent dnf upgrade from running
With this upgrade for tripleo packages also skipped which is needed before undercloud-upgrade runs, https://review.opendev.org/#/c/755238/ should fix that.

But with both above patches it's doing right thing for upgrades but issue still persist as openvswitch is not started after upgrade.

I did some hacky tests to get the job passing, test results in:- https://review.rdoproject.org/r/#/c/29881/
What i did was:-
1) Include https://review.opendev.org/#/c/755312/ and https://review.opendev.org/#/c/755238/
2) start ovs after upgrade https://review.opendev.org/#/c/755515/ , which would be not needed if tripleo_ovs_upgrade handles the upgrade case
Do not rely on br-ex at all patchset 2:- https://review.opendev.org/#/c/755607/2 + cleaning members: in net-config[1][2] which can be made optional for standalone when external interface not available, that patch is abandoned in favor of some other patches to correctly configuring br-ex. br-ex shouldn't be needed as it do not get configured on br-ctlplane:- ovs-vsctl[48758]: ovs|00002|db_ctl_base|ERR|cannot create a port named br-ex because a bridge named br-ex already exists

[1] https://github.com/openstack/tripleo-heat-templates/blob/master/net-config-standalone.j2.yaml#L150-L155
[2] https://github.com/openstack/tripleo-ansible/blob/master/tripleo_ansible/roles/tripleo_network_config/templates/standalone.j2#L28-L32

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/755312
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=801333ad0f48dd9f2f22284ebbde5fcc14556648
Submitter: Zuul
Branch: master

commit 801333ad0f48dd9f2f22284ebbde5fcc14556648
Author: Sofer Athlan-Guyot <email address hidden>
Date: Wed Sep 30 16:51:35 2020 +0200

    Prevent multinode upgrade from running dnf upgrade.

    The repo-setup role will, by default run a dnf upgrade when it's
    done. This behavior should be deactivated during upgrade job.

    This is in-line with all the other upgrade jobs configuration.

    Change-Id: I5808918b72514adeb8b7d47d5eaed3d4a6883c62
    Related-Bug: #1897863

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/755238
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=89773862006717da2d0338699d88cf884c87e088
Submitter: Zuul
Branch: master

commit 89773862006717da2d0338699d88cf884c87e088
Author: yatinkarel <email address hidden>
Date: Wed Sep 30 14:47:46 2020 +0530

    Use ansible_pkg_mgr instead of hardcoded yum

    Also fix following:-
    - Fix usage of python-tripleoclient, package
      name is python3-tripleoclient for EL8 distros,
      using python*-tripleoclient will work for both
      EL7 and EL8.
    - Move package update tasks to container-prep
      as that's run before standalone upgrade and package
      needs to be updated before container prep.
    - Add tripleo-ansible to package update list as that's
      need to be latest like other tripleo packages.

    Related-Bug: #1897863
    Depends-On: https://review.opendev.org/#/c/755312/
    Change-Id: I20c3814e367a6d062961b7d81615b22a35a8e67b

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/760294

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/victoria
Review: https://review.opendev.org/761404

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/755252
Committed: https://git.openstack.org/cgit/openstack/tripleo-ansible/commit/?id=31ed062a9c1cc2bbccc73ece1ed7307e58b390d1
Submitter: Zuul
Branch: master

commit 31ed062a9c1cc2bbccc73ece1ed7307e58b390d1
Author: Sofer Athlan-Guyot <email address hidden>
Date: Wed Sep 30 11:34:04 2020 +0200

    Add support for rdo openvswitch layered upgrade special treatment.

    The special treatment for openvswitch works on Red Hat. This patch add
    RDO support.

    We also adjust for the all naming convention of the openvswitch
    package.

    This was discovered during the related-bug investigation. Note that it
    in itself doesn't solve it.

    Co-Authored-By: yatinkarel <email address hidden>
    Depends-On: https://review.opendev.org/#/c/761463/
    Change-Id: I1986ef8109061dab8213f8ec29160b6d830cdc49
    Related-Bug: #1897863

[OpenStack Infra (hudson-openstack)]

Change abandoned by yatin (<email address hidden>) on branch: master
Review: https://review.opendev.org/760294
Reason: In favor of https://review.opendev.org/755252

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/761856

[Sandeep Yadav (sandeepyadav93)]

Hello,

I want to highlight that victoria upgrade job is also failing because of the same issue:-

https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-standalone-upgrade-victoria

periodic-tripleo-ci-centos-8-standalone-upgrade-victoria

https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-upgrade-victoria/7bf329f/logs/undercloud/var/log/extra/podman/containers/container-puppet-ovn_controller/stdout.log.txt.gz

~~~
<13>Nov 9 23:29:58 puppet-user: Error: Could not prefetch vs_config provider 'ovs': Execution of '/usr/bin/ovs-vsctl list Open_vSwitch .' returned 1: ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)
<13>Nov 9 23:29:58 puppet-user: Error: Failed to apply catalog: Execution of '/usr/bin/ovs-vsctl list Open_vSwitch .' returned 1: ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)
~~~

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/761404
Committed: https://git.openstack.org/cgit/openstack/tripleo-ansible/commit/?id=4c6fc527e550def09464e04e966fe7e6cf05afd4
Submitter: Zuul
Branch: stable/victoria

commit 4c6fc527e550def09464e04e966fe7e6cf05afd4
Author: Sofer Athlan-Guyot <email address hidden>
Date: Wed Sep 30 11:34:04 2020 +0200

    Add support for rdo openvswitch layered upgrade special treatment.

    The special treatment for openvswitch works on Red Hat. This patch add
    RDO support.

    We also adjust for the all naming convention of the openvswitch
    package.

    This was discovered during the related-bug investigation. Note that it
    in itself doesn't solve it.

    Co-Authored-By: yatinkarel <email address hidden>
    Depends-On: https://review.opendev.org/#/c/761463/
    Change-Id: I1986ef8109061dab8213f8ec29160b6d830cdc49
    Related-Bug: #1897863
    (cherry picked from commit 31ed062a9c1cc2bbccc73ece1ed7307e58b390d1)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/761856
Committed: https://git.openstack.org/cgit/openstack/tripleo-ansible/commit/?id=c4a181b8558226d41c5d2fa7ceacf7cada16b34c
Submitter: Zuul
Branch: stable/ussuri

commit c4a181b8558226d41c5d2fa7ceacf7cada16b34c
Author: Sofer Athlan-Guyot <email address hidden>
Date: Wed Sep 30 11:34:04 2020 +0200

    Add support for rdo openvswitch layered upgrade special treatment.

    The special treatment for openvswitch works on Red Hat. This patch add
    RDO support.

    We also adjust for the all naming convention of the openvswitch
    package.

    This was discovered during the related-bug investigation. Note that it
    in itself doesn't solve it.

    Conflicts:
        tripleo_ansible/ansible_plugins/modules/tripleo_ovs_upgrade.py

    Co-Authored-By: yatinkarel <email address hidden>
    Depends-On: https://review.opendev.org/#/c/761463/
    Change-Id: I1986ef8109061dab8213f8ec29160b6d830cdc49
    Related-Bug: #1897863
    (cherry picked from commit 31ed062a9c1cc2bbccc73ece1ed7307e58b390d1)

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Jesse Pretorius <email address hidden>" on branch: stable/train
Review: https://review.opendev.org/c/openstack/tripleo-ansible/+/755756
Reason: There appears to be no motivation to have this, so abandoning it. It can be restored if need be.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Jesse Pretorius <email address hidden>" on branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/tripleo-ansible/+/755755
Reason: There appears to be no motivation to have this, so abandoning it. It can be restored if need be.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "James Slagle <email address hidden>" on branch: stable/train
Review: https://review.opendev.org/c/openstack/tripleo-ansible/+/775106
Reason: Abandoning this patch per the TripleO Patch Abandonment guidelines
(https://specs.openstack.org/openstack/tripleo-specs/specs/policy/patch-abandonment.html).
If you wish to have this restored and cannot do so yourself, please reach out
via #tripleo on OFTC or the OpenStack Dev mailing list.