Error: 'ovs-vsctl --timeout=5 remove Bridge br-ctlplane other-config:mac-table-size' returned 1 instead of one of [0]

[chandan kumar]

39554: DNM standalone upgrade cs9 master testing | https://review.rdoproject.org/r/c/testproject/+/39554 is adding and testing standalone upgrade job to CentOS 9 integration line.

Currently the job is failing with following error during upgrade:
from standalone upgrade log [1]
---------------------
2022-02-22 20:26:26 | + rc=4
2022-02-22 20:26:26 | + '[' false = false ']'
2022-02-22 20:26:26 | + set +x
2022-02-22 20:26:26 | 2022-02-22 20:26:26.576688 | fa163e28-e89d-e058-b6f6-000000001579 | FATAL | Manage Puppet containers (generate config) for step 1 | standalone | error={"changed": false, "msg": "Failed containers: container-puppet-ovn_controller"}
2022-02-22 20:26:26 | 2022-02-22 20:26:26.577509 | fa163e28-e89d-e058-b6f6-000000001579 | TIMING | Manage Puppet containers (generate config) for step 1 | standalone | 0:07:05.228381 | 73.74s
---------------------

By taking a look at container-puppet-ovn_controller stdout logs [2]
-----------------
Notice: Compiled catalog for standalone.localdomain in environment production in 0.48 seconds
<13>Feb 22 20:26:04 puppet-user: Notice: /Stage[main]/Ovn::Controller/Exec[br-ctlplane]/returns: ovs-vsctl: 'remove' command requires at least 4 arguments
<13>Feb 22 20:26:04 puppet-user: Error: 'ovs-vsctl --timeout=5 remove Bridge br-ctlplane other-config:mac-table-size' returned 1 instead of one of [0]
<13>Feb 22 20:26:04 puppet-user: Error: /Stage[main]/Ovn::Controller/Exec[br-ctlplane]/returns: change from 'notrun' to ['0'] failed: 'ovs-vsctl --timeout=5 remove Bridge br-ctlplane other-config:mac-table-size' returned 1 instead of one of [0]
<13>Feb 22 20:26:04 puppet-user: Notice: Applied catalog in 0.48 seconds
-----------------

I think the issue seems to be here: https://opendev.org/openstack/puppet-ovn/src/branch/master/manifests/controller.pp#L230

---------------------
 # TODO(numans): Right now puppet-vswitch's vs_bridge doesn't support
        # setting the column 'other-config' for the Bridge table.
        # Switch to using vs_bridge once the support is available.
        if $mac_table_size != undef {
          exec { $br:
            command => "ovs-vsctl --timeout=5 set Bridge ${br} other-config:mac-table-size=${mac_table_size}",
            unless => "ovs-vsctl get bridge ${br} other-config:mac-table-size | grep -q -w ${mac_table_size}",
            path => '/usr/sbin:/usr/bin:/sbin:/bin',
            onlyif => "ovs-vsctl br-exists ${br}",
            require => [ Service['openvswitch'], Vs_bridge[$br] ],
          }
        } else {
          exec { $br:
            command => "ovs-vsctl --timeout=5 remove Bridge ${br} other-config:mac-table-size",
            path => '/usr/sbin:/usr/bin:/sbin:/bin',
            onlyif => ["ovs-vsctl br-exists ${br}", "ovs-vsctl get bridge ${br} other-config:mac-table-size"],
            require => [ Service['openvswitch'], Vs_bridge[$br] ],
---------------------

Logs:

[1]. https://logserver.rdoproject.org/54/39554/1/check/periodic-tripleo-ci-centos-9-standalone-upgrade-master/a7f4351/logs/undercloud/home/zuul/standalone_upgrade.log.txt.gz

[2]. https://logserver.rdoproject.org/54/39554/1/check/periodic-tripleo-ci-centos-9-standalone-upgrade-master/a7f4351/logs/undercloud/var/log/extra/podman/containers/container-puppet-ovn_controller/stdout.log.txt.gz

[yatin (yatinkarel)]

Pushed fix https://review.opendev.org/c/openstack/puppet-ovn/+/830552

[Takashi Kajinami (kajinamit)]

This is not a bug in TripleO itself but the one in puppet-ovn. I've added puppet-ovn as that is the appropriate component.

I'd leave tripleo now as I don't have full understanding about the way CI folks maintain problems affecting TripleO but I tend to unassociate this from TripleO.

[yatin (yatinkarel)]

Job moved ahead with the fix and no longer failing for ovn_controller https://logserver.rdoproject.org/54/39554/2/check/periodic-tripleo-ci-centos-9-standalone-upgrade-master/049666a/logs/undercloud/var/log/extra/podman/containers/container-puppet-ovn_controller/stdout.log.txt.gz

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/puppet-ovn/+/830552
Committed: https://opendev.org/openstack/puppet-ovn/commit/3c80ff104029ab2464275a131050addd3ed4eeb7
Submitter: "Zuul (22348)"
Branch: master

commit 3c80ff104029ab2464275a131050addd3ed4eeb7
Author: yatinkarel <email address hidden>
Date: Wed Feb 23 12:07:31 2022 +0530

    Fix Syntax for ovs-vsctl remove command

    It was introduced with [1] and causing issues during upgrade as
    the ovs-vsctl remove command expects atleast 4 arguments, but only 3
    were passed, this patch fixes it.

    [1] https://review.opendev.org/c/openstack/puppet-ovn/+/825924

    Closes-Bug: #1961873
    Change-Id: I84627ab07ccdb5b1e3597bda5c7fa99ad0675934

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-ovn 20.2.0 release.