Standalone upgrade job is failing - On ovn upgrade

[mathieu bultel]

Standalone Upgrade jobs is failing since today due to ovn upgrade:

2019-11-18 06:36:12 | 2019-11-18 06:36:12.898 85614 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] "2019-11-18T06:33:22Z|00001|lockfile|WARN|/var/lib/openvswitch/.ovnnb.db.~lock~: failed to lock file: Resource temporarily unavailable",
2019-11-18 06:36:12 | 2019-11-18 06:36:12.898 85614 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] "ovsdb-tool: I/O error: /var/lib/openvswitch/ovnnb.db: failed to lock lockfile (Resource temporarily unavailable)",

[wes hayutin (weshayutin)]

 periodic-tripleo-ci-centos-7-standalone-upgrade-train failing

           * 2020-01-03 11:03:47.805 ERROR /var/log/paunch.log: 111042 ERROR paunch [ ] stderr: net_mlx5: cannot load glue library: libibverbs.so.1: cannot open shared object file: No such file or directory
             * http://logs.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-standalone-upgrade-train/ab59596/logs/undercloud/var/log/extra/errors.txt.txt.gz
             * causes networking to fail and connection to mariadb to fail

[Marios Andreou (marios-b)]

so this bug is pretty old .. but the job itself is in pretty bad shape https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-standalone-upgrade-train

BUt it isn't in the promotion criteria so it isn't promotion blocker *per se* but it should be? (and it should be in criteria )

13:01 < matbu> marios: maybe i'm missing something but this one is a promotion blocker https://bugs.launchpad.net/tripleo/+bug/1853012 ?
13:01 <@openstack> Launchpad bug 1853012 in tripleo "Standalone upgrade job is failing - On ovn upgrade" [Critical,In progress] -
                   Assigned to mathieu bultel (mat-bultel)
13:01 < matbu> marios: i'm catching up thing
13:01 < matbu> CI seems all green (update & upgrade i mean)
13:07 < marios> matbu: o/ looking
13:07 < marios> matbu: happy new year ;) \o/
13:08 < marios> matbu: not sure... the bug is pretty old but there is a recent comment by weshay there fromfriday
13:08 < marios> matbu: and the job is in pretty bad shape
                https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-standalone-upgrade-train
13:10 < marios> matbu: but looks like the job isn't in promotion criteria
                https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/config/CentOS-7/train.ini

[mathieu bultel (mat-bultel)]

This review should fix the issue:
https://review.opendev.org/#/c/697237/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/701920

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/701920
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=7f9b6c40f329fb69606c03b6a99c19d8762c8f46
Submitter: Zuul
Branch: stable/train

commit 7f9b6c40f329fb69606c03b6a99c19d8762c8f46
Author: Mathieu Bultel <email address hidden>
Date: Fri Jan 10 10:45:50 2020 +0100

    Ovn upgrade - test if db already exist

    Test if db already exist before doing the ovstool create.
    If not it will failed during upgrade

    Closes-Bug: #1853012
    Change-Id: I6d97a9dcf5609003663920e8762e07ceea2e7933

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/701970
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=eb245497ef5c22c79fa7ad742796beca8fb0a4f1
Submitter: Zuul
Branch: stable/train

commit eb245497ef5c22c79fa7ad742796beca8fb0a4f1
Author: Mathieu Bultel <email address hidden>
Date: Fri Jan 10 15:06:44 2020 +0100

    Remove docker_config step 3 for ovn already cover by kolla script

    The kolla script should take care of the ovn database creation,
    we dont need anymore the docker_config step3

    Partial backport from:
    I1fbfaf43af17b558497fd2b46fc4278b4703ec74

    Closes-Bug: #1853012
    Change-Id: I505368feafc42d308ba5d52c894abb45e3ea878f

[Alex Schultz (alex-schultz)]

This bug has had a related patch abandoned and has been automatically un-assigned due to inactivity. Please re-assign yourself if you are continuing work or adjust the state as appropriate if it is no longer valid.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Alex Schultz (<email address hidden>) on branch: master
Review: https://review.opendev.org/695098
Reason: This review is > 90 days without comment, and failed Zuul the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results. For more details check policy https://specs.openstack.org/openstack/tripleo-specs/specs/policy/patch-abandonment.html

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 11.4.0 release.