[promotion][master][ovn] jobs failing with ovsdb-server: I/O error: /var/lib/openvswitch/ovnnb.db: open failed (No such file or directory)

[yatin]

Multiple OVN jobs are failing post kolla patch https://review.opendev.org/#/c/691507 with Error:-
ovsdb-server: I/O error: /var/lib/openvswitch/ovnnb.db: open failed (No such file or directory)
ovsdb-server: I/O error: /var/lib/openvswitch/ovnsb.db: open failed (No such file or directory)

Logs:-
https://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-standalone-master/297d760/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz
https://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-standalone-master/297d760/logs/undercloud/var/log/extra/podman/containers/ovn_north_db_server/stdout.log.txt.gz
https://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset010-master/09ec485/logs/subnode-1/var/log/extra/podman/containers/ovn_south_db_server/stdout.log.txt.gz

Other failures can be seen at https://trunk-primary.rdoproject.org/api-centos-master-uc/api/civotes_detail.html?commit_hash=ea38a58fa88acb2e5312767d77d1c86fb7366686&distro_hash=085fbf9b2c565918055f1ded5ee1ccdc58334923.
rhel jobs are also affected:- https://trunk.rdoproject.org/api-redhat-master/api/civotes_detail.html?commit_hash=ea38a58fa88acb2e5312767d77d1c86fb7366686&distro_hash=085fbf9b2c565918055f1ded5ee1ccdc58334923

The kolla patch no longer creates db which are still referenced in tht for ex:- https://github.com/openstack/tripleo-heat-templates/blob/master/deployment/ovn/ovn-dbs-container-puppet.yaml#L93

Can someone from ovn take a look?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/694543

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/694543
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=e8ecbecb6a36c1222a5ff4459ceaa6e7e3b9a680
Submitter: Zuul
Branch: master

commit e8ecbecb6a36c1222a5ff4459ceaa6e7e3b9a680
Author: Numan Siddique <email address hidden>
Date: Fri Nov 15 19:40:44 2019 +0530

    ovn-db standalone: Create ovndbs before starting the OVN ovsdb servers

    The kolla commit [1] deleted the extend_start script for ovn-nb-db-server/ovn-sb-db-server
    kolla images. This script created the ovn dbs from the schema. After this commit
    tripleo CI is broken as the ovn dbs are not present.

    To fix this issue, this patch first creates OVN dbs before starting the OVN ovsdb-servers.

    [1] - I31a372d801dace92950cd85b29b308431f78bbdb
    Closes-bug: #1852682
    Change-Id: I9640529455e1d14c98c3b53993a71d77de8d7a25
    Signed-off-by: Numan Siddique <email address hidden>

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 12.0.0 release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/697285

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/697285
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=0f1ffe9a10ea27b404a638fc11f12ad8cb42609d
Submitter: Zuul
Branch: stable/train

commit 0f1ffe9a10ea27b404a638fc11f12ad8cb42609d
Author: Numan Siddique <email address hidden>
Date: Fri Nov 15 19:40:44 2019 +0530

    ovn-db standalone: Create ovndbs before starting the OVN ovsdb servers

    The kolla commit [1] deleted the extend_start script for ovn-nb-db-server/ovn-sb-db-server
    kolla images. This script created the ovn dbs from the schema. After this commit
    tripleo CI is broken as the ovn dbs are not present.

    To fix this issue, this patch first creates OVN dbs before starting the OVN ovsdb-servers.

    [1] - I31a372d801dace92950cd85b29b308431f78bbdb
    Closes-bug: #1852682
    Closes-bug: #1855055

    (cherry-picked from master commit e8ecbecb6a36c1222a5ff4459ceaa6e7e3b9a680)
    Change-Id: I9640529455e1d14c98c3b53993a71d77de8d7a25
    Signed-off-by: Numan Siddique <email address hidden>

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 11.3.1 release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/704553

[OpenStack Infra (hudson-openstack)]

Change abandoned by Denis Karpov (<email address hidden>) on branch: stable/stein
Review: https://review.opendev.org/704553
Reason: There is little chance for this to be accepted and even less for OVN 2.12 RPM package to be accepted in RDO Stein, so abandoning. Found a different solution that applies in my deployment an fixes the OVN scalability/performance issue - cbasically if one builds OVN packages from the branch 'branch-2.12', it's runtime director structure is not changed there, so it is a matter of only patching containers ovsdb and ovn-controller containers installing these patches during deployment/update, no source level patches are needed.