503 error from octavia after overcloud deployment

[Rabi Mishra]

1. Standard overcloud deployment with octavia (with default ovn) using -e $THT/environments/services/octavia.yaml

2. openstack loadbalancer list fails with below error.

REQ: curl -g -i -X GET http://192.168.24.11:9876/v2.0/lbaas/loadbalancers -H "User-Agent: openstacksdk/0.26.0 keystoneauth1/3.13.1 python-requests/2.21.0 CPython/2.7.5" -H "X-Auth-Token: {SHA256}84d969d0d4600fc558a49e3c27536689bacab81979ab0df4af5c0c3fd10fca69"
Starting new HTTP connection (1): 192.168.24.11:9876
http://192.168.24.11:9876 "GET /v2.0/lbaas/loadbalancers HTTP/1.1" 503 None
RESP: [503] Cache-Control: no-cache Connection: close Content-Type: text/html
RESP BODY: Omitted, Content-Type is set to text/html. Only application/json responses have their bodies logged.
Request returned failure status: 503
Service Unavailable (HTTP 503)

3. octavia api does not start and loops all the time failing to load ovn provider driver

2019-04-17 09:36:30.299 21 DEBUG octavia.api.app [-] {'"ovn"': '"Octavia OVN driver"}', '{"amphora"': '"Octavia Amphora Driver"'} _init_drivers /usr/lib/python2.7/site-packages/octavia/api/app.py:42
2019-04-17 09:36:30.299 21 DEBUG octavia.api.app [-] "ovn" _init_drivers /usr/lib/python2.7/site-packages/octavia/api/app.py:43
2019-04-17 09:36:30.299 21 WARNING stevedore.named [-] Could not load "ovn"
2019-04-17 09:36:30.300 21 ERROR octavia.api.drivers.driver_factory [-] Unable to load provider driver "ovn" due to: No 'octavia.api.drivers' driver found, looking for '"ovn"': NoMatches: No 'octavia.api.drivers' driver found, looking for '"ovn"'

We seem to have added provider_driver based on neutron_driver in https://github.com/openstack/puppet-tripleo/commit/97c46ca76803fb238279d24544e8aba9c5685632

However, there are 2 issues with that patch

1. The octavia_api image does not have networking-ovn to have the provider driver for ovn.
2. The way we specify provider_drivers (oslo.config DictOpt) seems incorrect

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/653375

[Rabi Mishra (rabi)]

OK https://review.openstack.org/#/c/644587/ adds the networking-ovn package to the octavia_api image.

But I also had to add below to octavia.conf to make it work as the ovn provider driver connects to nb database.

[ovn]
ovn_nb_connection=tcp:x.x.x.x:6641

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/653672

[Rabi Mishra (rabi)]

Just for reference:

Without the above ovn config, octavia api fails to strat with below error.

2019-04-18 07:25:32.806 20 DEBUG octavia.api.app [-] ovn _init_drivers /usr/lib/python2.7/site-packages/octavia/api/app.py:43
2019-04-18 07:25:32.807 20 ERROR ovsdbapp.backend.ovs_idl.idlutils [-] Unable to open stream to tcp:127.0.0.1:6641 to retrieve schema: Connection refused
2019-04-18 07:25:32.807 20 ERROR octavia.api.drivers.driver_factory [-] Unable to load provider driver ovn due to: Could not retrieve schema from tcp:127.0.0.1:6641: Exception: Could not retrieve schema from tcp:127.0.0.1:6641

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/653375
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=2a1167dd7b99f5ba5040b2cd90f897a26e590e77
Submitter: Zuul
Branch: master

commit 2a1167dd7b99f5ba5040b2cd90f897a26e590e77
Author: Rabi Mishra <email address hidden>
Date: Wed Apr 17 15:18:18 2019 +0530

    Set octavia provider_drivers config option correctly

    Change-Id: I4cb95de0cd963899d2db7e08704ed734e87751d7
    Partial-Bug: #1825146

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/654808

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/653672
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=d88db3ad716561e5526afc1a0cdb4de08e599741
Submitter: Zuul
Branch: master

commit d88db3ad716561e5526afc1a0cdb4de08e599741
Author: Rabi Mishra <email address hidden>
Date: Thu Apr 18 09:25:47 2019 +0530

    Add support to set ovs_nb_connection

    When using ovn provider unless set, it would try to
    connect to 127.0.0.1.

    Change-Id: If8eb685dd21791b65845afd44fb483592b8bcc0c
    Depends-On: https://review.openstack.org/653673
    Closes-Bug: #1825146

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/654808
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=a2e3f3445cfbcbc995451abc165695fa8a7aaa68
Submitter: Zuul
Branch: stable/stein

commit a2e3f3445cfbcbc995451abc165695fa8a7aaa68
Author: Rabi Mishra <email address hidden>
Date: Wed Apr 17 15:18:18 2019 +0530

    Set octavia provider_drivers config option correctly

    Change-Id: I4cb95de0cd963899d2db7e08704ed734e87751d7
    Depends-On: https://review.opendev.org/655105
    Partial-Bug: #1825146
    (cherry picked from commit 2a1167dd7b99f5ba5040b2cd90f897a26e590e77)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/655813

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/655813
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=ad0e5975fb8a8fae50eb07b1a2b692f79d10ba8c
Submitter: Zuul
Branch: stable/stein

commit ad0e5975fb8a8fae50eb07b1a2b692f79d10ba8c
Author: Rabi Mishra <email address hidden>
Date: Thu Apr 18 09:25:47 2019 +0530

    Add support to set ovs_nb_connection

    When using ovn provider unless set, it would try to
    connect to 127.0.0.1.

    Change-Id: If8eb685dd21791b65845afd44fb483592b8bcc0c
    Depends-On: https://review.openstack.org/653673
    Closes-Bug: #1825146
    (cherry picked from commit d88db3ad716561e5526afc1a0cdb4de08e599741)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-tripleo 11.0.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-tripleo 10.5.0 release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/753399

[OpenStack Infra (hudson-openstack)]

Change abandoned by Brent Eagles (<email address hidden>) on branch: stable/rocky
Review: https://review.opendev.org/753399