module level init of db transation contexts cause failure under mod_wsgi on module reload

[sean mooney]

Description
===========
This is related to a downstream bug first reported here:https://bugzilla.redhat.com/show_bug.cgi?id=1630069 where in it was discovers that due to how
triplo currently deploys the placemetn api under mod_wsgi

when deployed under mod_wsgi if the wsgi application exits with an error it is reloaded
back into the same python interperter instance. As a result of this behavior module level
variable have a longer lifetime then the lifetime of the application. when run under uwsgi
when the application is reloaded it is reloaded in a new python interperter meaning
the lifetime of the module level variables is scoped to the life time of the application.

As a result of the life time semantics of mod_wsgi the placment api and nova-api
must assume that the wsgi applications init can be invoked multiple times on failure.
The current use of the sqlalcamey enginefacade transaction_context is not reentrant
resulting in a type error being raised on subsequet calls to configure when the wsgi
application is reloaded.

Expected result
===============
it should be possible to reload the nova and placement api wsgi application
under mod_wsgi on failure

Actual result
=============

46087 [Wed Oct 10 15:10:49.433284 2018] [:error] [pid 14] [remote 172.25.0.10:208] mod_wsgi (pid=14): Target WSGI script '/var/www/cgi-bin/nova/nova-placement-api' cannot be loaded as Python module.
46088 [Wed Oct 10 15:10:49.433305 2018] [:error] [pid 14] [remote 172.25.0.10:208] mod_wsgi (pid=14): Exception occurred processing WSGI script '/var/www/cgi-bin/nova/nova-placement-api'.
46089 [Wed Oct 10 15:10:49.433320 2018] [:error] [pid 14] [remote 172.25.0.10:208] Traceback (most recent call last):
46090 [Wed Oct 10 15:10:49.433333 2018] [:error] [pid 14] [remote 172.25.0.10:208] File "/var/www/cgi-bin/nova/nova-placement-api", line 54, in <module>
46091 [Wed Oct 10 15:10:49.433354 2018] [:error] [pid 14] [remote 172.25.0.10:208] application = init_application()
46092 [Wed Oct 10 15:10:49.433361 2018] [:error] [pid 14] [remote 172.25.0.10:208] File "/usr/lib/python2.7/site-packages/nova/api/openstack/placement/wsgi.py", line 108, in init_application
46093 [Wed Oct 10 15:10:49.433386 2018] [:error] [pid 14] [remote 172.25.0.10:208] db_api.configure(conf.CONF)
46094 [Wed Oct 10 15:10:49.433392 2018] [:error] [pid 14] [remote 172.25.0.10:208] File "/usr/lib/python2.7/site-packages/nova/api/openstack/placement/db_api.py", line 35, in configure
46095 [Wed Oct 10 15:10:49.433403 2018] [:error] [pid 14] [remote 172.25.0.10:208] **_get_db_conf(conf.placement_database))
46096 [Wed Oct 10 15:10:49.433408 2018] [:error] [pid 14] [remote 172.25.0.10:208] File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 788, in configure
46097 [Wed Oct 10 15:10:49.433420 2018] [:error] [pid 14] [remote 172.25.0.10:208] self._factory.configure(**kw)
46098 [Wed Oct 10 15:10:49.433425 2018] [:error] [pid 14] [remote 172.25.0.10:208] File "/usr/lib/python2.7/site-packages/debtcollector/renames.py", line 43, in decorator
46099 [Wed Oct 10 15:10:49.433435 2018] [:error] [pid 14] [remote 172.25.0.10:208] return wrapped(*args, **kwargs)
46100 [Wed Oct 10 15:10:49.433440 2018] [:error] [pid 14] [remote 172.25.0.10:208] File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 312, in configure
46101 [Wed Oct 10 15:10:49.433449 2018] [:error] [pid 14] [remote 172.25.0.10:208] self._configure(False, kw)
46102 [Wed Oct 10 15:10:49.433453 2018] [:error] [pid 14] [remote 172.25.0.10:208] File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 317, in _configure
46103 [Wed Oct 10 15:10:49.433462 2018] [:error] [pid 14] [remote 172.25.0.10:208] raise TypeError("this TransactionFactory is already started")
46104 [Wed Oct 10 15:10:49.433473 2018] [:error] [pid 14] [remote 172.25.0.10:208] TypeError: this TransactionFactory is already started

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/610034
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=00d08a3288f61268f8a823991fadf0d83c36c04d
Submitter: Zuul
Branch: master

commit 00d08a3288f61268f8a823991fadf0d83c36c04d
Author: Sean Mooney <email address hidden>
Date: Fri Oct 12 14:26:14 2018 +0100

    Harden placement init under wsgi

    - This change tries to address an edge case discovered
      when running placement under mod_wsgi where
      if the placement wsgi application is re-initialize the db_api
      configure method attempts to reconfigure a started transaction
      factory.

    - This since oslo.db transaction factories do not support
      reconfiguration at runtime this result in an exception being
      raised preventing reloading of the Placement API without
      restarting apache to force mod_wsgi to recreate the
      python interpreter.

    - This change introduces a run once decorator to allow annotating
      functions that should only be executed once for the lifetime fo
      an interpreter.

    - This change applies the run_once decorator to the db_api configure
      method, to suppress the attempt to reconfigure the current
      TransactionFactory on application reload.

    Co-Authored-By: Balazs Gibizer <email address hidden>
    Closes-Bug: #1799246
    Related-Bug: #1784155
    Change-Id: I704196711d30c1124e713ac31111a8ea6fa2f1ba

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/617297

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/617297
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=74ed2fe69f15d86252b1c7779c916cb93ffc252d
Submitter: Zuul
Branch: stable/rocky

commit 74ed2fe69f15d86252b1c7779c916cb93ffc252d
Author: Sean Mooney <email address hidden>
Date: Fri Oct 12 14:26:14 2018 +0100

    Harden placement init under wsgi

    - This change tries to address an edge case discovered
      when running placement under mod_wsgi where
      if the placement wsgi application is re-initialize the db_api
      configure method attempts to reconfigure a started transaction
      factory.

    - This since oslo.db transaction factories do not support
      reconfiguration at runtime this result in an exception being
      raised preventing reloading of the Placement API without
      restarting apache to force mod_wsgi to recreate the
      python interpreter.

    - This change introduces a run once decorator to allow annotating
      functions that should only be executed once for the lifetime fo
      an interpreter.

    - This change applies the run_once decorator to the db_api configure
      method, to suppress the attempt to reconfigure the current
      TransactionFactory on application reload.

    NOTE(lyarwood): Simple positional conflict due to
    Ie7bf5d012e2ccbcd63c262ddaf739782afcdaf56 and
    I81d13418d75b46fbdb9f6d44889a207528c8d6de not being present in
    stable/rocky.

    Conflicts:
            nova/utils.py

    Co-Authored-By: Balazs Gibizer <email address hidden>
    Closes-Bug: #1799246
    Related-Bug: #1784155
    Change-Id: I704196711d30c1124e713ac31111a8ea6fa2f1ba
    (cherry picked from commit 00d08a3288f61268f8a823991fadf0d83c36c04d)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 18.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 19.0.0.0rc1 release candidate.