Unresponsive services during failover

[Pierre Riteau]

When the internal VIP is moved in the event of a failure of the active controller, OpenStack services can become unresponsive for a while. Logs show errors like this one:

oslo_db.sqlalchemy.engines oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query'
)

This appears to be caused by services trying to use DB connections in the SQLAlchemy pool which have become invalid due to the move of the VIP.

[Mark Goddard (mgoddard)]

https://review.opendev.org/#/c/749632/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/749632
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=c81772024c70b84564cfddb29645390ae378498f
Submitter: Zuul
Branch: master

commit c81772024c70b84564cfddb29645390ae378498f
Author: Pierre Riteau <email address hidden>
Date: Tue Sep 22 17:52:36 2020 +0200

    Reduce the use of SQLAlchemy connection pooling

    When the internal VIP is moved in the event of a failure of the active
    controller, OpenStack services can become unresponsive as they try to
    talk with MariaDB using connections from the SQLAlchemy pool.

    It has been argued that OpenStack doesn't really need to use connection
    pooling with MariaDB [1]. This commit reduces the use of connection
    pooling via two configuration options:

    - max_pool_size is set to 1 to allow only a single connection in the
      pool (it is not possible to disable connection pooling entirely via
      oslo.db, and max_pool_size = 0 means unlimited pool size)
    - lower connection_recycle_time from the default of one hour to 10
      seconds, which means the single connection in the pool will be
      recreated regularly

    These settings have shown better reactivity of the system in the event
    of a failover.

    [1] http://lists.openstack.org/pipermail/openstack-dev/2015-April/061808.html

    Change-Id: Ib6a62d4428db9b95569314084090472870417f3d
    Closes-Bug: #1896635

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/754928

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/754929

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/754931

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/754929
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=903efc5a9880eee960f131f7954ec568e6f60df1
Submitter: Zuul
Branch: stable/train

commit 903efc5a9880eee960f131f7954ec568e6f60df1
Author: Pierre Riteau <email address hidden>
Date: Tue Sep 22 17:52:36 2020 +0200

    Reduce the use of SQLAlchemy connection pooling

    When the internal VIP is moved in the event of a failure of the active
    controller, OpenStack services can become unresponsive as they try to
    talk with MariaDB using connections from the SQLAlchemy pool.

    It has been argued that OpenStack doesn't really need to use connection
    pooling with MariaDB [1]. This commit reduces the use of connection
    pooling via two configuration options:

    - max_pool_size is set to 1 to allow only a single connection in the
      pool (it is not possible to disable connection pooling entirely via
      oslo.db, and max_pool_size = 0 means unlimited pool size)
    - lower connection_recycle_time from the default of one hour to 10
      seconds, which means the single connection in the pool will be
      recreated regularly

    These settings have shown better reactivity of the system in the event
    of a failover.

    [1] http://lists.openstack.org/pipermail/openstack-dev/2015-April/061808.html

    Change-Id: Ib6a62d4428db9b95569314084090472870417f3d
    Closes-Bug: #1896635
    (cherry picked from commit c81772024c70b84564cfddb29645390ae378498f)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/754928
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=1d4fd52e4e77d5b93f21ccf9f382ae110459f3ff
Submitter: Zuul
Branch: stable/ussuri

commit 1d4fd52e4e77d5b93f21ccf9f382ae110459f3ff
Author: Pierre Riteau <email address hidden>
Date: Tue Sep 22 17:52:36 2020 +0200

    Reduce the use of SQLAlchemy connection pooling

    When the internal VIP is moved in the event of a failure of the active
    controller, OpenStack services can become unresponsive as they try to
    talk with MariaDB using connections from the SQLAlchemy pool.

    It has been argued that OpenStack doesn't really need to use connection
    pooling with MariaDB [1]. This commit reduces the use of connection
    pooling via two configuration options:

    - max_pool_size is set to 1 to allow only a single connection in the
      pool (it is not possible to disable connection pooling entirely via
      oslo.db, and max_pool_size = 0 means unlimited pool size)
    - lower connection_recycle_time from the default of one hour to 10
      seconds, which means the single connection in the pool will be
      recreated regularly

    These settings have shown better reactivity of the system in the event
    of a failover.

    [1] http://lists.openstack.org/pipermail/openstack-dev/2015-April/061808.html

    Change-Id: Ib6a62d4428db9b95569314084090472870417f3d
    Closes-Bug: #1896635
    (cherry picked from commit c81772024c70b84564cfddb29645390ae378498f)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/754931
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=9ee209080982e66acff3f15fd64c9db965084f8b
Submitter: Zuul
Branch: stable/stein

commit 9ee209080982e66acff3f15fd64c9db965084f8b
Author: Pierre Riteau <email address hidden>
Date: Tue Sep 22 17:52:36 2020 +0200

    Reduce the use of SQLAlchemy connection pooling

    When the internal VIP is moved in the event of a failure of the active
    controller, OpenStack services can become unresponsive as they try to
    talk with MariaDB using connections from the SQLAlchemy pool.

    It has been argued that OpenStack doesn't really need to use connection
    pooling with MariaDB [1]. This commit reduces the use of connection
    pooling via two configuration options:

    - max_pool_size is set to 1 to allow only a single connection in the
      pool (it is not possible to disable connection pooling entirely via
      oslo.db, and max_pool_size = 0 means unlimited pool size)
    - lower connection_recycle_time from the default of one hour to 10
      seconds, which means the single connection in the pool will be
      recreated regularly

    These settings have shown better reactivity of the system in the event
    of a failover.

    [1] http://lists.openstack.org/pipermail/openstack-dev/2015-April/061808.html

    Change-Id: Ib6a62d4428db9b95569314084090472870417f3d
    Closes-Bug: #1896635
    (cherry picked from commit c81772024c70b84564cfddb29645390ae378498f)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 8.3.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 10.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 9.3.0 release.