heat does not read sql db config option

[Steven Dake]

We need to tune connections to database and AMQP.

Some of those options include daemons startup behavior and specifically:

max_retries = -1
retry_interval = 1

to allow the daemon to continue retrying connecting.

[database]
# maximum db connection retries during startup. (setting -1
# implies an infinite retry count) (integer value)
max_retries=-1

# interval between retries of opening a sql connection
# (integer value)
retry_interval=1

[root@rhos4-heat1 heat]# /etc/init.d/openstack-heat-engine start
with mysql down, will result in a traceback (this is with or without max_retries/retry_interval

ideally there should be no traceback, and instead the daemon should retry until the database is available.

[Steven Hardy (shardy)]

I can't reproduce this with latest master:

# maximum db connection retries during startup. (setting -1
# implies an infinite retry count) (integer value)
max_retries=10

# interval between retries of opening a sql connection
# (integer value)
retry_interval=10

2014-02-05 15:35:38.534 11265 INFO heat.openstack.common.rpc.impl_qpid [-] Connected to AMQP server on localhost:5672
2014-02-05 15:35:38.535 11265 DEBUG heat.openstack.common.rpc.service [-] Creating Consumer connection for Service engine start /home/shardy/git/heat/heat/openstack/common/rpc/service.py:48
2014-02-05 15:35:38.552 11265 WARNING heat.openstack.common.db.sqlalchemy.session [req-9c32cb73-9953-4dd1-9277-fc3b4e0d5316 None] This application has not enabled MySQL traditional mode, which means silent data corruption may occur. Please encourage the application developers to enable this mode.
2014-02-05 15:35:38.555 11265 WARNING heat.openstack.common.db.sqlalchemy.session [req-9c32cb73-9953-4dd1-9277-fc3b4e0d5316 None] SQL connection failed. 10 attempts left.
2014-02-05 15:35:48.557 11265 WARNING heat.openstack.common.db.sqlalchemy.session [req-9c32cb73-9953-4dd1-9277-fc3b4e0d5316 None] SQL connection failed. 9 attempts left.
2014-02-05 15:35:58.559 11265 WARNING heat.openstack.common.db.sqlalchemy.session [req-9c32cb73-9953-4dd1-9277-fc3b4e0d5316 None] SQL connection failed. 8 attempts left.
2014-02-05 15:36:08.561 11265 WARNING heat.openstack.common.db.sqlalchemy.session [req-9c32cb73-9953-4dd1-9277-fc3b4e0d5316 None] SQL connection failed. 7 attempts left.

And with the exact settings mentioned above:

# maximum db connection retries during startup. (setting -1
# implies an infinite retry count) (integer value)
max_retries=-1

# interval between retries of opening a sql connection
# (integer value)
retry_interval=1

2014-02-05 15:38:44.461 11675 INFO heat.openstack.common.rpc.impl_qpid [-] Connected to AMQP server on localhost:5672
2014-02-05 15:38:44.461 11675 DEBUG heat.openstack.common.rpc.service [-] Creating Consumer connection for Service engine start /home/shardy/git/heat/heat/openstack/common/rpc/service.py:48
2014-02-05 15:38:44.477 11675 WARNING heat.openstack.common.db.sqlalchemy.session [req-bd52823e-0de8-475c-ad87-7c0f10818f58 None] This application has not enabled MySQL traditional mode, which means silent data corruption may occur. Please encourage the application developers to enable this mode.
2014-02-05 15:38:44.479 11675 WARNING heat.openstack.common.db.sqlalchemy.session [req-bd52823e-0de8-475c-ad87-7c0f10818f58 None] SQL connection failed. infinite attempts left.
2014-02-05 15:38:45.481 11675 WARNING heat.openstack.common.db.sqlalchemy.session [req-bd52823e-0de8-475c-ad87-7c0f10818f58 None] SQL connection failed. infinite attempts left.
2014-02-05 15:38:46.481 11675 WARNING heat.openstack.common.db.sqlalchemy.session [req-bd52823e-0de8-475c-ad87-7c0f10818f58 None] SQL connection failed. infinite attempts left.
2014-02-05 15:38:47.482 11675 WARNING heat.openstack.common.db.sqlalchemy.session [req-bd52823e-0de8-475c-ad87-7c0f10818f58 None] SQL connection failed. infinite attempts left.
2014-02-05 15:38:48.483 11675 WARNING heat.openstack.common.db.sqlalc...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/71819

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/71819
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=472c1440677440e0888570926b49542ddf260724
Submitter: Jenkins
Branch: master

commit 472c1440677440e0888570926b49542ddf260724
Author: Ian McLeod <email address hidden>
Date: Wed Feb 5 15:51:58 2014 -0600

    Don't raise MySQL 2013 'Lost connection' errors

    cherry pick, including original commit message below from
    oslo-incubator commit:

    b4f72b2987d53bb434e3a833f06d70a2167f5efc

    Closes bug: 1275838

    Although this code error is part of the `ping_listener` tests, it was
    missing from the list of known - and ignored - connection errors.

    The code error 2013 refers to a connection lost during a query, see:
    https://dev.mysql.com/doc/refman/5.0/en/error-messages-client.html#error_cr_server_lost

    Without this error code, the session won't try to reconnect to mysql,
    which makes `max_retries` useless.

    Change-Id: Id8a1c0665468b503b092c4f01ffd40e2be9c6fa9

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/72191

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/73314

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/73314
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=8bb7830a5068000ca265493a93883b90114ccde7
Submitter: Jenkins
Branch: stable/havana

commit 8bb7830a5068000ca265493a93883b90114ccde7
Author: Ian McLeod <email address hidden>
Date: Wed Feb 5 15:51:58 2014 -0600

    Don't raise MySQL 2013 'Lost connection' errors

    cherry pick, including original commit message below from
    oslo-incubator commit:

    b4f72b2987d53bb434e3a833f06d70a2167f5efc

    Closes bug: 1275838

    Although this code error is part of the `ping_listener` tests, it was
    missing from the list of known - and ignored - connection errors.

    The code error 2013 refers to a connection lost during a query, see:
    https://dev.mysql.com/doc/refman/5.0/en/error-messages-client.html#error_cr_server_lost

    Without this error code, the session won't try to reconnect to mysql,
    which makes `max_retries` useless.

    (cherry picked from commit 472c1440677440e0888570926b49542ddf260724)

    Conflicts:
     heat/openstack/common/db/sqlalchemy/session.py

    Change-Id: Id8a1c0665468b503b092c4f01ffd40e2be9c6fa9