[library] ironic-conductor service hangs on start when mysql is not available

[Andrey Shestakov]

After all nodes reboot, ironic-conductor failed to start properly.

logs form ironic-conductor.log
2016-01-18 10:09:27.171 6889 DEBUG oslo_db.api [req-43c3a799-4192-421b-95c3-7716a0bf2107 admin - - - -] Loading backend 'sqlalchemy' from 'ironic.db.sqlalchemy.api' _load_backend /usr/lib/python2.7/dist-packages/oslo_db/api.py:230
2016-01-18 10:09:27.321 6889 WARNING oslo_db.sqlalchemy.engines [req-43c3a799-4192-421b-95c3-7716a0bf2107 admin - - - -] SQL connection failed. 10 attempts left.
2016-01-18 10:09:37.332 6889 WARNING oslo_db.sqlalchemy.engines [req-43c3a799-4192-421b-95c3-7716a0bf2107 admin - - - -] SQL connection failed. 9 attempts left.
2016-01-18 10:09:47.352 6889 WARNING oslo_db.sqlalchemy.engines [req-43c3a799-4192-421b-95c3-7716a0bf2107 admin - - - -] SQL connection failed. 8 attempts left.
2016-01-18 10:09:57.360 6889 WARNING oslo_db.sqlalchemy.engines [req-43c3a799-4192-421b-95c3-7716a0bf2107 admin - - - -] SQL connection failed. 7 attempts left.
2016-01-18 10:10:07.376 6889 WARNING oslo_db.sqlalchemy.engines [req-43c3a799-4192-421b-95c3-7716a0bf2107 admin - - - -] SQL connection failed. 6 attempts left.
2016-01-18 10:10:17.387 6889 WARNING oslo_db.sqlalchemy.engines [req-43c3a799-4192-421b-95c3-7716a0bf2107 admin - - - -] SQL connection failed. 5 attempts left.
2016-01-18 10:10:27.396 6889 WARNING oslo_db.sqlalchemy.engines [req-43c3a799-4192-421b-95c3-7716a0bf2107 admin - - - -] SQL connection failed. 4 attempts left.
2016-01-18 10:10:37.411 6889 WARNING oslo_db.sqlalchemy.engines [req-43c3a799-4192-421b-95c3-7716a0bf2107 admin - - - -] SQL connection failed. 3 attempts left.
2016-01-18 10:10:47.420 6889 WARNING oslo_db.sqlalchemy.engines [req-43c3a799-4192-421b-95c3-7716a0bf2107 admin - - - -] SQL connection failed. 2 attempts left.
2016-01-18 10:10:57.429 6889 WARNING oslo_db.sqlalchemy.engines [req-43c3a799-4192-421b-95c3-7716a0bf2107 admin - - - -] SQL connection failed. 1 attempts left.

process failed to connect to mysql 10 times and just hangs.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/268998

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/268998
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=e7738597474377c27e20d14d1f7c7ca9e10307a1
Submitter: Jenkins
Branch: master

commit e7738597474377c27e20d14d1f7c7ca9e10307a1
Author: Andrey Shestakov <email address hidden>
Date: Mon Jan 18 13:31:54 2016 +0200

    Fix database_max_retries for Ironic services

    Ironic services should try to connect to database infinity.
    database_max_retries set to '-1'

    Change-Id: I594f296f3e5dade1c7f8d728b2bcdec17101470d
    Closes-bug: #1535269

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/270740

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/270740
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=e8fec46a4892b396d150c0f5b82121f23ce47a2c
Submitter: Jenkins
Branch: stable/8.0

commit e8fec46a4892b396d150c0f5b82121f23ce47a2c
Author: Andrey Shestakov <email address hidden>
Date: Mon Jan 18 13:31:54 2016 +0200

    Fix database_max_retries for Ironic services

    Ironic services should try to connect to database infinity.
    database_max_retries set to '-1'

    Change-Id: I594f296f3e5dade1c7f8d728b2bcdec17101470d
    Closes-bug: #1535269
    (cherry picked from commit e7738597474377c27e20d14d1f7c7ca9e10307a1)

[Pavlo Shchelokovskyy (pshchelo)]

setting as fix commited for 8.0, bot has not closed it automagically

[Kyrylo Romanenko (kromanenko)]

Verified on ISO:
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "529"
  build_id: "529"
  fuel-nailgun_sha: "baec8643ca624e52b37873f2dbd511c135d236d9"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "658be72c4b42d3e1436b86ac4567ab914bfb451b"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "e2d79330d5d708796330fac67722c21f85569b87"
  fuel-ostf_sha: "3bc76a63a9e7d195ff34eadc29552f4235fa6c52"
  fuel-mirror_sha: "fb45b80d7bee5899d931f926e5c9512e2b442749"
  fuelmenu_sha: "e071216cb214e34b4d861478033425ee6a54a3be"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "a43cf96cd9532f10794dce736350bf5bed350e9d"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "a365f05b903368225da3fea9aa42afc1d50dc9b4"

[Sofiia Andriichenko (sandriichenko)]

[root@nailgun mos-rally-verify]# shotgun2 short-report
cat /etc/fuel_build_id:
 172
cat /etc/fuel_build_number:
 172
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 mitaka-9.0
rpm -qa | egrep 'fuel|astute|network-checker|nailgun|packetary|shotgun':
 fuel-release-9.0.0-1.mos6331.noarch
 fuel-misc-9.0.0-1.mos8275.noarch
 python-packetary-9.0.0-1.mos131.noarch
 fuel-openstack-metadata-9.0.0-1.mos8635.noarch
 fuel-provisioning-scripts-9.0.0-1.mos8635.noarch
 python-fuelclient-9.0.0-1.mos302.noarch
 fuel-9.0.0-1.mos6331.noarch
 fuel-nailgun-9.0.0-1.mos8635.noarch
 rubygem-astute-9.0.0-1.mos736.noarch
 fuel-library9.0-9.0.0-1.mos8275.noarch
 fuel-agent-9.0.0-1.mos272.noarch
 fuel-ui-9.0.0-1.mos2646.noarch
 fuel-setup-9.0.0-1.mos6331.noarch
 nailgun-mcagents-9.0.0-1.mos736.noarch
 shotgun-9.0.0-1.mos87.noarch
 network-checker-9.0.0-1.mos72.x86_64
 fuel-bootstrap-cli-9.0.0-1.mos272.noarch
 fuel-migrate-9.0.0-1.mos8275.noarch
 fuelmenu-9.0.0-1.mos265.noarch
 fuel-notify-9.0.0-1.mos8275.noarch
 fuel-ostf-9.0.0-1.mos923.noarch
 fuel-mirror-9.0.0-1.mos131.noarch
 fuel-utils-9.0.0-1.mos8275.noarch
[root@nailgun mos-rally-verify]#