nova + glance services die if started before db is reachable

[Nick Moffitt]

I've got a system where all the controller-type services (glance, api, scheduler, rabbit, etc) are all on the same machine as the mysql DB, and the compute nodes are on the same network.

When I reboot my precise machine, the logs are full of "could not connect to mysql" errors, and half the services aren't running. The upstart scripts should probably make an allowance for starting everything up after databases, or the services should re-try the connections for a while before giving up and crashing.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[David Kranz (david-kranz)]

I just wasted three hours on this one. It is the same as the ubuntu bug
https://bugs.launchpad.net/ubuntu/+source/nova/+bug/959426

My path was as a successful diablo deployer/developer trying to get essex working using a single node to make it easier. Or so I thought. Many people will be hurt by this bug in the near future. It at least needs to be highlighted in the nova install docs if it is not fixed.

[Alan Pevec (apevec)]

> or the services should re-try the connections for a while before giving up and crashing.

Isn't this fixed in bug 943031 which should be in essex-rc1 ?

[Adam Gandelman (gandelman-a)]

This is not resolved by the fix for bug 943031 which, AFAICS, fixes the issue of an existing db connection going away. The bug here is at service start up, ie starting nova-compute with a mysql server that is not up (yet) ends in:

(nova): TRACE: OperationalError: (OperationalError) (2003, "Can't connect to MySQL server on 'outo.home.base' (111)") None None

and a dead process.

Initial setup of the database connection does not retry, and instead the service dies (quickly). This can't really be worked around in packaging/upstart and shouldn't need to be, as a database is truly and external resource anyway. In contrast, the rabbitmq connection initialization loops at start up if it cannot connect to its server, dumping tracebacks to nova-$foo.log but at least keeps the service up and makes a connection when the server appears.

[Adam Gandelman (gandelman-a)]

The fix in bug 943031 was recently re-implemented in Glance @ https://review.openstack.org/#change,5552 While this change currently doesn't resolve the issue of an unreachable database on service startup, it can easily be extended to check the ability to actually connect thru the sqlalchemy engine at startup / during configure_db(). It's been mentioned that the same logic should be a part of nova or openstack-common as well. If it lands there, it will also resolve this issue for nova.

Marking as critical in Ubuntu, adding a task for Glance + tagging essex-rc-potential as this is a serious issue for anyone expecting these services to start reliably.

Note, Keystone seems to initialize its database lazily and not touch till a request comes in. I'm not sure about quantum?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/5760

[Clint Byrum (clint-fewbar)]

Note that this really has nothing to do with whether the services are all on the same host or not. The service being unavailable is not a terminal error, since this host and the other host(s) may be starting up in parallel (think massive powerfailure or complicated system bring-up). So retrying for a considerable amount of time seems to be the appropriate thing to do.

[David Kranz (david-kranz)]

Yes. It could also happen that even after startup the connection is temporarily unavailable and the nova processes should not crash immediately. It should be noted that this problem did not happen ever (for me) in oneiric/diablo-stable and I was/am using a "complicated system bring-up". Right now I cannot reboot the machine running mysql and nova-scheduler because nova-scheduler crashes every time.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/5760
Committed: http://github.com/openstack/glance/commit/12757d70fe6cf0d0fe6e46c5ee09a4f4d3efbd49
Submitter: Jenkins
Branch: master

commit 12757d70fe6cf0d0fe6e46c5ee09a4f4d3efbd49
Author: Adam Gandelman <email address hidden>
Date: Fri Mar 23 18:23:54 2012 -0700

    Ensure functional db connection in configure_db()

    During initial database setup, ensure we can physically connect
    to the database and allow a failed connection to make use of the
    new retry mechanism instead of registry startup failing outright.

    Fixes lp bug #959426.

    Change-Id: I1c87b19913c4204465e5d2027f2f184f0f358fd0

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: milestone-proposed
Review: https://review.openstack.org/5911

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/5911
Committed: http://github.com/openstack/glance/commit/127101a44c0708ef26572abf98a025983ae35aa4
Submitter: Jenkins
Branch: milestone-proposed

commit 127101a44c0708ef26572abf98a025983ae35aa4
Author: Adam Gandelman <email address hidden>
Date: Fri Mar 23 18:23:54 2012 -0700

    Ensure functional db connection in configure_db()

    During initial database setup, ensure we can physically connect
    to the database and allow a failed connection to make use of the
    new retry mechanism instead of registry startup failing outright.

    Fixes lp bug #959426.

    Change-Id: I1c87b19913c4204465e5d2027f2f184f0f358fd0

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/5939

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/5939
Committed: http://github.com/openstack/nova/commit/a4dd6b6f06d222f49bd0d2582dfe0f2925a1638f
Submitter: Jenkins
Branch: master

commit a4dd6b6f06d222f49bd0d2582dfe0f2925a1638f
Author: Adam Gandelman <email address hidden>
Date: Wed Mar 28 18:52:41 2012 -0700

    Ensure a functional database connection

    Allow retrying database connection in get_engine() at an interval. Resolves
    the issue of nova components erroring at startup if a database connection is
    unavailable, particularly at boot. Borrowed from a similar commit to glance,
    (https://review.openstack.org/#change,5552).

    Fixes Bug #959426 for nova.

    Update: * Properly return an engine (fixes tests)
            * Setting sql_max_retries to -1 will retry infinitely
            * Bumped options count in nova.conf.sample
            * i18n log warning
            * Add note to flag help about -1 == infinite
            * Pep8 fix

    Change-Id: Id34eda9e0bad6b477a74e9a7d3575e513e6291d5

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: milestone-proposed
Review: https://review.openstack.org/6075

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/6075
Committed: http://github.com/openstack/nova/commit/ccb93c47370d774dc7b1959f2b2d038a4819855d
Submitter: Jenkins
Branch: milestone-proposed

commit ccb93c47370d774dc7b1959f2b2d038a4819855d
Author: Thierry Carrez <email address hidden>
Date: Mon Apr 2 11:55:35 2012 +0200

    Ensure a functional database connection

    Allow retrying database connection in get_engine() at an interval. Resolves
    the issue of nova components erroring at startup if a database connection is
    unavailable, particularly at boot. Borrowed from a similar commit to glance,
    (https://review.openstack.org/#change,5552).

    This also fixes code duplication due to a half-backport of
    commit 155ef7daab08d7f3fb8f7838df1d715bf1dc2f3f

    Fixes Bug #959426 for nova.

    Change-Id: Ifea94da8347714887c8cae02cc48288f3fa4fa7f

[Adam Gandelman (gandelman-a)]

Fixed in Ubuntu with rc2 upload (nova-2012.1~rc2-0ubuntu1)

[Adam Gandelman (gandelman-a)]

Fixed in Ubuntu as of glance-2012.1~rc2-0ubuntu1