db sync must be run once from the bootstrap node

Bug #1339047 reported by Michael Kerrin
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
High
Adam Gandelman

Bug Description

For each service we need to run db sync once only to initlaize the database. In an HA setup we run the risk of the migration stepping on each other in which case it can case the db sync call to hang on one machine, error out out on an other controller.

Revision history for this message
Adam Gandelman (gandelman-a) wrote :

This is true for initial bootstrapping of the cloud but also post-rebuild. Currently, the boot-stack element calls a 'reset-db' script that creates databases and conditionally migrates them from the bootstrap node and only once (based on existence of a flag file). Note the argument to migrate the db here is never passed and it currently only initializes empty databases. Currently, DB migration really happens as part of the indvidual service elements post-configure, with the exception of neutron which actually *does* get migrated from reset-db (for some reason?)

os-create-db is idempotent enough to run every execution, and so are the migrations (in theory). I think we should be good to update boot-stack to alwasy run migrations only from the bootstrap node, and remove the duplicate calls from the individual service elements.

Changed in tripleo:
status: New → In Progress
assignee: nobody → Adam Gandelman (gandelman-a)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-image-elements (master)

Fix proposed to branch: master
Review: https://review.openstack.org/105640

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/105678

Changed in tripleo:
assignee: Adam Gandelman (gandelman-a) → Michael Kerrin (michael-kerrin-w)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tripleo-image-elements (master)

Change abandoned by Michael Kerrin (<email address hidden>) on branch: master
Review: https://review.openstack.org/105678
Reason: seems to be alrady done

Ben Nemec (bnemec)
Changed in tripleo:
importance: Undecided → High
Changed in tripleo:
assignee: Michael Kerrin (michael-kerrin-w) → Adam Gandelman (gandelman-a)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Change abandoned by Adam Gandelman (<email address hidden>) on branch: master
Review: https://review.openstack.org/105640

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-image-elements (master)

Fix proposed to branch: master
Review: https://review.openstack.org/106463

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-image-elements (master)

Reviewed: https://review.openstack.org/106463
Committed: https://git.openstack.org/cgit/openstack/tripleo-image-elements/commit/?id=025110241e08087bbead80ab4cef78f0e345244c
Submitter: Jenkins
Branch: master

commit 025110241e08087bbead80ab4cef78f0e345244c
Author: Adam Gandelman <email address hidden>
Date: Fri Jul 11 11:32:52 2014 -0700

    Run migrations from only the bootstrap host

    This ensures database migrations only happen on one node. It moves database
    migration from post-configure.d to configure.d within individual service
    elements and protects all 'db sync' call with an os-is-bootstrap-host check
    until we have a better story regarding leadership checks. Also removes all
    redundant 'db sync' calls from the boot-stack element (they were not actually
    being run there, anyway).

    Change-Id: Idf3c642614699cf3f79aefd7434b3eb4983df1ae
    Closes-bug: #1339047

Changed in tripleo:
status: In Progress → Fix Committed
Jay Dobies (jdob)
Changed in tripleo:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.