Race condition causes resize-flavor to fail in Percona

Bug #1284989 reported by Steve Leon
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack DBaaS (Trove)
Fix Released
Undecided
Steve Leon

Bug Description

In Percona, when resizing an instance, Trove reports success (ACTIVE) even though percona hasnt truly finished up loading. ALthough mysql service is started and port 3306 is open, percona still does some tasks under the hood.

Normally this wouldnt matter that much. However when an instance is resized immediately after it has finished provisioning, the migrate fails and mysql gets corrupted.

To fix this issue, we need to make sure that percona mysql has shutdown successfully before doing a resize.

Since this is a race condition that happens when a resize is perform immediately after a instance-create finishes, this is only reproducible in automated tests.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to trove (master)

Fix proposed to branch: master
Review: https://review.openstack.org/76446

Changed in trove:
assignee: nobody → Steve Leon (steve-leon)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to trove (master)

Reviewed: https://review.openstack.org/76446
Committed: https://git.openstack.org/cgit/openstack/trove/commit/?id=c88c88fa3ddc2f171b6fec5cce3c09e268294b02
Submitter: Jenkins
Branch: master

commit c88c88fa3ddc2f171b6fec5cce3c09e268294b02
Author: Steve Leon <email address hidden>
Date: Tue Feb 25 22:06:58 2014 -0800

    Fixes a race condition in resize flavor for Percona

    When a resize happens immediately after an instances gets created,
    the mysql process doesnt have enough time to finished up
    provisioning before the resize-action invokes a stop and start
    resizing. Although the trove tries to stop mysql, the stop command
    doesnt ensure that mysql was stopped successfully. This causes
    mysql to be corrupted when migrated to a different VM. Normally
    this is not an issue since mysql provisioning has plenty of time
    to finish. This becomes an issue when doing an resize when mysql
    has just finished provisioned. Only happens in automated tests.
    This fix will ensure that the mysql server is stopped before a
    resize is performed.

    Change-Id: I0a3c351b61b82b88dfe05325a38ce93d6973a9f6
    Closes-Bug: 1284989

Changed in trove:
status: In Progress → Fix Committed
Changed in trove:
milestone: none → icehouse-3
Thierry Carrez (ttx)
Changed in trove:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in trove:
milestone: icehouse-3 → 2014.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.