[LVM] Volume migration destroys volume data

Bug #1454835 reported by Mitsuhiro Tanino
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
Fix Released
Critical
Mitsuhiro Tanino
Juno
Fix Released
High
Mitsuhiro Tanino
Kilo
Fix Released
High
Mitsuhiro Tanino

Bug Description

When I migrated 1GB volume between two LVM backends on the same server, it looks like volume migration was succeeded, but the volume data was broken in fact.

The root cause is migrate_volume in lvm.py does not pass volume size in MiB but passes volume size in GiB to copy_volume().
The size argument of copy_volume() requires MiB value. As a result, copy_volume() copied only 1MiB head of 1GiB volume to the destination volume, and then the volume data was corrupted.

<dd command>
"sudo cinder-rootwrap /etc/cinder/rootwrap.conf dd if=/dev/mapper/vg1-volume--a192e7d1--d38b--49c3--bc4f--22e5a3f0de64 of=/dev/mapper/vg2-volume--a192e7d1--d38b--49c3--bc4f--22e5a3f0de64 count=1 bs=1M iflag=direct oflag=direct"

<Log>
2015-05-12 19:56:57.647 DEBUG oslo_concurrency.processutils [req-94fbff8c-754d-4c47-ad0f-89019d13a613 c0996698721042efa6d1c9620750cabf 2e82056108c34d1e8409275f52da90bd] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf dd if=/dev/mapper/vg1-volume--a192e7d1--d38b--49c3--bc4f--22e5a3f0de64 of=/dev/mapper/vg2-volume--a192e7d1--d38b--49c3--bc4f--22e5a3f0de64 count=1 bs=1M iflag=direct oflag=direct from (pid=6479) execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:199
2015-05-12 19:56:57.706 DEBUG oslo_concurrency.processutils [req-94fbff8c-754d-4c47-ad0f-89019d13a613 c0996698721042efa6d1c9620750cabf 2e82056108c34d1e8409275f52da90bd] CMD "sudo cinder-rootwrap /etc/cinder/rootwrap.conf dd if=/dev/mapper/vg1-volume--a192e7d1--d38b--49c3--bc4f--22e5a3f0de64 of=/dev/mapper/vg2-volume--a192e7d1--d38b--49c3--bc4f--22e5a3f0de64 count=1 bs=1M iflag=direct oflag=direct" returned: 0 in 0.059s from (pid=6479) execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:225
2015-05-12 19:56:57.706 DEBUG cinder.volume.utils [req-94fbff8c-754d-4c47-ad0f-89019d13a613 c0996698721042efa6d1c9620750cabf 2e82056108c34d1e8409275f52da90bd] Volume copy details: src /dev/mapper/vg1-volume--a192e7d1--d38b--49c3--bc4f--22e5a3f0de64, dest /dev/mapper/vg2-volume--a192e7d1--d38b--49c3--bc4f--22e5a3f0de64, size 1.00 MB, duration 1.00 sec from (pid=6479) _copy_volume /opt/stack/cinder/cinder/volume/utils.py:324

Changed in cinder:
assignee: nobody → Mitsuhiro Tanino (mitsuhiro-tanino)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.openstack.org/182832

Changed in cinder:
status: New → In Progress
Jay Bryant (jsbryant)
Changed in cinder:
importance: Undecided → High
milestone: none → liberty-1
Revision history for this message
John Griffith (john-griffith) wrote :

Data Loss bug counts as critical IMO

Changed in cinder:
importance: High → Critical
Revision history for this message
John Griffith (john-griffith) wrote :

Looks like this was introduced here: commit: 30397d8a0a22f3ecad0001ac50a084cc41f4a43d

I'm not sure I agree that the fix shouldn't be to put the copy routine back to using Gig, given that everything related to volumes in Cinder is Gig it would seem safer to me.

Given this appears to have been around since July of 2013 maybe not so critical after all.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (master)

Reviewed: https://review.openstack.org/182832
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=a5da4c353a1df17341c1e745796e00a4ec1afd21
Submitter: Jenkins
Branch: master

commit a5da4c353a1df17341c1e745796e00a4ec1afd21
Author: Mitsuhiro Tanino <email address hidden>
Date: Wed May 13 11:59:13 2015 -0400

    LVM: Pass volume size in MiB to copy_volume() during volume migration

    Currently migrate_volume() in lvm.py does not pass volume size in
    MiB but passes volume size in GiB to copy_volume(). The size argument
    of copy_volume() requires MiB value. As a result, if the volume size
    is 1GiB, copy_volume() copies only 1MiB head of 1GiB volume to the
    destination volume, and then the volume data is corrupted.
    This patch fixes to pass volume size in MiB.

    Closes-bug: #1454835
    Change-Id: I8e06195dc3625ef07bc9858d844541e37a231b73

Changed in cinder:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (stable/kilo)

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/183198

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (stable/kilo)

Reviewed: https://review.openstack.org/183198
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=c0f12f28be44a2e05932de1f222698dce221b2b7
Submitter: Jenkins
Branch: stable/kilo

commit c0f12f28be44a2e05932de1f222698dce221b2b7
Author: Mitsuhiro Tanino <email address hidden>
Date: Wed May 13 11:59:13 2015 -0400

    LVM: Pass volume size in MiB to copy_volume() during volume migration

    Currently migrate_volume() in lvm.py does not pass volume size in
    MiB but passes volume size in GiB to copy_volume(). The size argument
    of copy_volume() requires MiB value. As a result, if the volume size
    is 1GiB, copy_volume() copies only 1MiB head of 1GiB volume to the
    destination volume, and then the volume data is corrupted.
    This patch fixes to pass volume size in MiB.

    (cherry picked from commit a5da4c353a1df17341c1e745796e00a4ec1afd21)
    Conflicts:
        cinder/tests/test_volume.py
        (Added mock for create_export.)

    Closes-bug: #1454835
    Change-Id: I8e06195dc3625ef07bc9858d844541e37a231b73

tags: added: in-stable-kilo
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (stable/juno)

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/183361

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/183498

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on cinder (stable/juno)

Change abandoned by Mitsuhiro Tanino (<email address hidden>) on branch: stable/juno
Review: https://review.openstack.org/183361

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (stable/juno)

Reviewed: https://review.openstack.org/183498
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=7955d5fc66340f826255ac81a3984d7984f56041
Submitter: Jenkins
Branch: stable/juno

commit 7955d5fc66340f826255ac81a3984d7984f56041
Author: Mitsuhiro Tanino <email address hidden>
Date: Thu May 14 19:35:11 2015 -0400

    LVM: Pass volume size in MiB to copy_volume() during volume migration

    Currently migrate_volume() in lvm.py does not pass volume size in
    MiB but passes volume size in GiB to copy_volume(). The size argument
    of copy_volume() requires MiB value. As a result, if the volume size
    is 1GiB, copy_volume() copies only 1MiB head of 1GiB volume to the
    destination volume, and then the volume data is corrupted.
    This patch fixes to pass volume size in MiB.

    (cherry picked from commit a5da4c353a1df17341c1e745796e00a4ec1afd21)
    Conflicts:
        cinder/tests/test_volume.py
        (Added mock for create_export. Changed to use contextlib.nested)

    Closes-bug: #1454835
    Change-Id: I8e06195dc3625ef07bc9858d844541e37a231b73

tags: added: in-stable-juno
Thierry Carrez (ttx)
Changed in cinder:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in cinder:
milestone: liberty-1 → 7.0.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.