glusterfs: Instance is not using the correct volume snapshot file after reboot

[Thang Pham]

Instance is not using the correct volume snapshot file after reboot.

Steps to recreate bug:
1. Create a volume

2. Attach volume to a running instance.

3. Take an online snapshot of the volume.
Note that the active volume used by the instance is now switched to volume-<uuid>.<snapshot-uuid>.

4. Shutdown the instance.

5. Start the instance.
If you invoke virsh dumpxml <instance>, you will see that it is re-attaching the base volume ( volume-<uuid>) to the instance and not the snapshot volume (volume-<uuid>.<snapshot-uuid>). The expected behavior is to have the snapshot volume re-attach to the instance.

This bug will cause data corruption in the snapshot and volume.

It looks like the nova volume manager is using a stale copy of the block_device_mapping. The block_device_mapping needs to be refreshed in order for the updated volume snapshot to be used.

On power on, the nova manager (nova/compute/manager.py ) does:
1. start_instance
2. _power_on
3. _get_instance_volume_block_device_info

The structure for this method is:
def _get_instance_volume_block_device_info(self, context, instance, refresh_conn_info=False, bdms=None):
    if not bdms:
        bdms = (block_device_obj.BlockDeviceMappingList.
                    get_by_instance_uuid(context, instance['uuid']))
        block_device_mapping = (
            driver_block_device.convert_volumes(bdms) +
            driver_block_device.convert_snapshots(bdms) +
            driver_block_device.convert_images(bdms))
    ....
block_device_obj.BlockDeviceMappingList.get_by_instance_uuid() goes and queries the database to construct the bdms, which will contain stale data.

[Eric Harney (eharney)]

This analysis does not sound quite right to me, but I could be missing something. connect_volume() uses data['name'] to determine the filename. This is populated with the info from the Cinder GlusterFS driver's initialize_connection(), which sets it correctly.

The responsibility for calculating which filename should be attached to the VM is purely on the Cinder side, Nova doesn't do that calculation.

[Thang Pham (thang-pham)]

You are correct. I was only looking at where the file name came from in nova, but not the actual root source of the file name, which is in initialize_connection.

[Thang Pham (thang-pham)]

I debugged this problem a bit more. It looks like the nova volume manager is using a stale copy of the block_device_mapping. The block_device_mapping needs to be refreshed in order for the updated volume snapshot to be used.

On power on, the nova manager (nova/compute/manager.py ) does:
1. start_instance
2. _power_on
3. _get_instance_volume_block_device_info

The structure for this method is:
def _get_instance_volume_block_device_info(self, context, instance, refresh_conn_info=False, bdms=None):
    if not bdms:
        bdms = (block_device_obj.BlockDeviceMappingList.
                    get_by_instance_uuid(context, instance['uuid']))
        block_device_mapping = (
            driver_block_device.convert_volumes(bdms) +
            driver_block_device.convert_snapshots(bdms) +
            driver_block_device.convert_images(bdms))
    ....
block_device_obj.BlockDeviceMappingList.get_by_instance_uuid() goes and queries the database to construct the bdms, which will contain stale data.

In order for this to work properly, refresh_conn_info must be set to True. However, since _get_instance_volume_block_device_info is used by everyone, I am not sure if setting refresh_conn_info = True is such a good solution, since it will affect other cinder drivers also.

[Nikola Đipanov (ndipanov)]

Looking at it closer, I'd say that we just need to refresh the connection info once we are done with the snapshot.

We should be careful to avoid any race conditions tho, but basically what we need to do in both volume_snapshot_create and _delete methods of the libvirt driver is:

  from nova.virt import block_device as driver_block_device
  bdm = block_device_obj.BlockDeviceMapping.get_by_volume_id(
                    context, volume_id)
  driver_bdm = driver_block_device.DriverVolumeBlockDevice(bdm)
  driver_bdm.refresh_connection_info(context, instance, self._volume_api, self)

and you should be good to go.

The only caveat here is that refresh_connection_info will call out to Cinder's initialize_connection which may be problematic since the snapshot is in "creating" state. But nothing that a dumb polling loop won't solve (sigh). In reality we should probably rething these interactions.

[Thang Pham (thang-pham)]

Nikola: Thank you for the pointer :) I will try out your advice above.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/87432

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/87432
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=329b594436aad814e43740ea26841342a8772aff
Submitter: Jenkins
Branch: master

commit 329b594436aad814e43740ea26841342a8772aff
Author: Thang Pham <email address hidden>
Date: Mon Apr 14 22:47:06 2014 -0400

    libvirt: Refresh volume connection_info after volume snapshot

    The following patch is related to the guest assisted snapshot
    functionality. When you take a snapshot of a volume
    (e.g. GlusterFS) attached to a running instance, a new snapshot
    file is created, i.e. volume-<uuid>.<snapshot-uuid>. The
    instance uses this file as the active volume. If you shutdown
    and restart the instance, nova will reattach the base volume
    (volume-<uuid>) to the instance instead of the snapshot volume
    (volume-<uuid>.<snapshot-uuid>). The expected behavior is to
    have the snapshot volume reattach to the instance. This is
    caused by stale data being returned from the database when
    _get_instance_volume_block_device_info is called during
    _power_on. To fix this bug, this patch calls
    refresh_connection_info to update the database in both
    volume_snapshot_create and _volume_snapshot_delete methods of the
    libvirt driver.

    Change-Id: I0f340a3f879580e7981d97863bc299e33d71aa84
    Closes-Bug: #1304695

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/98255

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/98255
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=96212b194d390c4de9298376aefe84e3a70008de
Submitter: Jenkins
Branch: stable/icehouse

commit 96212b194d390c4de9298376aefe84e3a70008de
Author: Thang Pham <email address hidden>
Date: Mon Apr 14 22:47:06 2014 -0400

    libvirt: Refresh volume connection_info after volume snapshot

    The following patch is related to the guest assisted snapshot
    functionality. When you take a snapshot of a volume
    (e.g. GlusterFS) attached to a running instance, a new snapshot
    file is created, i.e. volume-<uuid>.<snapshot-uuid>. The
    instance uses this file as the active volume. If you shutdown
    and restart the instance, nova will reattach the base volume
    (volume-<uuid>) to the instance instead of the snapshot volume
    (volume-<uuid>.<snapshot-uuid>). The expected behavior is to
    have the snapshot volume reattach to the instance. This is
    caused by stale data being returned from the database when
    _get_instance_volume_block_device_info is called during
    _power_on. To fix this bug, this patch calls
    refresh_connection_info to update the database in both
    volume_snapshot_create and _volume_snapshot_delete methods of the
    libvirt driver.

    Conflicts:
     nova/tests/virt/libvirt/test_libvirt.py

    Change-Id: I0f340a3f879580e7981d97863bc299e33d71aa84
    Closes-Bug: #1304695