Cinder LVM over iSCSI driver under qemu 2.0 and kernel <3.11

[Pavel Boldin]

Usage of the Cinder LVM over iSCSI in Fuel 5.1.1 leads to messages of hanged qemu-system-x86_64 process being produced by the kernel.

This is caused by the call 'fdatasync' called by the qemu on the iSCSI device when the kernel inside qemu emits 'SYNCHRONIZE_CACHE' SCSI command. The kernel code that handles this route (blkdev_flush_cache) waits on synchronization completion (by wait_for_completion_io) and therefore blocks process in the UNINTERRUPTIBLE state.

The solution is either to update to a newer kernel (3.12) that has a completely new block queue implementation or to update qemu not to try to sync entire iSCSI partition at once.

[Pavel Boldin (pboldin)]

Using a novel kernel (3.13.0-36-generic) does not help to solve this problem.

The only other solution is to flush a device step by step. For this to be implemented `qemu` and `tgtd` must be updated.

[Pavel Boldin (pboldin)]

Uploaded is the enhanced log file with backtraces of all the CPUs on the machine.

[Pavel Boldin (pboldin)]

New logs uploaded. These are results of invocation of the `show_state` kernel function on hung-up action.

[Pavel Boldin (pboldin)]

According to investigation results it was a `tgt` daemon who was responsible for the over-caching. This led to a `fsync(2)` call taking too long to flush cache, since the cache can be as big as few percents of memory (controlled by /proc/sys/vm/dirty_*).

The solution is one of the following:

1) Use 'write-cache off' option for the `tgt` volume configuration. This generates a lot of `fsync(2)`s being issued to the iSCSI LUN. Because of these synchronizations performance degrades drastically (around 50%).

Implementing this only requires updating the python-cinder OpenStack package for Fuel 5.1.1.

2) Use 'bsoflags direct' option for `tgt` volume configuration. This instructs `tgt` to open a backing-store device using O_DIRECT. All of the operations then miss the page cache and go directly to underlying device (a LVM device served by the 'dm' Linux module). This has almost no impact on the device performance.

Implementing this, however, requires updating both `tgt`(i.e. the `tgt-admin` script) and python-cinder packages for Fuel 5.1.1.

Currently both of the solutions are being benchmarked. More results to follow.

[Pavel Boldin (pboldin)]

According to benchmark results only former (i.e. `bsoflags direct`) gives true results for the `ext` FS benchmark via `fio` utility with `direct=true` option.

In my opinion, the guest VM must be responsible for all the caching so it happens transparently for the user application. All other layers of this monster (qemu <-> iscsi <-> tgt <-> lvm <-> disks) must use O_DIRECT or its analogs so no false feeling of safety is experienced by the user application (and fewer abstractions to break).

I'm going to incorporate changes into the Fuel's cinder and update `tgt-admin` script as appropriately.

[OSCI Robot (oscirobot)]

Package cinder has been built from changeset: https://review.fuel-infra.org/364
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1.1-stable-364/centos

[OSCI Robot (oscirobot)]

Package cinder has been built from changeset: https://review.fuel-infra.org/364
DEB Repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1.1-stable-364/ubuntu

[OSCI Robot (oscirobot)]

Package tgt has been built from changeset: https://review.fuel-infra.org/379
DEB Repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1.1-stable-379/ubuntu

[OSCI Robot (oscirobot)]

Package cinder has been built from changeset: https://review.fuel-infra.org/364
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1.1-stable-364/centos

[OSCI Robot (oscirobot)]

Package cinder has been built from changeset: https://review.fuel-infra.org/364
DEB Repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1.1-stable-364/ubuntu

[Dmitry Borodaenko (angdraug)]

If there commits ready to be merged the status should be In Progress. Where's a fix for 6.0? And why is this targeted for maintenance releases if it's not High priority?

[Pavel Boldin (pboldin)]

It is still unclear if there is need to apply this for the upcoming 6.0 release.

The process here is to consider the performance benchmarks and to derive a decision from these.

Status and priority are to be updated.

[OSCI Robot (oscirobot)]

Package tgt has been built from changeset: https://review.fuel-infra.org/379
DEB Repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1.1-stable-379/ubuntu

[OSCI Robot (oscirobot)]

Package tgt has been built from changeset: https://review.fuel-infra.org/379
DEB Repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1.1-stable/ubuntu

[OSCI Robot (oscirobot)]

Package cinder-2014.1.1-fuel5.1.1.mira13 has been built from changeset: https://review.fuel-infra.org/364
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1.1-stable/centos

[OSCI Robot (oscirobot)]

Package cinder-2014.1.1-fuel5.1.1~mira20 has been built from changeset: https://review.fuel-infra.org/364
DEB Repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1.1-stable/ubuntu

[Dmitry Borodaenko (angdraug)]

Is this still in progress for 5.1.1?

[Pavel Boldin (pboldin)]

The appropriate glitch in 6.0 should be fixed by the 'write-cache off'. Since this is not truly a bug moving to a 'Won't Fix' status.

Will merge `bsoflags` patch with the upstream version of Cinder.

[Michael Semenov (msemenov)]

That was a concern - to add this fix into 6.0 or not, because we were need to test the performance first. We have not tested it yet and considered to make this decision in 6.1 scope.

Hence, Won't fix in 6.0.

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-6.1/2014.2
Change author: Pavel Boldin <email address hidden>
Review: https://review.fuel-infra.org/6295

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Pavel Boldin <email address hidden> on branch: openstack-ci/fuel-6.1/2014.2
Review: https://review.fuel-infra.org/5462
Reason: https://review.fuel-infra.org/#/c/6295/

[Pavel Boldin (pboldin)]

Since it seems like Ceph would be a default storage backend for 6.1 I move that to 7.0 and link it with the upstream bug:

https://bugs.launchpad.net/cinder/+bug/1441935

[Pavel Boldin (pboldin)]

My bad, we actually implemented a local patch for our Mirantis OpenStack. So, this is fixed in the 6.1 and I need to push the fix to the upstream.

[Kyrylo Romanenko (kromanenko)]

Ubuntu Deployment:
1 Controller
1 Compute
1 Cinder with LVM over iSCSI.

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.1"
  openstack_version: "2014.2.2-6.1"
  api: "1.0"
  build_number: "437"
  build_id: "2015-05-19_10-05-51"
  nailgun_sha: "593c99f2b46cf52b2be6c7c6e182b6ba9f2232cd"
  python-fuelclient_sha: "e19f1b65792f84c4a18b5a9473f85ef3ba172fce"
  astute_sha: "96801c5bccb14aa3f2a0d7f27f4a4b6dd2b4a548"
  fuel-library_sha: "2814c51668f487e97e1449b078bad1942421e6b9"
  fuel-ostf_sha: "9ce1800749081780b8b2a4a7eab6586583ffaf33"
  fuelmain_sha: "68796aeaa7b669e68bc0976ffd616709c937187a"

Verification steps:

1. Create Volume
2. Run Instance.
3. Attache volume to Instance
4. Check on Cinder Node:
# tgt-admin -s
Target 1: iqn.2010-10.org.openstack:volume-bbd0fe91-0057-4f12-924b-83d08c2e8d45
    System information:
        Driver: iscsi
        State: ready
    I_T nexus information:
        I_T nexus: 1
            Initiator: iqn.1993-08.org.debian:01:b8c3f770cfbd alias: node-2.domain.tld
            Connection: 0
                IP Address: 192.168.1.2
    LUN information:
        LUN: 0
            Type: controller
            SCSI ID: IET 00010000
            SCSI SN: beaf10
            Size: 0 MB, Block size: 1
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            SWP: No
            Thin-provisioning: No
            Backing store type: null
            Backing store path: None
            Backing store flags:
        LUN: 1
            Type: disk
            SCSI ID: IET 00010001
            SCSI SN: beaf11
            Size: 1074 MB, Block size: 512
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            SWP: No
            Thin-provisioning: No
            Backing store type: rdwr
            Backing store path: /dev/cinder/volume-bbd0fe91-0057-4f12-924b-83d08c2e8d45
            Backing store flags:
    Account information:
        344jnGsTMZc4zFk8swxd
    ACL information:
        ALL
5.
 # cat /var/lib/cinder/volumes/*
<target iqn.2010-10.org.openstack:volume-bbd0fe91-0057-4f12-924b-83d08c2e8d45>
    backing-store /dev/cinder/volume-bbd0fe91-0057-4f12-924b-83d08c2e8d45
    lld iscsi
    bsoflags direct
    incominguser 344jnGsTMZc4zFk8swxd 4Xp2kkm9uyYtKeHB
    write-cache on

Check that "bsoflags direct" is used.

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: Pavel Boldin <email address hidden>
Review: https://review.fuel-infra.org/8095

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Anton Arefiev <email address hidden> on branch: openstack-ci/fuel-7.0/2015.1.0
Review: https://review.fuel-infra.org/8095
Reason: No needs anymore, cherry-picked from upstream version: https://review.fuel-infra.org/#/c/9421/