ceph caching enable

[Andrey Grebennikov]

It would be nice to have caching turned on for Ceph in case we place the VMs into.

We actually need the options:
[client]
    rbd cache = true
    rbd cache writethrough until flush = true

to be added to ceph.conf;
we need the option:

disk_cachemodes="network=writeback"

to be added to nova.conf on the compute nodes.

[Tyler Wilson (loth)]

Tried this out on current master 5.1 build (ceph version 0.80.4 (7c241cfaa6c8c068bc9da8578ca00b9f4fc7567f))
{"build_id": "2014-08-26_21-42-16", "ostf_sha": "4dcd99cc4bfa19f52d4b87ed321eb84ff03844da", "build_number": "9", "auth_required": true, "api": "1.0", "nailgun_sha": "04e3f9d9ad3140cd63a9b5a1a302c03ebe64fd0a", "production": "docker", "fuelmain_sha": "74a97d500bb2fe9528f99771ccc2ec657ae3f76e", "astute_sha": "bc60b7d027ab244039f48c505ac52ab8eb0a990c", "feature_groups": ["experimental"], "release": "5.1", "fuellib_sha": "1e43ca00fe4fb05a485de4bea55bd00d16bda532"}

After changing ceph.conf on all nodes & nova.conf on computes no instances can spawn as they are stuck on 'BUILD: Spawning' for what appears to be forever.

ceph -w shows that the image was cloned from the layered parent in a short time;

2014-08-27 01:10:47.164236 mon.0 [INF] pgmap v880: 7360 pgs: 7360 active+clean; 14419 MB data, 60243 MB used, 7082 GB / 7141 GB avail; 0 B/s rd, 2235 B/s wr, 0 op/s
2014-08-27 01:10:57.208402 mon.0 [INF] pgmap v881: 7360 pgs: 7360 active+clean; 14419 MB data, 60243 MB used, 7082 GB / 7141 GB avail; 0 B/s rd, 2142 B/s wr, 0 op/s
2014-08-27 01:11:06.915590 mon.0 [INF] pgmap v882: 7360 pgs: 7360 active+clean; 14419 MB data, 60243 MB used, 7082 GB / 7141 GB avail; 0 B/s rd, 829 B/s wr, 0 op/s
2014-08-27 01:11:07.934096 mon.0 [INF] pgmap v883: 7360 pgs: 7360 active+clean; 14419 MB data, 60243 MB used, 7082 GB / 7141 GB avail; 9738 B/s rd, 2291 B/s wr, 13 op/s
2014-08-27 01:11:11.921437 mon.0 [INF] pgmap v884: 7360 pgs: 7360 active+clean; 14419 MB data, 60243 MB used, 7082 GB / 7141 GB avail; 22113 B/s rd, 4299 B/s wr, 31 op/s
2014-08-27 01:11:12.936762 mon.0 [INF] pgmap v885: 7360 pgs: 7360 active+clean; 14419 MB data, 60243 MB used, 7082 GB / 7141 GB avail; 16381 B/s rd, 2661 B/s wr, 23 op/s
2014-08-27 01:11:17.212523 mon.0 [INF] pgmap v886: 7360 pgs: 7360 active+clean; 14419 MB data, 60243 MB used, 7082 GB / 7141 GB avail; 14318 B/s rd, 3095 B/s wr, 19 op/s
2014-08-27 01:11:37.214665 mon.0 [INF] pgmap v887: 7360 pgs: 7360 active+clean; 14419 MB data, 60243 MB used, 7082 GB / 7141 GB avail; 0 B/s rd, 1855 B/s wr, 0 op/s
2014-08-27 01:11:42.214276 mon.0 [INF] pgmap v888: 7360 pgs: 7360 active+clean; 14419 MB data, 60243 MB used, 7082 GB / 7141 GB avail; 0 B/s rd, 2293 B/s wr, 0 op/s

It looks as if the disk was created correctly;

root@node-1:~# rbd info compute/2f0910b2-585d-4023-b6b7-1c029a992ea8_disk
rbd image '2f0910b2-585d-4023-b6b7-1c029a992ea8_disk':
        size 20480 MB in 2560 objects
        order 23 (8192 kB objects)
        block_name_prefix: rbd_data.16952e13c42c
        format: 2
        features: layering
        parent: images/82e58933-0127-4665-8bc7-8e54279ffbd0@snap
        overlap: 4096 MB
root@node-1:~# rbd children images/82e58933-0127-4665-8bc7-8e54279ffbd0@snap
compute/147cfd3c-712a-4f98-94c7-d78a2059a564_disk
compute/2f0910b2-585d-4023-b6b7-1c029a992ea8_disk

I've verified data is correctly on the spawned image as well;

root@node-1:~# rbd map compute/2f0910b2-585d-4023-b6b7-1c029a992ea8_disk
root@node-1:~# mkdir /mnt/linux
root@node-1:~# mount /dev/rbd/compute/2f0910b2-585d-4023-b6b7-1c029a992ea8_disk-part1 /mnt/linux
root@n...

[Tyler Wilson (loth)]

To add on to the previous comment, it appears libvirt errors when it tries to spawn the instance"Caught Segmentation violation dumping internal log buffer" Log is attached.

[Tyler Wilson (loth)]

I got this working (on ubuntu 12.04 by doing the following)

Add
[client]
rbd_cache = true
#rbd_cache_max_dirty = 0

To ceph.conf, enabling rbd_cache_max_dirty resulted in about half the write iops vs disabling
Change #disk_cachemodes="file=directsync","block=none" in nova.conf to
disk_cachemodes=network=writeback,block=writeback
It seemed libvirt/nova was having issues with the quotes and crashing.
restart libvirt-bin, then nova-compute
Both services should come back online and VM's should reflect they are in writeback via the XML

[Vladimir Kuklin (vkuklin)]

Tyler, we are not sure that this change will work with all types of hypervisors we support in FUEL, so I am moving this bug to the next release.

[Dmitry Borodaenko (angdraug)]

Not having rbd cache enabled causes huge performance issues for our users, priority upgraded to High and milestone set back to 6.0.

[Mike Scherbakov (mihgen)]

+1 to Dmitry.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/135960

[Bogdan Dobrelya (bogdando)]

C@Tyler, could you please elaborate why we should use writeback *and* writethrough, which is completely opposite, at once?

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/135960
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=c0730b265fadbcf0a4ba105cddfe35de3516bef4
Submitter: Jenkins
Branch: master

commit c0730b265fadbcf0a4ba105cddfe35de3516bef4
Author: Stanislav Makar <email address hidden>
Date: Thu Nov 20 13:56:12 2014 +0000

    Enable ceph cache

    Add the options rbd cache and rbd cache writethrough until flush to
    client stanza
    Change the option libvirt_disk_cachemodes in nova.conf if ceph is enabled

    Change-Id: I0e7c0fb4556824837fb1aa64d8d13bece00b7b36
    Closes-bug: #1361391

[Mykola Golub (mgolub)]

Verified on

{
    "build_id": "2015-05-19_10-05-51",
    "build_number": "437",
    "auth_required": true,
    "fuel-ostf_sha": "9ce1800749081780b8b2a4a7eab6586583ffaf33",
    "fuel-library_sha": "2814c51668f487e97e1449b078bad1942421e6b9",
    "nailgun_sha": "593c99f2b46cf52b2be6c7c6e182b6ba9f2232cd",
    "openstack_version": "2014.2.2-6.1",
    "production": "docker",
    "api": "1.0",
    "python-fuelclient_sha": "e19f1b65792f84c4a18b5a9473f85ef3ba172fce",
    "astute_sha": "96801c5bccb14aa3f2a0d7f27f4a4b6dd2b4a548",
    "fuelmain_sha": "68796aeaa7b669e68bc0976ffd616709c937187a",
    "feature_groups": [
        "mirantis"
    ],
    "release": "6.1",
    "release_versions": {
        "2014.2.2-6.1": {
            "VERSION": {
                "build_id": "2015-05-19_10-05-51",
                "build_number": "437",
                "fuel-library_sha": "2814c51668f487e97e1449b078bad1942421e6b9",
                "nailgun_sha": "593c99f2b46cf52b2be6c7c6e182b6ba9f2232cd",
                "fuel-ostf_sha": "9ce1800749081780b8b2a4a7eab6586583ffaf33",
                "production": "docker",
                "api": "1.0",
                "python-fuelclient_sha": "e19f1b65792f84c4a18b5a9473f85ef3ba172fce",
                "astute_sha": "96801c5bccb14aa3f2a0d7f27f4a4b6dd2b4a548",
                "fuelmain_sha": "68796aeaa7b669e68bc0976ffd616709c937187a",
                "feature_groups": [
                    "mirantis"
                ],
                "release": "6.1",
                "openstack_version": "2014.2.2-6.1"
            }
        }
    }
}

using Juno on Ubuntu 14.04.1.

The settings are set correctly.