Perfomance degradation for Ceph

[Anastasia Palkina]

"build_id": "2014-11-25_22-41-00",
"ostf_sha": "a35f516f1606b0d03d51ff63bfe3fbe23de4b622",
"build_number": "7",
"auth_required": true,
"api": "1.0",
"nailgun_sha": "cbe7b96943d43397dc608a2f6c9dc1af14dd9a48",
"production": "docker",
"fuelmain_sha": "7db74b9f80180bf3936db1edc4aebfae310d024a",
"astute_sha": "c15623d05ccdf7ac10873e7a90df954de8726280",
"feature_groups": ["mirantis"],
"release": "6.0",
"release_versions": {"2014.2-6.0": {"VERSION": {"build_id": "2014-11-25_22-41-00", "ostf_sha": "a35f516f1606b0d03d51ff63bfe3fbe23de4b622", "build_number": "7", "api": "1.0", "nailgun_sha": "cbe7b96943d43397dc608a2f6c9dc1af14dd9a48", "production": "docker", "fuelmain_sha": "7db74b9f80180bf3936db1edc4aebfae310d024a", "astute_sha": "c15623d05ccdf7ac10873e7a90df954de8726280", "feature_groups": ["mirantis"], "release": "6.0", "fuellib_sha": "8c7eec6225184e0391569b2b5371196ab3e3fa19"}}},
"fuellib_sha": "8c7eec6225184e0391569b2b5371196ab3e3fa19"

1. Create new environment (CentOS, HA mode)
2. Choose GRE neutron
3. Choose Ceph for volumes
4. Choose Sahara, Ceilometer
5. Add 2 controller+ceph, 1 conroller, 1 compute, 3 mongo
6. Start deployment. It has failed with error: Deployment has failed. Method deploy. Upload cirros "TestVM" image failed.

Cause is perfomance degradation for Ceph

[root@node-23 ~]# time /usr/bin/glance image-create --name 'TestVM' --is-public true --container-format='bare' --disk-format='qcow2' --min-ram=64 --property murano_image_info='{"title": "Murano Demo", "type": "cirros.demo"}' --file '/opt/vm/cirros-x86_64-disk.img'+------------------------------+-------------------------------------------------+
| Property | Value |
+------------------------------+-------------------------------------------------+
| Property 'murano_image_info' | {"title": "Murano Demo", "type": "cirros.demo"} |
| checksum | 64d7c1cd2b6f60c92c14662941cb7913 |
| container_format | bare |
| created_at | 2014-11-28T09:24:40 |
| deleted | False |
| deleted_at | None |
| disk_format | qcow2 |
| id | e4cfe1b0-ce5f-49f0-a556-10b1a402ab6c |
| is_public | True |
| min_disk | 0 |
| min_ram | 64 |
| name | TestVM |
| owner | 7f0d7b974c5f45f0900f58d356a05e51 |
| protected | False |
| size | 13167616 |
| status | active |
| updated_at | 2014-11-28T09:25:27 |
| virtual_size | None |
+------------------------------+-------------------------------------------------+

real 1m6.429s
user 0m0.773s
sys 0m0.452s

[root@node-23 ~]# du -sh /opt/vm/cirros-0.3.2-x86_64-disk.img
13M /opt/vm/cirros-0.3.2-x86_64-disk.img

rados bench -p rbd 90 write
http://paste.openstack.org/show/140209/

rados bench -b 32768 -p rbd 90 write
http://paste.openstack.org/show/140221/

[Bogdan Dobrelya (bogdando)]

Please add logs

[Anastasia Palkina (apalkina)]

Logs are here: https://drive.google.com/a/mirantis.com/file/d/0B6SjzarTGFxaV2djc2Jlei1YTkk/view?usp=sharing

I reproduced this bug 2 times.

I have SSD and I'm using VBox. Virtual machines have 1 CPU
Andrey Korolyov says that the reason is the increased load on the controllers

[Bogdan Dobrelya (bogdando)]

According to astute.yaml:
  fqdn: node-23.domain.tld
  role: ceph-osd
  name: node-23
  fqdn: node-23.domain.tld
  role: primary-controller
  name: node-23
  fqdn: node-24.domain.tld
  role: ceph-osd
  name: node-24
  fqdn: node-24.domain.tld
  role: controller
  name: node-24

OSD roles were placed on controller nodes. That is a not recommended configuration and there is a note about that in the docs as well. Especially, then you use 1 vCPU only.

[Vladimir Sharshov (vsharshov)]

Also important notice. After Anastasia Palkina (apalkina) env discovery we found that VirtualBox used hard drive instead of SSD.

As result: HDD, 1 vCPU only and ceph-osd on controller in total give such problem.