Murano OSTF check fail

[Bogdan Dobrelya]

Murano OSTF fails with Fuel 4.0.
env: iso #202 {"build_id": "2013-12-25_01-01-41", "ostf_sha": "a27665008e961b773861b95969acf61d25f0f977", "build_number": "202", "nailgun_sha": "d4aecc4f1b60d66021bac195800952fb94cbc627", "fuelmain_sha": "15f575db2d35e3374a217e29a38091152801cfe5", "astute_sha": "9000b12aa0f3639bd4a18efeb82ef554fdbde9c0", "release": "4.0", "fuellib_sha": "cc0301c5f0399e7567230d5cf261f0796005d91c"}

Testing was prepared according to http://docs.mirantis.com/fuel/fuel-4.0/user-guide.html#platform-tests-description

(see ostf* and murano* logs for errors)

Additional info:
Looks like the issue is related to flavor settings OSTF use for launching testing instances.
I've managed to spawn IIS service in Win2012std Murano environment manually, only with flavor of 1/1/50/50 (cpu/RAM/root/ephemeral), and failed with 1/1/40/40 and 1/1/30/30 as well (nova-compute error:Instance type's disk is too small for requested image).
Note, free disk space consumed at /var/lib/nova (42G total) max was not more than 60% all the time.

[Timur Nurlygayanov (tnurlygayanov)]

Bogdan, what the 'correct' flavor we can use for Windows instances? Can we use m1.large for it?

[Bogdan Dobrelya (bogdando)]

I've managed to spawn IIS service in Win2012std Murano environment manually only with flavor of 1/1/50/50 (cpu/RAM/root/ephemeral), and failed with 1/1/40/40 and 1/1/30/30 as well (nova-compute error:Instance type's disk is too small for requested image)

[Timur Nurlygayanov (tnurlygayanov)]

I can not reproduce this issue with the standard 'm1.medium' flavor with 2 vCPU, 40 GB root disk and 4 GB of RAM and OSTF tests use this flavor for all Windows-based services. The one equirement for Windows VM is minimum 40GB of hard disk and 2 GB of RAM.

It will be very helpfull if we can collect logs from Murano/Heat and Nova to find the root of this problem. The additional fix to Murano OSTF tests to collect more information about FAILED tests: https://review.openstack.org/74322

Also, Bogdan, can you please recheck this issue for the latest Fuel 4.1 with Murano 0.4.1, using the updated user manual: https://github.com/stackforge/fuel-docs/blob/master/pages/user-guide/post-install-healthchecks.rst#preparing-murano-for-testing

I can help you with Windows Server 2012 image and with logs analysis.

[Bogdan Dobrelya (bogdando)]

Here is a logs snapshot from #211 ISO. Murano OSTF tests failed, there was also the errors mentioned in https://bugs.launchpad.net/fuel/+bug/1272217 found (Could not decode response body as JSON).

This time I cannot manually spawn IIS service in Win2012std neither with m1.medium, nor 1/1/50/50 (test_murano custom flavor)

[Bogdan Dobrelya (bogdando)]

After several retries, I've managed to manually spawn IIS service again, see details below (still no luck with OSTF tho)

{"build_id": "2014-02-26_13-39-45", "mirantis": "yes", "build_number": "211", "nailgun_sha": "ea08cef3e06a72f47cfaa8cd8fe6d034e2cf722e", "ostf_sha": "8e6681b6d06c7cb20a84c1cc740d5f2492fb9d85", "fuelmain_sha": "baa8bb07393698f1186cb67bb65f1b93907c59bd", "astute_sha": "10cccc87f2ee35510e43c8fa19d2bf916ca1fced", "release": "4.1", "fuellib_sha": "0a2e5bdc01c1e3bb285acb7b39125101e950ac72"}

Reproduce:
* Deploy Centos simple, Neutron with GRE, Murano and Ceilometer, Cinder LVM for volumes, Ceph RBD for images;
nodes: 1 controller, 1 compute, 1 cinder, 2 ceph
* Prepare Murano OSTF tests https://github.com/stackforge/fuel-docs/blob/master/pages/user-guide/post-install-healthchecks.rst#preparing-murano-for-testing
* Manually create Murano environment and deploy Win2012std IIS service using 1 vcpu/1G ram/50G disk/50G ephemeral/10000MB swap (murano_test) flavor.

[Timur Nurlygayanov (tnurlygayanov)]

Is it means that we should just change m1.medium flavor to m1.large ?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/77113

[Bogdan Dobrelya (bogdando)]

Probably we should:
1) For Murano OSTF based on KVM CI tests, use custom flavor 2/2/50/50/10000 and increase compute node powers approriately in devops templates.
2) For Murano OSTF based on BM labs, no comments because I've not tested Murano OSTF for BM case.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/4.1
Review: https://review.openstack.org/77116

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/81746

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/81746
Committed: https://git.openstack.org/cgit/stackforge/fuel-ostf/commit/?id=f6f25d85c2f8505bb4b234bc0b06a0b47eb0141f
Submitter: Jenkins
Branch: master

commit f6f25d85c2f8505bb4b234bc0b06a0b47eb0141f
Author: TimurNurlygayanov <email address hidden>
Date: Thu Mar 20 13:39:02 2014 +0400

    Fixed issue with failed OSTF tests

    Change-Id: Ife17c274fed3be2fbfdb6f83a39fef2b311ca280
    Related-Bug: #1264292

[Vladimir Kuklin (vkuklin)]

backport to 4.1.1, please

[Vladimir Kuklin (vkuklin)]

backport to stable/4.1 please

[Timur Nurlygayanov (tnurlygayanov)]

Vladimir, how we can backport this fix to 4.1.1?

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/4.1
Review: https://review.openstack.org/86247

[Openstack Gerrit (openstack-gerrit)]

Reviewed: https://review.openstack.org/86247
Committed: https://git.openstack.org/cgit/stackforge/fuel-ostf/commit/?id=3690709d95c1f42506b6d955b44221910cec8527
Submitter: Jenkins
Branch: stable/4.1

commit 3690709d95c1f42506b6d955b44221910cec8527
Author: Sergey Murashov <email address hidden>
Date: Wed Apr 9 11:27:27 2014 +0400

    Fixed issue with failed OSTF tests

    Change-Id: I428ece5d7e0cda6c7c8eb91623018d95b9e4998d
    Related-Bug: #1264292

[Tatyanka (tatyana-leontovich)]

can not verify on 127 iso - murano service do not start - [root@node-1 ~]# /etc/init.d/openstack-murano-api status
murano-api dead but subsys locked
[root@node-1 ~]#

[Tatyanka (tatyana-leontovich)]

should be re-verified after https://bugs.launchpad.net/fuel/+bug/1311590 will be fixed

[Timur Nurlygayanov (tnurlygayanov)]

Need to retest.