Platform services functional tests triggers an Error. Stack CREATE FAILED

[Sergii Turivnyi]

ℙ𝕣𝕖𝕔𝕠𝕟𝕕𝕚𝕥𝕚𝕠𝕟𝕤 :
𝐈𝐒𝐎:
api: '1.0'
astute_sha: 0bd72c72369e743376864e8e8dabfe873d40450a
auth_required: true
build_id: 2015-05-24_10-19-44
build_number: '461'
feature_groups:
- mirantis
fuel-library_sha: 889c2534ceadf8afd5d1540c1cadbd913c0c8c14
fuel-ostf_sha: 9a5f55602c260d6c840c8333d8f32ec8cfa65c1f
fuelmain_sha: 5c8ebddf64ea93000af2de3ccdb4aa8bb766ce93
nailgun_sha: 76441596e4fe6420cc7819427662fa244e150177
openstack_version: 2014.2.2-6.1
production: docker
python-fuelclient_sha: e19f1b65792f84c4a18b5a9473f85ef3ba172fce
release: '6.1'
release_versions:
  2014.2.2-6.1:
    VERSION:
      api: '1.0'
      astute_sha: 0bd72c72369e743376864e8e8dabfe873d40450a
      build_id: 2015-05-24_10-19-44
      build_number: '461'
      feature_groups:
      - mirantis
      fuel-library_sha: 889c2534ceadf8afd5d1540c1cadbd913c0c8c14
      fuel-ostf_sha: 9a5f55602c260d6c840c8333d8f32ec8cfa65c1f
      fuelmain_sha: 5c8ebddf64ea93000af2de3ccdb4aa8bb766ce93
      nailgun_sha: 76441596e4fe6420cc7819427662fa244e150177
      openstack_version: 2014.2.2-6.1
      production: docker
      python-fuelclient_sha: e19f1b65792f84c4a18b5a9473f85ef3ba172fce
      release: '6.1'

4 nodes
𝐄𝐧𝐯𝐢𝐫𝐨𝐧𝐦𝐞𝐧𝐭:

Juno on CentOS 6.5 (2014.2.2-6.1)
Compute QEMU
Networking Setup: Neutron with VLAN segmentation
Without Ceph
No additional services

𝕊𝕥𝕖𝕡𝕤 𝕥𝕠 𝕣𝕖𝕡𝕣𝕠𝕕𝕦𝕔𝕖:
1. Add 3 controllers
2. Add 1 compute
3. Hit Deploy changes
    -> Success. Deployment of environment '%env_name%' is done.
4. Run OSTF tests
    -> All OSTF tests have been passed
5. Navigate to the primary-controller (to find it execute 'hiera role' on each controller node)
6. On the primary-controller execute: shutdown -P
7. Wait 5 minutes
8. Run OSTF tests

𝐄𝐱𝐩𝐞𝐜𝐭𝐞𝐝 𝐫𝐞𝐬𝐮𝐥𝐭:
    All OSTF tests have been passed

𝐀𝐜𝐭𝐮𝐚𝐥 𝐫𝐞𝐬𝐮𝐥𝐭:
   Platform services functional tests triggers an Error. Stack CREATE FAILED (see attachments)

http://paste.openstack.org/show/237789/

[Kairat Kushaev (kkushaev)]

It seems that you don't have enough ram or cpu to create the new stack.
Could you please check this version?

[Sergii Turivnyi (sturivnyi)]

Yes? I'm going to check it

On Tue, May 26, 2015 at 5:41 PM, Kairat Kushaev <email address hidden>
wrote:

> It seems that you don't have enough ram or cpu to create the new stack.
> Could you please check this version?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1458887
>
> Title:
> Platform services functional tests triggers an Error. Stack CREATE
> FAILED
>
> Status in Mirantis OpenStack:
> New
>
> Bug description:
> ℙ𝕣𝕖𝕔𝕠𝕟𝕕𝕚𝕥𝕚𝕠𝕟𝕤 :
> 𝐈𝐒𝐎:
> api: '1.0'
> astute_sha: 0bd72c72369e743376864e8e8dabfe873d40450a
> auth_required: true
> build_id: 2015-05-24_10-19-44
> build_number: '461'
> feature_groups:
> - mirantis
> fuel-library_sha: 889c2534ceadf8afd5d1540c1cadbd913c0c8c14
> fuel-ostf_sha: 9a5f55602c260d6c840c8333d8f32ec8cfa65c1f
> fuelmain_sha: 5c8ebddf64ea93000af2de3ccdb4aa8bb766ce93
> nailgun_sha: 76441596e4fe6420cc7819427662fa244e150177
> openstack_version: 2014.2.2-6.1
> production: docker
> python-fuelclient_sha: e19f1b65792f84c4a18b5a9473f85ef3ba172fce
> release: '6.1'
> release_versions:
> 2014.2.2-6.1:
> VERSION:
> api: '1.0'
> astute_sha: 0bd72c72369e743376864e8e8dabfe873d40450a
> build_id: 2015-05-24_10-19-44
> build_number: '461'
> feature_groups:
> - mirantis
> fuel-library_sha: 889c2534ceadf8afd5d1540c1cadbd913c0c8c14
> fuel-ostf_sha: 9a5f55602c260d6c840c8333d8f32ec8cfa65c1f
> fuelmain_sha: 5c8ebddf64ea93000af2de3ccdb4aa8bb766ce93
> nailgun_sha: 76441596e4fe6420cc7819427662fa244e150177
> openstack_version: 2014.2.2-6.1
> production: docker
> python-fuelclient_sha: e19f1b65792f84c4a18b5a9473f85ef3ba172fce
> release: '6.1'
>
> 4 nodes
> 𝐄𝐧𝐯𝐢𝐫𝐨𝐧𝐦𝐞𝐧𝐭:
>
> Juno on CentOS 6.5 (2014.2.2-6.1)
> Compute QEMU
> Networking Setup: Neutron with VLAN segmentation
> Without Ceph
> No additional services
>
> 𝕊𝕥𝕖𝕡𝕤 𝕥𝕠 𝕣𝕖𝕡𝕣𝕠𝕕𝕦𝕔𝕖:
> 1. Add 3 controllers
> 2. Add 1 compute
> 3. Hit Deploy changes
> -> Success. Deployment of environment '%env_name%' is done.
> 4. Run OSTF tests
> -> All OSTF tests have been passed
> 5. Navigate to the primary-controller (to find it execute 'hiera role'
> on each controller node)
> 6. On the primary-controller execute: shutdown -P
> 7. Wait 5 minutes
> 8. Run OSTF tests
>
> 𝐄𝐱𝐩𝐞𝐜𝐭𝐞𝐝 𝐫𝐞𝐬𝐮𝐥𝐭:
> All OSTF tests have been passed
>
> 𝐀𝐜𝐭𝐮𝐚𝐥 𝐫𝐞𝐬𝐮𝐥𝐭:
> Platform services functional tests triggers an Error. Stack CREATE
> FAILED (see attachments)
>
> http://paste.openstack.org/show/237789/
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/mos/+bug/1458887/+subscriptions
>

--
Thanks a lot in advance.
Kind Regards, Sergey Turivnyi

[Kairat Kushaev (kkushaev)]

Folks,
it seems that nova generates No Valid host found because of some reason and returns it to Heat.
If you observe nova logs you can see the generated exception here:
http://paste.openstack.org/show/237976/.
Heat cannot do anything with these exceptions and just fails a stack. So I suggest for nova guys to analyze this issue.

[Roman Podoliaka (rpodolyaka)]

Nova could not spawn instances due to problem with binding VIFs by Neutron: http://paste.openstack.org/show/240272/

All events for this particular instance uuid: http://paste.openstack.org/show/240286/

Looks like neutron agents didn't become available after RabbitMQ cluster failover ... or failover just didn't complete in time before tests were run: I see multiple `rabbit on node 'rabbit@node-1' down` in rabbitmq logs during the time instances were launched.

Step "7. Wait 5 minutes" should really be "Wait until RabbitMQ cluster is reassembled properly.", which can possibly take more than 5 minutes.

[Mike Scherbakov (mihgen)]

Can't we understand from logs whether Rabbit was already assembled at the time of OSTF run? Do we still have living env, so we could attempt to run OSTF one more time?

[Roman Podoliaka (rpodolyaka)]

I took a closer look at RabbitMQ logs: it seems that cluster failover finished fairly fast (at 11:31 node-1 goes away, at 11:39 cluster was fully operational with 2 nodes left). However for some reason neutron-server continues to think all agents are down, despite the fact they are up and have successfully reconnected to other RabbitMQ nodes. Looks like neutron-server ignores agent state reports for some reason.

[Roman Podoliaka (rpodolyaka)]

mos-neutron, could you guys please take a look at this as well?

[Oleg Bondarev (obondarev)]

Sure, downloading snapshot... Will come up with the analysis shortly

[Oleg Bondarev (obondarev)]

I'm afraid I don't have much to add to Roman's observation: server doesn't see agents while on agent side everything looks ok.
It's really hard to investigate more without debug logs or env with repro. Hence marking it incomplete for now

[Timur Nurlygayanov (tnurlygayanov)]

Sergii, we need to reproduce the issue. Now we changed the status to invalid. If you will reproduce the issue please feel free to change status to Confimed and do not forget to add logs and some comments with detailed steps to reproduce.

[Sergii Turivnyi (sturivnyi)]

Accepted

On Fri, Jul 10, 2015 at 4:12 PM, Timur Nurlygayanov <
<email address hidden>> wrote:

> Sergii, we need to reproduce the issue. Now we changed the status to
> invalid. If you will reproduce the issue please feel free to change
> status to Confimed and do not forget to add logs and some comments with
> detailed steps to reproduce.
>
> ** Changed in: mos
> Assignee: MOS Neutron (mos-neutron) => Sergii (sturivnyi)
>
> ** Changed in: mos
> Status: Incomplete => Invalid
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1458887
>
> Title:
> Platform services functional tests triggers an Error. Stack CREATE
> FAILED
>
> Status in Mirantis OpenStack:
> Invalid
>
> Bug description:
> ℙ𝕣𝕖𝕔𝕠𝕟𝕕𝕚𝕥𝕚𝕠𝕟𝕤 :
> 𝐈𝐒𝐎:
> api: '1.0'
> astute_sha: 0bd72c72369e743376864e8e8dabfe873d40450a
> auth_required: true
> build_id: 2015-05-24_10-19-44
> build_number: '461'
> feature_groups:
> - mirantis
> fuel-library_sha: 889c2534ceadf8afd5d1540c1cadbd913c0c8c14
> fuel-ostf_sha: 9a5f55602c260d6c840c8333d8f32ec8cfa65c1f
> fuelmain_sha: 5c8ebddf64ea93000af2de3ccdb4aa8bb766ce93
> nailgun_sha: 76441596e4fe6420cc7819427662fa244e150177
> openstack_version: 2014.2.2-6.1
> production: docker
> python-fuelclient_sha: e19f1b65792f84c4a18b5a9473f85ef3ba172fce
> release: '6.1'
> release_versions:
> 2014.2.2-6.1:
> VERSION:
> api: '1.0'
> astute_sha: 0bd72c72369e743376864e8e8dabfe873d40450a
> build_id: 2015-05-24_10-19-44
> build_number: '461'
> feature_groups:
> - mirantis
> fuel-library_sha: 889c2534ceadf8afd5d1540c1cadbd913c0c8c14
> fuel-ostf_sha: 9a5f55602c260d6c840c8333d8f32ec8cfa65c1f
> fuelmain_sha: 5c8ebddf64ea93000af2de3ccdb4aa8bb766ce93
> nailgun_sha: 76441596e4fe6420cc7819427662fa244e150177
> openstack_version: 2014.2.2-6.1
> production: docker
> python-fuelclient_sha: e19f1b65792f84c4a18b5a9473f85ef3ba172fce
> release: '6.1'
>
> 4 nodes
> 𝐄𝐧𝐯𝐢𝐫𝐨𝐧𝐦𝐞𝐧𝐭:
>
> Juno on CentOS 6.5 (2014.2.2-6.1)
> Compute QEMU
> Networking Setup: Neutron with VLAN segmentation
> Without Ceph
> No additional services
>
> 𝕊𝕥𝕖𝕡𝕤 𝕥𝕠 𝕣𝕖𝕡𝕣𝕠𝕕𝕦𝕔𝕖:
> 1. Add 3 controllers
> 2. Add 1 compute
> 3. Hit Deploy changes
> -> Success. Deployment of environment '%env_name%' is done.
> 4. Run OSTF tests
> -> All OSTF tests have been passed
> 5. Navigate to the primary-controller (to find it execute 'hiera role'
> on each controller node)
> 6. On the primary-controller execute: shutdown -P
> 7. Wait 5 minutes
> 8. Run OSTF tests
>
> 𝐄𝐱𝐩𝐞𝐜𝐭𝐞𝐝 𝐫𝐞𝐬𝐮𝐥𝐭:
> All OSTF tests have been passed
>
> 𝐀𝐜𝐭𝐮𝐚𝐥 𝐫𝐞𝐬𝐮𝐥𝐭:
> Platform services functional tests triggers an Error. Stack CREATE
> FAILED (see attachments)
>
> http://paste.openstack.org/show/237789/
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/mos/+bug/1458887/+subscriptions
>

--
Thanks a lot in advance.
Kind Regards, Sergey Turivnyi