Huge deployment fails with Timeout of deployment is exceeded

[Igor Shishkin]

Tested with both of Ubuntu and CentOS on KVM the following configuration: HA + GRE + Savanna + Murano + Ceph for both of storage backends + 3x dedicated controllers, 2x dedicated computes, 2x dedicated cephs, 2x dedicated cinders.

ISO: fuel-3.2.1-2-2013-11-06_06-39-06
Logs attached, environment still alive.

[Nikolay Fedotov (nfedotov)]

ran into the issue too.
ISO: {"release": "3.2.1", "nailgun_sha": "d4765702c8dd8f5e9f83f3e9a23ca29294a8fece", "ostf_sha": "5440506c81544953c8c21f18b4f732dbe3a5a349", "astute_sha": "df6ddea3abc93fbe1cab9b4534d4d5e9508c95d6", "fuellib_sha": "df58134e47668221d9ab2dae2b4883caa5716774"}

Environment: CentOS HA mode, Neutron with GRE segmentation, 3 controllers, 1 compute, 1 cinder, 3 ceph
deployment status: Deployment has failed. Timeout of deployment is exceeded.

Puppet errors on first controller node:
2013-11-06 16:14:46 ERR
 (/Stage[main]/Quantum::Network::Predefined_netwoks/Quantum_net[net04]/ensure) change from absent to present failed: Execution of '/usr/bin/quantum --os-tenant-name services --os-username quantum --os-password wPaT2lK6 --os-auth-url http://192.168.0.10:35357/v2.0 net-create --tenant_id dc6f7005ebbb4e3cbc6f343fad9a2e45 net04 --provider:network_type gre --provider:segmentation_id 2' returned 1: Unable to create the network. The tunnel ID 2 is in use.
2013-11-06 16:14:32 ERR
 Could not prefetch quantum_net provider 'quantum': Execution of '/usr/bin/quantum --os-tenant-name services --os-username quantum --os-password wPaT2lK6 --os-auth-url http://192.168.0.10:35357/v2.0 net-list' returned 1: 400-{'message': ''}
2013-11-06 15:57:05 ERR
 (/Stage[main]/Quantum::Agents::Dhcp/Service[quantum-dhcp-service]/ensure) change from stopped to running failed: execution expired
2013-11-06 15:44:44 ERR
 Could not prefetch quantum_net provider 'quantum': Execution of '/usr/bin/quantum --os-tenant-name services --os-username quantum --os-password wPaT2lK6 --os-auth-url http://192.168.0.10:35357/v2.0 net-list' returned 1: 400-{'message': ''}

[Igor Shishkin (teran)]

It looks like the same as in PRD-2202

[Sergey Vasilenko (xenolog)]

This log should be read from down to up?
I guess right?

[Igor Shishkin (teran)]

Reproduced on fresh iso fuel-3.2.1-9-2013-11-06_06-39-06

[Sergey Vasilenko (xenolog)]

Please review: https://github.com/Mirantis/fuel/pull/801

[Nastya Urlapova (aurlapova)]

Reproduce on #159 under Ubuntu with HA cluster and nova.
Deploy primary controller require more than 1h in this case timeout for puppet is exceeded.
Please, increase deployment timeout in astute.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/62837

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/62837
Committed: https://git.openstack.org/cgit/stackforge/fuel-astute/commit/?id=c71e087a3bb0f4afbe8685e6614f637630f0c055
Submitter: Jenkins
Branch: master

commit c71e087a3bb0f4afbe8685e6614f637630f0c055
Author: Vladimir <email address hidden>
Date: Wed Dec 18 14:18:49 2013 +0400

    Increase puppet deploy timeout

      -Scenarios became more complicated and require more time to deploy

    Change-Id: Ic297c97dcbe54b14956dfea186189aad001d06a0
    Related-Bug: #1248744