Comment 0 for bug 1503657

CI System Test job [1] was filed with "AssertionError: Task 'deploy' has incorrect status. error != ready, 'Deployment has failed. Method granular_deploy. Deployment failed on nodes 1.
Inspect Astute logs for the details'"

At the Astute log:
2015-10-06 22:30:46 ERR [642] Error running RPC method granular_deploy: Deployment failed on nodes 1, trace:
["/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/deployment_engine.rb:68:in `block (3 levels) in deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/deployment_engine.rb:51:in `each_slice'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/deployment_engine.rb:51:in `block (2 levels) in deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/deployment_engine.rb:49:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/deployment_engine.rb:49:in `block in deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/deployment_engine.rb:45:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/deployment_engine.rb:45:in `deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/orchestrator.rb:179:in `deploy_cluster'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/orchestrator.rb:45:in `granular_deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/server/dispatcher.rb:91:in `granular_deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/server/server.rb:146:in `dispatch_message'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/server/server.rb:107:in `block in dispatch'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/server/task_queue.rb:64:in `call'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/server/task_queue.rb:64:in `block in each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/server/server.rb:105:in `each_with_index'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/server/server.rb:105:in `dispatch'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/server/server.rb:89:in `block in perform_main_job'"]

At the puppet log from node 1:
Execution of '/usr/bin/nova network-create novanetwork --fixed-range-v4 10.0.0.0/16 --dns1 8.8.4.4 --dns2 8.8.8.8' returned 1: ERROR (ClientException): The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-09ed79c2-395f-4e6e-9071-4caeab952d77)

When I manually executed this command after reverting environment I got the same error 500 and timeout for waiting message at the nova-all.log:
<179>Oct 7 10:39:17 node-1 nova-api 2015-10-07 10:39:17.380 12188 ERROR nova.api.openstack [req-2fc43ae5-b105-45b8-8109-fd2754e35143 ea5cb4d6f1bd40b8a51b575cfbf9e600 b81a978ecfea42568010d2cbc87a6a60 - - -] Caug
ht error: Timed out waiting for a reply to message ID aa435b402b7247c3b682cbde2f509855

For this test we have next steps:
1. Create cluster
2. Add 1 node with controller and ceph OSD roles
3. Add 2 node with compute and ceph OSD roles
4. Deploy the cluster

[1] https://product-ci.infra.mirantis.net/job/8.0.system_test.ubuntu.ceph_ha_one_controller/12/console

Fuel used:

api: '1.0'
astute_sha: 0f753467a3f16e4d46e7e9f1979905fb178e4d5b
auth_required: true
build_id: '126'
build_number: '126'
feature_groups:
- mirantis
fuel-agent_sha: e881f0dabd09af4be4f3e22768b02fe76278e20e
fuel-library_sha: 37734918592cef7d355f357d14cbf960540e09cd
fuel-nailgun-agent_sha: d66f188a1832a9c23b04884a14ef00fc5605ec6d
fuel-nailgun_sha: 70d8b7e80573728e04ac5478c112850afcfa9802
fuel-ostf_sha: 41aa5059243cbb25d7a80b97f8e1060a502b99dd
fuelmain_sha: 51614465980e5f62a5796779d3f6c3305c1d5739
openstack_version: 2015.1.0-8.0
production: docker
python-fuelclient_sha: 56fbd6bad7f60f0944b3845c2db14d0b8cabd4d3
release: '8.0'
release_versions:
  2015.1.0-8.0:
    VERSION:
      api: '1.0'
      astute_sha: 0f753467a3f16e4d46e7e9f1979905fb178e4d5b
      build_id: '126'
      build_number: '126'
      feature_groups:
      - mirantis
      fuel-agent_sha: e881f0dabd09af4be4f3e22768b02fe76278e20e
      fuel-library_sha: 37734918592cef7d355f357d14cbf960540e09cd
      fuel-nailgun-agent_sha: d66f188a1832a9c23b04884a14ef00fc5605ec6d
      fuel-nailgun_sha: 70d8b7e80573728e04ac5478c112850afcfa9802
      fuel-ostf_sha: 41aa5059243cbb25d7a80b97f8e1060a502b99dd
      fuelmain_sha: 51614465980e5f62a5796779d3f6c3305c1d5739
      openstack_version: 2015.1.0-8.0
      production: docker
      python-fuelclient_sha: 56fbd6bad7f60f0944b3845c2db14d0b8cabd4d3
      release: '8.0'