Fuel should correctly process situations when net-provider is not defined at the config

[Alexander Kurenyshev]

Let's modify description and say in other words.
The main point here is that fuel should correctly process situations when network provider is not defined.
We have a variable that is used for enable/disable Neutron at our ceph tests.
When variable is not set there is no word about net-provider at cluster config we try to upload. And fuel should use neutron. But at some reasons it calls nova.
This situations we can see at logs below.

CI System Test job [1] was filed with "AssertionError: Task 'deploy' has incorrect status. error != ready, 'Deployment has failed. Method granular_deploy. Deployment failed on nodes 1.
Inspect Astute logs for the details'"

At the Astute log:
2015-10-06 22:30:46 ERR [642] Error running RPC method granular_deploy: Deployment failed on nodes 1, trace:
["/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/deployment_engine.rb:68:in `block (3 levels) in deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/deployment_engine.rb:51:in `each_slice'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/deployment_engine.rb:51:in `block (2 levels) in deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/deployment_engine.rb:49:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/deployment_engine.rb:49:in `block in deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/deployment_engine.rb:45:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/deployment_engine.rb:45:in `deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/orchestrator.rb:179:in `deploy_cluster'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/orchestrator.rb:45:in `granular_deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/server/dispatcher.rb:91:in `granular_deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/server/server.rb:146:in `dispatch_message'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/server/server.rb:107:in `block in dispatch'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/server/task_queue.rb:64:in `call'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/server/task_queue.rb:64:in `block in each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/server/server.rb:105:in `each_with_index'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/server/server.rb:105:in `dispatch'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/server/server.rb:89:in `block in perform_main_job'"]

At the puppet log from node 1:
Execution of '/usr/bin/nova network-create novanetwork --fixed-range-v4 10.0.0.0/16 --dns1 8.8.4.4 --dns2 8.8.8.8' returned 1: ERROR (ClientException): The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-09ed79c2-395f-4e6e-9071-4caeab952d77)

When I manually executed this command after reverting environment I got the same error 500 and timeout for waiting message at the nova-all.log:
<179>Oct 7 10:39:17 node-1 nova-api 2015-10-07 10:39:17.380 12188 ERROR nova.api.openstack [req-2fc43ae5-b105-45b8-8109-fd2754e35143 ea5cb4d6f1bd40b8a51b575cfbf9e600 b81a978ecfea42568010d2cbc87a6a60 - - -] Caug
ht error: Timed out waiting for a reply to message ID aa435b402b7247c3b682cbde2f509855

For this test we have next steps:
1. Create cluster
2. Add 1 node with controller and ceph OSD roles
3. Add 2 node with compute and ceph OSD roles
4. Deploy the cluster

[1] https://product-ci.infra.mirantis.net/job/8.0.system_test.ubuntu.ceph_ha_one_controller/12/console

Fuel used:

api: '1.0'
astute_sha: 0f753467a3f16e4d46e7e9f1979905fb178e4d5b
auth_required: true
build_id: '126'
build_number: '126'
feature_groups:
- mirantis
fuel-agent_sha: e881f0dabd09af4be4f3e22768b02fe76278e20e
fuel-library_sha: 37734918592cef7d355f357d14cbf960540e09cd
fuel-nailgun-agent_sha: d66f188a1832a9c23b04884a14ef00fc5605ec6d
fuel-nailgun_sha: 70d8b7e80573728e04ac5478c112850afcfa9802
fuel-ostf_sha: 41aa5059243cbb25d7a80b97f8e1060a502b99dd
fuelmain_sha: 51614465980e5f62a5796779d3f6c3305c1d5739
openstack_version: 2015.1.0-8.0
production: docker
python-fuelclient_sha: 56fbd6bad7f60f0944b3845c2db14d0b8cabd4d3
release: '8.0'
release_versions:
  2015.1.0-8.0:
    VERSION:
      api: '1.0'
      astute_sha: 0f753467a3f16e4d46e7e9f1979905fb178e4d5b
      build_id: '126'
      build_number: '126'
      feature_groups:
      - mirantis
      fuel-agent_sha: e881f0dabd09af4be4f3e22768b02fe76278e20e
      fuel-library_sha: 37734918592cef7d355f357d14cbf960540e09cd
      fuel-nailgun-agent_sha: d66f188a1832a9c23b04884a14ef00fc5605ec6d
      fuel-nailgun_sha: 70d8b7e80573728e04ac5478c112850afcfa9802
      fuel-ostf_sha: 41aa5059243cbb25d7a80b97f8e1060a502b99dd
      fuelmain_sha: 51614465980e5f62a5796779d3f6c3305c1d5739
      openstack_version: 2015.1.0-8.0
      production: docker
      python-fuelclient_sha: 56fbd6bad7f60f0944b3845c2db14d0b8cabd4d3
      release: '8.0'

[Alexander Kurenyshev (akurenyshev)]

Another one reproduce is here [1]

[1] https://product-ci.infra.mirantis.net/job/8.0.system_test.ubuntu.thread_3/12/console

[Alexander Kurenyshev (akurenyshev)]

And another at the same job as second one. But test is the different.

[Matthew Mosesohn (raytrac3r)]

Why does this test use novanetwork? It should be on neutron by default for all tests.

[Matthew Mosesohn (raytrac3r)]

Possible regression caused by https://review.openstack.org/#/c/226296/ which was merged a few hours before the ISO https://product-ci.infra.mirantis.net/job/8.0-kilo.all/126/ was built. I propose we try to repeat ceph_ha_one_controller on https://product-ci.infra.mirantis.net/job/8.0-kilo.all/125/

[Denis Egorenko (degorenko)]

Here is upstream bug for this issue: https://bugs.launchpad.net/puppet-nova/+bug/1501767

For 7.0 it is working because of using nova-manage in nova_network provider - it writes to db directly without nova-network service.

[Denis Egorenko (degorenko)]

So, for fuel we need probably separate post-deployment task, which will be executed on CONTROLLER after installing COMPUTE node (at least 1).

[Ivan Berezovskiy (iberezovskiy)]

Problem was introduced because of broken nova providers in upstream module. Denis Egorenko is currently working on them in upstream: https://review.openstack.org/#/c/229548 https://review.openstack.org/#/c/229895/ https://review.openstack.org/#/c/226862/. We are going to test them

[Alexander Kurenyshev (akurenyshev)]

I've updated description.
Let's focus on the problem with default value for network provider when there is no info about it at the provided config

[Matthew Mosesohn (raytrac3r)]

That is nailgun logic then, not Fuel Library

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/234113

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/234113
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=3533ae59bc5c3d25b3599cb9203a6503f4cf03de
Submitter: Jenkins
Branch: master

commit 3533ae59bc5c3d25b3599cb9203a6503f4cf03de
Author: Igor Kalnitsky <email address hidden>
Date: Tue Oct 13 13:17:04 2015 +0300

    Change default network provider to Neutron

    In previous release (Fuel 7.0) nova-network is became deprecated, and
    that means it will be dropped in Fuel 8.0. So let's change default to
    neutron, because IIUC nova-network is already removed from
    fuel-library's master and some swarm tests are broken.

    Closes-Bug: #1503657
    Closes-Bug: #1505258

    Change-Id: I65a0ca2906503cb9c83fd99fddda9a8ee5156b16

[Dmitriy Kruglov (dkruglov)]

Verified on MOS 8.0, build 496. The issue is fixed.

ISO details:
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "496"
  build_id: "496"
  fuel-nailgun_sha: "934c2d11b234193c6ac8e41cd1388df0ed3583e0"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "e40bcfbb1f5518d7a464cc330909cf09b84e7c20"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "b91e96e848057be4acadd10c2b55a6529387fe51"
  fuel-ostf_sha: "ab5fd151fc6c1aa0b35bc2023631b1f4836ecd61"
  fuel-mirror_sha: "351d568fa3b3e4dd062054b91d766aa54d379867"
  fuelmenu_sha: "234cb4cbb30fbd2df00f388c28f31606d9cae15f"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "a43cf96cd9532f10794dce736350bf5bed350e9d"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "6b993b3004e8d97d840b672d6c1d44c320975cd9"