Network verification hangs with network template

[Artem Panchenko]

Fuel version info: (7.0 build #253 http://paste.openstack.org/show/434689/)

Sometimes network verification hangs for environment with network template:

# fuel task 2>/dev/null
id | status | name | cluster | progress | uuid
---|---------|-------------------------|---------|----------|-------------------------------------
6 | running | check_networks | 1 | 0 | 980864ce-47a6-4728-8d5f-52313a71cf6b
9 | ready | check_networks | 1 | 100 | 74163a26-efed-412a-b103-09f19e3fe7cb

Steps to reproduce:

1. Create environment (NeutronVLAN), add nodes
2. Configure networks using Fuel web interface
3. Run network verification
4. Upload network template for environment
5. Run network verification

Expected result:

- network check is passed or failed

Actual:

- check_networks task is hanging with running status

Diagnostic snapshot is attached.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/219600

[deactivateduser (deactivateduser-deactivatedaccount)]

I wasn't able to reproduce it on VirtualBox deployed environment.

{"build_id": "262", "build_number": "262", "release_versions": {"2015.1.0-7.0": {"VERSION": {"build_id": "262", "build_number": "262", "api": "1.0", "fuel-library_sha": "1556601b9b7503285714d7d1e02cc0807b1c68f0", "nailgun_sha": "b564ae20116297750bf6402b3a017e219bf4b468", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "d7027952870a35db8dc52f185bb1158cdd3d1ebd", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "082a47bf014002e515001be05f99040437281a2d", "production": "docker", "python-fuelclient_sha": "9643fa07f1290071511066804f962f62fe27b512", "astute_sha": "e63709d16bd4c1949bef820ac336c9393c040d25", "fuel-ostf_sha": "582a81ccaa1e439a3aec4b8b8f6994735de840f4", "release": "7.0", "fuelmain_sha": "4dc6799370da4cddf06c04e4ecb7646102298535"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "1556601b9b7503285714d7d1e02cc0807b1c68f0", "nailgun_sha": "b564ae20116297750bf6402b3a017e219bf4b468", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "d7027952870a35db8dc52f185bb1158cdd3d1ebd", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "082a47bf014002e515001be05f99040437281a2d", "production": "docker", "python-fuelclient_sha": "9643fa07f1290071511066804f962f62fe27b512", "astute_sha": "e63709d16bd4c1949bef820ac336c9393c040d25", "fuel-ostf_sha": "582a81ccaa1e439a3aec4b8b8f6994735de840f4", "release": "7.0", "fuelmain_sha": "4dc6799370da4cddf06c04e4ecb7646102298535"}

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ivan Kliuk (<email address hidden>) on branch: master
Review: https://review.openstack.org/219600
Reason: Bug's status has been set to 'Incomplete'.

[Ksenia Svechnikova (kdemina)]

I've seen the same issue on HW lab MOS 8.0 ISO#130(https://paste.mirantis.net/show/1240/)

Steps:

* Create VLAN HA env
* Add network-template (fuel
* Run network verification

[root@fuel-lab5-mos6 ~]# date
Thu Oct 8 15:38:20 UTC 2015

[root@fuel-lab5-mos6 ~]# fuel task
id | status | name | cluster | progress | uuid
---|---------|----------------|---------|----------|-------------------------------------
4 | running | check_networks | 1 | 0 | 06dfc176-2dfa-4026-8737-990b1138e6e2

[root@fuel-lab5-mos6 ~]# date
Thu Oct 8 15:45:37 UTC 2015
[root@fuel-lab5-mos6 ~]# fuel task
id | status | name | cluster | progress | uuid
---|---------|----------------|---------|----------|-------------------------------------
4 | running | check_networks | 1 | 0 | 06dfc176-2dfa-4026-8737-990b1138e6e2

In logs: [7fb445a13740] (logger) Response code '500 Internal Server Error' for PUT /api/clusters/1/network_configuration/neutron/verify from 172.16.68.16:45947

[Ksenia Svechnikova (kdemina)]

* Add network-template (fuel network-template --env 1 -u)

[Ksenia Svechnikova (kdemina)]

verify_networks hangs, when we assign some roles to the nodes, that are not described in the network-template. In my example we have nodes "controller, mongo" roles, but in network-template there was no mongo. As soon as it was added it to the template, verification task finish:

templates_for_node_role:
      compute:
      - admin
      - mgmt-cpu
      - public-private-cpu
      - storage-cpu
      controller:
      - admin
      - mgmt-ctrl
      - public-private-ctrl
      - storage-ctrl
      mongo:
      - admin
      - mgmt-cpu
      - public-private-cpu
      - storage-cpu

[Aleksey Kasatkin (alekseyk-ru)]

So, it is lack of validation. It should be verified that roles of existing nodes are the same as defined in template (set of roles on real nodes may be the subset of those defined in a template but cannot be a superset of those).

[Aleksey Kasatkin (alekseyk-ru)]

So, all node roles which are in use must be described in the template.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/247687

[Sergey Slipushenko (sslypushenko)]

Comment from Aleksey Kasatkin

Different roles are usually deployed in different node groups (controllers are always in one node group only). Need to compare node groups one by one.

Seems, this check should be done before network verification or deploy is started. As nodes can be added later, after template is loaded. It's more important than to check this here.

Please look at CheckNetworksTask.execute. NetworkCheck.check_interface_mapping is called there in two situations described above. You could include this check into CheckNetworksTask.execute after check_interface_mapping (and rename check_admin_untagged parameter to smth like check_all_parameters) to run this check when needed.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/247687
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=03b7c507b87208631c5732ebef6d8a6e399998e6
Submitter: Jenkins
Branch: master

commit 03b7c507b87208631c5732ebef6d8a6e399998e6
Author: sslypushenko <email address hidden>
Date: Thu Nov 19 19:00:11 2015 +0200

    Add validation for templates_for_node_role

    Add check for templates for node role in network template.
    All nodes roles existing in cluster should be present in network template.
    If it is not true InvalidData exception will be raised.

    Change-Id: I6be3c115b59d9f0a17074995d4cea88b31f53ae1
    Closes-Bug: #1490536

[Dmitry Belyaninov (dbelyaninov)]

Verified on iso #427
[root@nailgun ~]# cat /etc/fuel/version.yaml
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "427"
  build_id: "427"
  fuel-nailgun_sha: "9ebbaa0473effafa5adee40270da96acf9c7d58a"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "df16d41cd7a9445cf82ad9fd8f0d53824711fcd8"
  fuel-nailgun-agent_sha: "92ebd5ade6fab60897761bfa084aefc320bff246"
  astute_sha: "c7ca63a49216744e0bfdfff5cb527556aad2e2a5"
  fuel-library_sha: "fae42170a54b98d8e8c8db99b0fbb312633c693c"
  fuel-ostf_sha: "214e794835acc7aa0c1c5de936e93696a90bb57a"
  fuel-mirror_sha: "b62f3cce5321fd570c6589bc2684eab994c3f3f2"
  fuelmenu_sha: "85de57080a18fda18e5325f06eaf654b1b931592"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "9f0ba4577915ce1e77f5dc9c639a5ef66ca45896"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "e8e36cff332644576d7853c80b8a53d5b955420a"

3 controllers and 2 mongo were added to the cluster #1.
Default network template from [1] was uploaded:

[root@nailgun ~]# fuel --env 1 network-template --upload --dir /root
500 Server Error: Internal Server Error (u'mongo')

Network verification can't be started:
"Node role 'mongo' is assigned for some nodes in node group 'default', but not found in network template for this node group"

After template changing(adding):
        mongo:
          - common
          - private
          - storage
new template was uploaded:

[root@nailgun ~]# fuel --env 1 network-template --upload --dir /root
Network template /root/network_template_1.yaml has been uploaded.

and network verification was finished successfully:

"Verification succeeded. Your network is configured correctly."

[1] = https://github.com/openstack/fuel-qa/tree/master/fuelweb_test/network_templates

[Alexey Stupnikov (astupnikov)]

We no longer support MOS5.1, MOS6.0, MOS6.1
We deliver only Critical/Security fixes to MOS7.0, MOS8.0.
We deliver only High/Critical/Security fixes to MOS9.2.