Network verification hangs with network template

Bug #1490536 reported by Artem Panchenko on 2015-08-31
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Sergey Slipushenko
MOS Maintenance

Bug Description

Fuel version info: (7.0 build #253

Sometimes network verification hangs for environment with network template:

# fuel task 2>/dev/null
id | status | name | cluster | progress | uuid
6 | running | check_networks | 1 | 0 | 980864ce-47a6-4728-8d5f-52313a71cf6b
9 | ready | check_networks | 1 | 100 | 74163a26-efed-412a-b103-09f19e3fe7cb

Steps to reproduce:

1. Create environment (NeutronVLAN), add nodes
2. Configure networks using Fuel web interface
3. Run network verification
4. Upload network template for environment
5. Run network verification

Expected result:

- network check is passed or failed


- check_networks task is hanging with running status

Diagnostic snapshot is attached.

Artem Panchenko (apanchenko-8) wrote :
Changed in fuel:
status: New → Confirmed
Dmitry Pyzhov (dpyzhov) on 2015-08-31
Changed in fuel:
assignee: Fuel Python Team (fuel-python) → Ivan Kliuk (ivankliuk)
Ivan Kliuk (ivankliuk) on 2015-09-01
Changed in fuel:
status: Confirmed → In Progress
Dmitry Pyzhov (dpyzhov) on 2015-09-02
Changed in fuel:
milestone: 7.0 → 8.0
Ivan Kliuk (ivankliuk) wrote :

I wasn't able to reproduce it on VirtualBox deployed environment.

{"build_id": "262", "build_number": "262", "release_versions": {"2015.1.0-7.0": {"VERSION": {"build_id": "262", "build_number": "262", "api": "1.0", "fuel-library_sha": "1556601b9b7503285714d7d1e02cc0807b1c68f0", "nailgun_sha": "b564ae20116297750bf6402b3a017e219bf4b468", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "d7027952870a35db8dc52f185bb1158cdd3d1ebd", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "082a47bf014002e515001be05f99040437281a2d", "production": "docker", "python-fuelclient_sha": "9643fa07f1290071511066804f962f62fe27b512", "astute_sha": "e63709d16bd4c1949bef820ac336c9393c040d25", "fuel-ostf_sha": "582a81ccaa1e439a3aec4b8b8f6994735de840f4", "release": "7.0", "fuelmain_sha": "4dc6799370da4cddf06c04e4ecb7646102298535"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "1556601b9b7503285714d7d1e02cc0807b1c68f0", "nailgun_sha": "b564ae20116297750bf6402b3a017e219bf4b468", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "d7027952870a35db8dc52f185bb1158cdd3d1ebd", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "082a47bf014002e515001be05f99040437281a2d", "production": "docker", "python-fuelclient_sha": "9643fa07f1290071511066804f962f62fe27b512", "astute_sha": "e63709d16bd4c1949bef820ac336c9393c040d25", "fuel-ostf_sha": "582a81ccaa1e439a3aec4b8b8f6994735de840f4", "release": "7.0", "fuelmain_sha": "4dc6799370da4cddf06c04e4ecb7646102298535"}

Changed in fuel:
status: In Progress → Incomplete
assignee: Ivan Kliuk (ivankliuk) → Fuel Python Team (fuel-python)

Change abandoned by Ivan Kliuk (<email address hidden>) on branch: master
Reason: Bug's status has been set to 'Incomplete'.

Ksenia Svechnikova (kdemina) wrote :

I've seen the same issue on HW lab MOS 8.0 ISO#130(


* Create VLAN HA env
* Add network-template (fuel
* Run network verification

[root@fuel-lab5-mos6 ~]# date
Thu Oct 8 15:38:20 UTC 2015

[root@fuel-lab5-mos6 ~]# fuel task
id | status | name | cluster | progress | uuid
4 | running | check_networks | 1 | 0 | 06dfc176-2dfa-4026-8737-990b1138e6e2

[root@fuel-lab5-mos6 ~]# date
Thu Oct 8 15:45:37 UTC 2015
[root@fuel-lab5-mos6 ~]# fuel task
id | status | name | cluster | progress | uuid
4 | running | check_networks | 1 | 0 | 06dfc176-2dfa-4026-8737-990b1138e6e2

In logs: [7fb445a13740] (logger) Response code '500 Internal Server Error' for PUT /api/clusters/1/network_configuration/neutron/verify from

Ksenia Svechnikova (kdemina) wrote :

* Add network-template (fuel network-template --env 1 -u)

Changed in fuel:
status: Incomplete → Confirmed
Ksenia Svechnikova (kdemina) wrote :

verify_networks hangs, when we assign some roles to the nodes, that are not described in the network-template. In my example we have nodes "controller, mongo" roles, but in network-template there was no mongo. As soon as it was added it to the template, verification task finish:

      - admin
      - mgmt-cpu
      - public-private-cpu
      - storage-cpu
      - admin
      - mgmt-ctrl
      - public-private-ctrl
      - storage-ctrl
      - admin
      - mgmt-cpu
      - public-private-cpu
      - storage-cpu

Aleksey Kasatkin (alekseyk-ru) wrote :

So, it is lack of validation. It should be verified that roles of existing nodes are the same as defined in template (set of roles on real nodes may be the subset of those defined in a template but cannot be a superset of those).

Aleksey Kasatkin (alekseyk-ru) wrote :

So, all node roles which are in use must be described in the template.

Changed in fuel:
assignee: Fuel Python Team (fuel-python) → Sergey Slipushenko (sslypushenko)
Dmitry Pyzhov (dpyzhov) on 2015-10-22
tags: added: area-python
Changed in fuel:
status: Confirmed → In Progress
Dmitry Pyzhov (dpyzhov) on 2015-11-20
tags: added: team-networks
Dmitry Pyzhov (dpyzhov) on 2015-11-21
tags: added: team-network
removed: team-networks
Changed in fuel:
assignee: Sergey Slipushenko (sslypushenko) → Ivan Kliuk (ivankliuk)
Changed in fuel:
assignee: Ivan Kliuk (ivankliuk) → Sergey Slipushenko (sslypushenko)

Comment from Aleksey Kasatkin

Different roles are usually deployed in different node groups (controllers are always in one node group only). Need to compare node groups one by one.

Seems, this check should be done before network verification or deploy is started. As nodes can be added later, after template is loaded. It's more important than to check this here.

Please look at CheckNetworksTask.execute. NetworkCheck.check_interface_mapping is called there in two situations described above. You could include this check into CheckNetworksTask.execute after check_interface_mapping (and rename check_admin_untagged parameter to smth like check_all_parameters) to run this check when needed.

Submitter: Jenkins
Branch: master

commit 03b7c507b87208631c5732ebef6d8a6e399998e6
Author: sslypushenko <email address hidden>
Date: Thu Nov 19 19:00:11 2015 +0200

    Add validation for templates_for_node_role

    Add check for templates for node role in network template.
    All nodes roles existing in cluster should be present in network template.
    If it is not true InvalidData exception will be raised.

    Change-Id: I6be3c115b59d9f0a17074995d4cea88b31f53ae1
    Closes-Bug: #1490536

Changed in fuel:
status: In Progress → Fix Committed

Verified on iso #427
[root@nailgun ~]# cat /etc/fuel/version.yaml
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "427"
  build_id: "427"
  fuel-nailgun_sha: "9ebbaa0473effafa5adee40270da96acf9c7d58a"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "df16d41cd7a9445cf82ad9fd8f0d53824711fcd8"
  fuel-nailgun-agent_sha: "92ebd5ade6fab60897761bfa084aefc320bff246"
  astute_sha: "c7ca63a49216744e0bfdfff5cb527556aad2e2a5"
  fuel-library_sha: "fae42170a54b98d8e8c8db99b0fbb312633c693c"
  fuel-ostf_sha: "214e794835acc7aa0c1c5de936e93696a90bb57a"
  fuel-mirror_sha: "b62f3cce5321fd570c6589bc2684eab994c3f3f2"
  fuelmenu_sha: "85de57080a18fda18e5325f06eaf654b1b931592"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "9f0ba4577915ce1e77f5dc9c639a5ef66ca45896"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "e8e36cff332644576d7853c80b8a53d5b955420a"

3 controllers and 2 mongo were added to the cluster #1.
Default network template from [1] was uploaded:

[root@nailgun ~]# fuel --env 1 network-template --upload --dir /root
500 Server Error: Internal Server Error (u'mongo')

Network verification can't be started:
"Node role 'mongo' is assigned for some nodes in node group 'default', but not found in network template for this node group"

After template changing(adding):
          - common
          - private
          - storage
new template was uploaded:

[root@nailgun ~]# fuel --env 1 network-template --upload --dir /root
Network template /root/network_template_1.yaml has been uploaded.

and network verification was finished successfully:

"Verification succeeded. Your network is configured correctly."

[1] =

Changed in fuel:
status: Fix Committed → Fix Released
Alexey Stupnikov (astupnikov) wrote :

We no longer support MOS5.1, MOS6.0, MOS6.1
We deliver only Critical/Security fixes to MOS7.0, MOS8.0.
We deliver only High/Critical/Security fixes to MOS9.2.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers