Verify network fail when try check public net on compute and ceph

[Dmitry Tyzhnenko]

api: '1.0'
astute_sha: ef8aa0fd0e3ce20709612906f1f0551b5682a6ce
auth_required: true
build_id: 2014-11-27_23-41-13
build_number: '45'
feature_groups:
- mirantis
fuellib_sha: 15a387462f7be50c4f87ad986d0c81535025c125
fuelmain_sha: 51e66db7750e9c856ba128f35cfb6724895bf479
nailgun_sha: 500e36d08a45dbb389bf2bd97673d9bff48ee84d
ostf_sha: 64cb59c681658a7a55cc2c09d079072a41beb346
production: docker
release: 5.1.1
release_versions:
  2014.1.3-5.1.1:
    VERSION:
      api: '1.0'
      astute_sha: ef8aa0fd0e3ce20709612906f1f0551b5682a6ce
      build_id: 2014-11-27_23-41-13
      build_number: '45'
      feature_groups:
      - mirantis
      fuellib_sha: 15a387462f7be50c4f87ad986d0c81535025c125
      fuelmain_sha: 51e66db7750e9c856ba128f35cfb6724895bf479
      nailgun_sha: 500e36d08a45dbb389bf2bd97673d9bff48ee84d
      ostf_sha: 64cb59c681658a7a55cc2c09d079072a41beb346
      production: docker
      release: 5.1.1

Configuration of environment:
Ubuntu HA
Network Neutron GRE, use tags for all networks
Use Ceph for Images
Install Sahara

3 Controller
1 Compute
1 Cinder
2 Ceph

Expected result:
  Verify network pass
  Health check pass

Actual result:
  Verify network fail
  Health check pass

Verify network error:

Node Name Node MAC address Node Interface Expected VLAN (not received)
Untitled (ba:5d) 52:54:00:1f:f2:d2 eth4 677
Untitled (dd:69) 52:54:00:3f:03:44 eth4 677

[Dima Shulyak (dshulyak)]

When configuration for netcheck created public network is not filtered,
I think we can simply add a note to release notes about it, any opinions?

[Aleksey Kasatkin (alekseyk-ru)]

Let's add release note for 6.0 and fix configuration serialization later (in 6.1).

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/169841

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/169841
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=424ae2fa4ae299887fe04f0c61449e92ebd2178e
Submitter: Jenkins
Branch: master

commit 424ae2fa4ae299887fe04f0c61449e92ebd2178e
Author: Aleksey Kasatkin <email address hidden>
Date: Wed Apr 1 19:56:49 2015 +0300

    Consider whether node has public network for network-verify config

    Public VLANs should be added into network-verify configuration only
    for nodes which have br-ex to be configured.

    Closes-Bug: #1397369
    Change-Id: If3ec552b895e5e4f9763c18e23f3a3b7461e20c7

[Aleksey Kasatkin (alekseyk-ru)]

Was reverted https://review.openstack.org/#/c/171237/ due to issue in Astute.

[Aleksey Kasatkin (alekseyk-ru)]

This came into astute:

[663] 9fafd8d5-34b7-4cbf-a3b0-a19edbe062a0: MC agent 'net_probe', method 'get_probing_info', results: {:sender=>"1", :statuscode=>0, :statusmsg=>"OK", :data=>{:neighbours=>{"eth4"=>{"0"=>{"1"=>["eth4"], "3"=>["eth4"], "2"=>["eth4"]}}, "eth2"=>{"0"=>{"1"=>["eth2"], "3"=>["eth2"], "2"=>["eth2"]}}, "eth1"=>{"0"=>{"1"=>["eth1"]}}, "eth0"=>{"0"=>{"1"=>["eth0"], "3"=>["eth0"], "2"=>["eth0"]}}}, :uid=>"1"}}

This came from astute:

{"method"=>"verify_networks_resp", "args"=>{"task_uuid"=>"9fafd8d5-34b7-4cbf-a3b0-a19edbe062a0", "status"=>"ready", "progress"=>100, "nodes"=>[{"uid"=>"3", "networks"=>[{"iface"=>"eth4", "vlans"=>[0]}, {"iface"=>"eth2", "vlans"=>[0]}, {"iface"=>"eth0", "vlans"=>[0]}]}, {"uid"=>"2", "networks"=>[{"iface"=>"eth4", "vlans"=>[0]}, {"iface"=>"eth2", "vlans"=>[0]}, {"iface"=>"eth0", "vlans"=>[0]}]}, {"uid"=>"1", "networks"=>[{"iface"=>"eth4", "vlans"=>[0]}, {"iface"=>"eth2", "vlans"=>[0]}, {"iface"=>"eth1", "vlans"=>[]}, {"iface"=>"eth0", "vlans"=>[0]}]}]}}

Data for node 1 became broken:

{"iface"=>"eth1", "vlans"=>[]}

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/178133

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/178133
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=fa1f569c5b135a1801ebc01b4d8e5c509a8629e7
Submitter: Jenkins
Branch: master

commit fa1f569c5b135a1801ebc01b4d8e5c509a8629e7
Author: Aleksey Kasatkin <email address hidden>
Date: Tue Apr 28 13:43:03 2015 +0300

    Consider whether nodes have public network for net-checker config

    Public VLANs should be added into net-checker configuration only
    for nodes which have br-ex to be configured and only when cluster have
    more than one of such nodes.
    When there is only one such node in cluster, public VLAN cannot be checked
    as there is no sense to run data transmission test within one node.

    Closes-Bug: #1397369
    Change-Id: I4bb17d6f2f797b90a045cb953922518e4d0bfe85

[Dmitry Tyzhnenko (dtyzhnenko)]

Network verification failed when it was trying to check config eth1 interface on controllers

Node Name Node MAC address Node Interface Expected VLAN (not received)
conrt1 (0a:8a) 64:83:87:32:fb:05 eth1 untagged
conrt1 (2b:37) 64:5e:e5:2e:e3:a8 eth1 untagged
conrt1 (4d:d9) 64:2c:db:d9:d7:fd eth1 untagged

custom ISO :
{
    'build_id': '2015-04-28_11-17-08',
    'build_number': '1099',
    'release_versions': {'2014.2.2-6.1': {'VERSION': {
        'build_id': '2015-04-28_11-17-08',
        'build_number': '1099',
        'api': '1.0',
        'fuel-library_sha': '0e5b82d24853304befb22145ac4aaf3545d295e1',
        'nailgun_sha': 'f81b31aa303b4b318e4f588431294453d1fe3ae2',
        'feature_groups': ['mirantis'],
        'openstack_version': '2014.2.2-6.1',
        'production': 'docker',
        'python-fuelclient_sha': '8cd6cf575d3c101dee1032abb6877dfa8487e077',
        'astute_sha': 'c1793f982fda7e3fc7b937ccaa613c649be6a144',
        'fuel-ostf_sha': 'b38602c841deaa03ddffc95c02f319360462cbe3',
        'release': '6.1',
        'fuelmain_sha': 'dcb26e0543d672f82af6aa516c7b80328855f767',
        }}},
    'auth_required': true,
    'api': '1.0',
    'fuel-library_sha': '0e5b82d24853304befb22145ac4aaf3545d295e1',
    'nailgun_sha': 'f81b31aa303b4b318e4f588431294453d1fe3ae2',
    'feature_groups': ['mirantis'],
    'openstack_version': '2014.2.2-6.1',
    'production': 'docker',
    'python-fuelclient_sha': '8cd6cf575d3c101dee1032abb6877dfa8487e077',
    'astute_sha': 'c1793f982fda7e3fc7b937ccaa613c649be6a144',
    'fuel-ostf_sha': 'b38602c841deaa03ddffc95c02f319360462cbe3',
    'release': '6.1',
    'fuelmain_sha': 'dcb26e0543d672f82af6aa516c7b80328855f767',
    }

[deactivateduser (deactivateduser-deactivatedaccount)]

The bug is reproduced when more then one controller is assigned.

[Dmitry Pyzhov (dpyzhov)]

Looks like comment #9 is not related to initial issue. Because original bug was about checking of public networks on nodes without public network.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/181178

[Łukasz Oleś (loles)]

I think this should fix #9 https://review.openstack.org/#/c/181493/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/181493
Committed: https://git.openstack.org/cgit/stackforge/fuel-astute/commit/?id=597361709b2de1928488a94ac0a4b5d3d47f5adf
Submitter: Jenkins
Branch: master

commit 597361709b2de1928488a94ac0a4b5d3d47f5adf
Author: Łukasz Oleś <email address hidden>
Date: Fri May 8 20:26:54 2015 +0200

    Change data sent from netchecher to nailgun

    There are two changes here:
    - remove from the hash data sent by node to itself
    - send to nailgun only non empty data

    In https://review.openstack.org/#/c/178133/3 data sent to net checker changed.
    Before it expected all tags to be recived on all nodes. Now tags for public network
    are only recived on controllers.

    Related-bug: #1397369

    Change-Id: I8389dc4a05a6be6560135747020e74889fe8e18c

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ivan Kliuk (<email address hidden>) on branch: master
Review: https://review.openstack.org/181178
Reason: The problem was fixed here: https://review.openstack.org/#/c/181493/

[Mike Scherbakov (mihgen)]

> Reason: The problem was fixed here: https://review.openstack.org/#/c/181493/
Do we consider this as fixed then?

[deactivateduser (deactivateduser-deactivatedaccount)]

Yes. It needs to be verified.

[Dennis Dmitriev (ddmitriev)]

Fix released:
ISO version: {u'build_id': u'2015-05-13_22-55-26', u'build_number': u'414', u'auth_required': True, u'fuel-ostf_sha': u'1ae384a28b42d5325bd5d2546aead1e9d502f6a8', u'fuel-library_sha': u'f9310aa52628d70b45a8d9cb735dd309f9b556f8', u'nailgun_sha': u'afbcea1d4928e588a8858e154770cd9d031a1fc7', u'openstack_version': u'2014.2.2-6.1', u'production': u'docker', u'api': u'1.0', u'python-fuelclient_sha': u'e4ebbc720c2d5e4546b51758c5796821ed4377f6', u'astute_sha': u'484d5038b81b7260597be027b1265dc88091026b', u'fuelmain_sha': u'1c03fb884d3e79165a2406d00a5ea8679e35f4f7', u'feature_groups': [u'mirantis'], u'release': u'6.1', u'release_versions': {u'2014.2.2-6.1': {u'VERSION': {u'build_id': u'2015-05-13_22-55-26', u'build_number': u'414', u'fuel-library_sha': u'f9310aa52628d70b45a8d9cb735dd309f9b556f8', u'nailgun_sha': u'afbcea1d4928e588a8858e154770cd9d031a1fc7', u'fuel-ostf_sha': u'1ae384a28b42d5325bd5d2546aead1e9d502f6a8', u'production': u'docker', u'api': u'1.0', u'python-fuelclient_sha': u'e4ebbc720c2d5e4546b51758c5796821ed4377f6', u'astute_sha': u'484d5038b81b7260597be027b1265dc88091026b', u'fuelmain_sha': u'1c03fb884d3e79165a2406d00a5ea8679e35f4f7', u'feature_groups': [u'mirantis'], u'release': u'6.1', u'openstack_version': u'2014.2.2-6.1'}}}}