BVT fail on nodes discovery step: MAC addresses of fuel-devops nodes don't match MAC addresses of nailgun nodes

Bug #1508986 reported by Artem Panchenko
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
Critical
Artem Panchenko
7.0.x
Fix Released
High
Artem Panchenko

Bug Description

Fuel version info: http://paste.openstack.org/show/477169/

BVT and other system tests fail on nodes discovery step because MAC addresses of fuel-devops nodes don't match MAC addresses of nailgun nodes:

ERROR: Deploy ceph HA with RadosGW for objects
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/8.0-kilo.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 80, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/8.0-kilo.ubuntu.bvt_2/fuelweb_test/tests/test_ceph.py", line 461, in ceph_rados_gw
    self.env.d_env.nodes().slaves[:6])
  File "/home/jenkins/workspace/8.0-kilo.ubuntu.bvt_2/fuelweb_test/models/environment.py", line 131, in bootstrap_nodes
    wait(lambda: all(self.nailgun_nodes(devops_nodes)), 15, timeout)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/helpers/helpers.py", line 94, in wait
    raise TimeoutError(timeout_msg)
TimeoutError: Waiting timed out

2015-10-22 13:35:40,562 - DEBUG fuel_web_client.py:834 -- Look for nailgun node by macs set([u'64:B0:DF:CC:E9:C0', u'64:3A:F9:FF:6D:36', u'64:22:8C:AB:5B:D5', u'64:C7:2B:48:CF:53', u'64:96:A6:CE:9F:E8'])
2015-10-22 13:35:40,562 - DEBUG fuel_web_client.py:837 -- Look for macs returned by nailgun set([u'64:d8:57:f4:65:73', u'64:36:ed:d2:26:3f', u'64:3a:54:47:ec:2e', u'64:38:67:48:b4:2e', u'64:80:a5:8f:f4:39'])
2015-10-22 13:35:40,562 - DEBUG fuel_web_client.py:837 -- Look for macs returned by nailgun set([u'64:62:0a:94:14:c5', u'64:f9:35:05:94:1f', u'64:02:88:26:0a:32', u'64:7d:ac:44:d7:e4', u'64:97:7d:6d:b1:4b'])
2015-10-22 13:35:40,562 - DEBUG fuel_web_client.py:837 -- Look for macs returned by nailgun set([u'64:b0:df:cc:e9:c0', u'64:3a:f9:ff:6d:36', u'64:22:8c:ab:5b:d5', u'64:c7:2b:48:cf:53', u'64:96:a6:ce:9f:e8'])
2015-10-22 13:35:40,562 - DEBUG fuel_web_client.py:837 -- Look for macs returned by nailgun set([u'64:35:4a:8d:a1:d1', u'64:c3:c0:53:06:22', u'64:50:72:64:ce:b5', u'64:df:8c:92:a3:c8', u'64:f7:b6:18:e8:9b'])
2015-10-22 13:35:40,562 - DEBUG fuel_web_client.py:837 -- Look for macs returned by nailgun set([u'64:57:80:6e:9b:4a', u'64:f5:72:13:9e:71', u'64:34:47:b8:88:d5', u'64:bf:ad:c3:05:22', u'64:59:fd:91:c1:74'])
2015-10-22 13:35:40,562 - DEBUG fuel_web_client.py:837 -- Look for macs returned by nailgun set([u'64:c7:4f:4b:70:82', u'64:b0:87:79:0a:cc', u'64:ef:39:e9:1d:1b', u'64:b8:db:4e:e2:7d', u'64:f5:21:ba:9e:b3'])

This issue is caused by the following patch to nailgun-agent:

https://review.openstack.org/#/c/223991/

Previously it always returned MAC in upper case, but currently they are reported by agent in lower case.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-nailgun-agent (master)

Fix proposed to branch: master
Review: https://review.openstack.org/238568

Changed in fuel:
assignee: Fuel Python Team (fuel-python) → Artem Panchenko (apanchenko-8)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/238587

Changed in fuel:
assignee: Artem Panchenko (apanchenko-8) → Michael Polenchuk (mpolenchuk)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-qa (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/238591

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-qa (master)

Reviewed: https://review.openstack.org/238591
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=bd5e806553a89b41b97eae71b703d97b66623214
Submitter: Jenkins
Branch: master

commit bd5e806553a89b41b97eae71b703d97b66623214
Author: Artem Panchenko <email address hidden>
Date: Thu Oct 22 18:51:45 2015 +0300

    Improve MAC comparasion for node lookup

    Change-Id: I60666959c92b30c04c0e70745f66dac53458f0df
    Related-bug: #1508986

Revision history for this message
Ksenia Svechnikova (kdemina) wrote :

The fix to fuel-qa should be also back ported to 7.0 branch, as the main commit (https://review.openstack.org/#/c/223991/) would be backported to 7.0 as part of the MU1

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-nailgun-agent (master)

Change abandoned by Artem Panchenko (<email address hidden>) on branch: master
Review: https://review.openstack.org/238568

Changed in fuel:
status: In Progress → Fix Committed
assignee: Michael Polenchuk (mpolenchuk) → Artem Panchenko (apanchenko-8)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-qa (stable/7.0)

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/238854

Changed in fuel:
status: Fix Committed → Fix Released
Revision history for this message
Nastya Urlapova (aurlapova) wrote :

@Ksenia, please at the next time be more carefull with verification such changes https://review.openstack.org/#/c/223991/ if you see from product's review that we should align test - do it at first priority.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-nailgun-agent (master)

Change abandoned by Michael Polenchuk (<email address hidden>) on branch: master
Review: https://review.openstack.org/238587
Reason: outdated

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-qa (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/238909

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-qa (stable/7.0)

Related fix proposed to branch: stable/7.0
Review: https://review.openstack.org/238910

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-qa (master)

Reviewed: https://review.openstack.org/238909
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=f0a5cf0901fe5fd76af35b2d5721fdbc04414176
Submitter: Jenkins
Branch: master

commit f0a5cf0901fe5fd76af35b2d5721fdbc04414176
Author: Artem Panchenko <email address hidden>
Date: Fri Oct 23 15:49:51 2015 +0300

    Add 'netaddr' Python module to requirements

    Change-Id: Ia101814eb7db806d58d690ca2ee11d25339ca66a
    Related-bug: #1508986

Dmitry Pyzhov (dpyzhov)
tags: added: area-qa
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-qa (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/239978

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-qa (stable/7.0)

Related fix proposed to branch: stable/7.0
Review: https://review.openstack.org/239989

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-qa (stable/7.0)

Reviewed: https://review.openstack.org/238910
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=deb8edce37f8ebbb066326e72198ff99ba5626d4
Submitter: Jenkins
Branch: stable/7.0

commit deb8edce37f8ebbb066326e72198ff99ba5626d4
Author: Artem Panchenko <email address hidden>
Date: Fri Oct 23 15:49:51 2015 +0300

    Add 'netaddr' Python module to requirements

    Change-Id: Ia101814eb7db806d58d690ca2ee11d25339ca66a
    Related-bug: #1508986

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-qa (stable/7.0)

Reviewed: https://review.openstack.org/238854
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=15c1467e68b7b9c381a1863b342927e4181042ae
Submitter: Jenkins
Branch: stable/7.0

commit 15c1467e68b7b9c381a1863b342927e4181042ae
Author: Artem Panchenko <email address hidden>
Date: Thu Oct 22 18:51:45 2015 +0300

    Improve MAC comparison for node lookup

    Change-Id: I60666959c92b30c04c0e70745f66dac53458f0df
    Closes-bug: #1508986
    (cherry picked from commit bd5e806553a89b41b97eae71b703d97b66623214)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-qa (master)

Reviewed: https://review.openstack.org/239978
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=9356a1fa0cbc1bcf1be935d8eeb3a0707e605251
Submitter: Jenkins
Branch: master

commit 9356a1fa0cbc1bcf1be935d8eeb3a0707e605251
Author: Artem Panchenko <email address hidden>
Date: Wed Oct 28 18:06:24 2015 +0200

    Improve MAC comparison in system tests

    Change-Id: I78745dfbca1e850121bbfb8733f9cd355bf2c0c1
    Closes-bug: #1511003
    Related-bug: #1508986

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-qa (stable/7.0)

Reviewed: https://review.openstack.org/239989
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=d456cbbbdf9f27fb336977cff2089ae795e3d549
Submitter: Jenkins
Branch: stable/7.0

commit d456cbbbdf9f27fb336977cff2089ae795e3d549
Author: Artem Panchenko <email address hidden>
Date: Wed Oct 28 18:06:24 2015 +0200

    Improve MAC comparison in system tests

    Change-Id: I78745dfbca1e850121bbfb8733f9cd355bf2c0c1
    Closes-bug: #1511003
    Related-bug: #1508986

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-qa (stable/6.1)

Related fix proposed to branch: stable/6.1
Review: https://review.openstack.org/348219

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-qa (stable/6.1)

Change abandoned by Fuel DevOps Robot (<email address hidden>) on branch: stable/6.1
Review: https://review.openstack.org/348219
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.