Discovered Nodes Conflict with each other

Bug #1650646 reported by Jainesh Nand
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
High
Dmitry Sutyagin
Mitaka
High
Dmitry Sutyagin
Newton
High
Dmitry Sutyagin
Ocata
High
Dmitry Sutyagin

Bug Description

Detailed bug description:
 First Node discovered gets conflicted with second node.

Steps to reproduce:
1) Install Fuel 8.0 or 9.0 on a node. Default Settings.
2) Boot first node through PXE and wait for Fuel to discover.
3) View Details of the Node discovered (MAC Address, Serial Number, etc)
4) Boot second node through PXE and wait for fuel to discover.
5) Fuel doesn't add as second node but rather overwrite the first discovered node.
6) View Details of the Node discovered (MAC Address, Serial Number, etc) - Different from Step 3
7) Continue to wait, and notice that Fuel switches between both nodes details

Expected results:
 Fuel to discover two nodes as separate nodes.

Actual result:
 Discovered as single node and conflicts.

Reproducibility:
 Fuel 8.0 and 9.0

Workaround:
 Nonde

Impact:
 High

Description of the environment:
Dell C6100 - 4 Node Model
Using BIOS 1.71, BMC 1.33
Mellanox ConnectX-2 Dual Port QDR (MHQH29C-XTR) Per Node
SAS2008 RAID Controller (RAID 1 with 300GB)
32GB 1066MHz per Node

Switches: Dell PowerConnect 2724 (BMC/Management), 2 x Dell PowerConnect 5448 (PXE / Public Switch)[VLAN1 for PXE], [VLAN5 for Uplink], 2 x Mellanox IS5035 (Private/Storage)

Additional information:
 Using Fuel Node, the MAC Address and IP Address changes between the first and second discovered node. But always showing 1 discovered node.

Attached Snapshot

Revision history for this message
Jainesh Nand (jainesh) wrote :
Jainesh Nand (jainesh)
description: updated
Revision history for this message
Evgeniy L (rustyrobot) wrote :

From IRC, macs of nodes 00:26:6C:FE:57:4C and 00:26:6C:FE:28:C4

Revision history for this message
Evgeniy L (rustyrobot) wrote :
Revision history for this message
Evgeniy L (rustyrobot) wrote :

After it was not able to find by interface, it tries to use other interfaces, and it finds mellanox interface with mac 00:01, which exists in multiple nodes

"interfaces":[{"name":"enp3s0d1","state":"down","mac":"00:00:00:00:00:01","pxe":false,"driver":"mlx4_en","bus_info":"0000:03:00.0","max_speed":10000,"current_speed":null

Changed in fuel:
status: New → Triaged
importance: Undecided → High
milestone: none → 9.2
tags: added: area-python
tags: added: module-nailgun
Revision history for this message
Jainesh Nand (jainesh) wrote :

After changes were suggested, same results.

`dockerctl shell nailgun`, and `find /usr/lib --name node.py`
add change mac=meta.get('mac') to mac=meta.get('mac', "").lower() and restart the container.

Revision history for this message
Jainesh Nand (jainesh) wrote :

Adding screenshot of changes.

Revision history for this message
Evgeniy L (rustyrobot) wrote :
no longer affects: fuel/8.0.x
no longer affects: fuel/mitaka
no longer affects: fuel/newton
Changed in fuel:
status: Triaged → New
milestone: 9.2 → none
Revision history for this message
Evgeniy L (rustyrobot) wrote :
Revision history for this message
Jainesh Nand (jainesh) wrote :

screenshot of changes number 2

Revision history for this message
Jainesh Nand (jainesh) wrote :

snapshot number 3

Changed in fuel:
milestone: none → 9.2
assignee: nobody → Fuel Sustaining (fuel-sustaining-team)
status: New → Confirmed
Revision history for this message
Dmitry Sutyagin (dsutyagin) wrote :

I am able to reproduce this bug in a test environment by setting identical macs on two arbitrary interfaces on two nodes. We need to use a different method of node identification instead of MACs, for example system UUID + S/N from ohai output.

Revision history for this message
Jainesh Nand (jainesh) wrote :

The UUID is difference on each Nodes in my environment.

The Mellanox Mac Address defaults to 00:00:00:00:00:01 when it is in Ethernet Mode. It would be nice for Mirantis Fuel can have the Mellanox switchable between IB and EN modes.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-web (master)

Fix proposed to branch: master
Review: https://review.openstack.org/414664

Changed in fuel:
assignee: Fuel Sustaining (fuel-sustaining-team) → Dmitry Sutyagin (dsutyagin)
status: Confirmed → In Progress
Revision history for this message
Dmitry Sutyagin (dsutyagin) wrote :

@Jainesh Nand
Please try the proposed fix and let us know if it resolves the issue. You will need to:
1. change /usr/lib/python2.7/site-packages/nailgun/objects/node.py as in the following review https://review.openstack.org/414664
2. restart nailgun (systemctl restart nailgun)
3. reboot bootstrapped nodes

Revision history for this message
Dmitry Sutyagin (dsutyagin) wrote :

@Jainesh Nand
The current version of the review (patch set 18) should resolve the issue, however the review is for master branch and might not apply directly on 8.0 or 9.0. If you you feel confident in Python please try to apply the changes similarly on your Fuel system and restart nailgun (if running 8.0 - do not restart the container if running 8.0 - this will revert any changes inside it; only restart nailgun service inside). Otherwise we need to wait for the change to be reviewed, and if approved we will need to make a backport.

Changed in fuel:
assignee: Dmitry Sutyagin (dsutyagin) → Vladimir Kozhukalov (kozhukalov)
Changed in fuel:
assignee: Vladimir Kozhukalov (kozhukalov) → Dmitry Sutyagin (dsutyagin)
Changed in fuel:
assignee: Dmitry Sutyagin (dsutyagin) → Stanislaw Bogatkin (sbogatkin)
Revision history for this message
Dmitry Pyzhov (dpyzhov) wrote :

Very invasive change and rare case. Moving out of 9.2 scope.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-web (master)

Change abandoned by Dmitry Sutyagin (<email address hidden>) on branch: master
Review: https://review.openstack.org/414664
Reason: Approach disapproved

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-web (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/421551

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-web (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/421552

Revision history for this message
Dmitry Sutyagin (dsutyagin) wrote :

New proposed fix for master: https://review.openstack.org/#/c/421539

Changed in fuel:
assignee: Dmitry Sutyagin (dsutyagin) → Georgy Kibardin (gkibardin)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-web (stable/newton)

Change abandoned by Andreas Jaeger (<email address hidden>) on branch: stable/newton
Review: https://review.opendev.org/421552
Reason: This repo is retired now, no further work will get merged.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-web (master)

Change abandoned by Andreas Jaeger (<email address hidden>) on branch: master
Review: https://review.opendev.org/421539
Reason: This repo is retired now, no further work will get merged.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-web (stable/mitaka)

Change abandoned by Andreas Jaeger (<email address hidden>) on branch: stable/mitaka
Review: https://review.opendev.org/421551
Reason: This repo is retired now, no further work will get merged.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers