Discovered Nodes Conflict with each other

[Jainesh Nand]

Detailed bug description:
 First Node discovered gets conflicted with second node.

Steps to reproduce:
1) Install Fuel 8.0 or 9.0 on a node. Default Settings.
2) Boot first node through PXE and wait for Fuel to discover.
3) View Details of the Node discovered (MAC Address, Serial Number, etc)
4) Boot second node through PXE and wait for fuel to discover.
5) Fuel doesn't add as second node but rather overwrite the first discovered node.
6) View Details of the Node discovered (MAC Address, Serial Number, etc) - Different from Step 3
7) Continue to wait, and notice that Fuel switches between both nodes details

Expected results:
 Fuel to discover two nodes as separate nodes.

Actual result:
 Discovered as single node and conflicts.

Reproducibility:
 Fuel 8.0 and 9.0

Workaround:
 Nonde

Impact:
 High

Description of the environment:
Dell C6100 - 4 Node Model
Using BIOS 1.71, BMC 1.33
Mellanox ConnectX-2 Dual Port QDR (MHQH29C-XTR) Per Node
SAS2008 RAID Controller (RAID 1 with 300GB)
32GB 1066MHz per Node

Switches: Dell PowerConnect 2724 (BMC/Management), 2 x Dell PowerConnect 5448 (PXE / Public Switch)[VLAN1 for PXE], [VLAN5 for Uplink], 2 x Mellanox IS5035 (Private/Storage)

Additional information:
 Using Fuel Node, the MAC Address and IP Address changes between the first and second discovered node. But always showing 1 discovered node.

Attached Snapshot

[Evgeniy L (rustyrobot)]

From IRC, macs of nodes 00:26:6C:FE:57:4C and 00:26:6C:FE:28:C4

[Evgeniy L (rustyrobot)]

Bug #1, mac must be lowered before attempt to find it in the database

https://github.com/openstack/fuel-web/blob/efdd063170062d1bc881cd31c52bcee08bcdc9ee/nailgun/nailgun/objects/node.py#L219-L220

[Evgeniy L (rustyrobot)]

After it was not able to find by interface, it tries to use other interfaces, and it finds mellanox interface with mac 00:01, which exists in multiple nodes

"interfaces":[{"name":"enp3s0d1","state":"down","mac":"00:00:00:00:00:01","pxe":false,"driver":"mlx4_en","bus_info":"0000:03:00.0","max_speed":10000,"current_speed":null

[Jainesh Nand (jainesh)]

After changes were suggested, same results.

`dockerctl shell nailgun`, and `find /usr/lib --name node.py`
add change mac=meta.get('mac') to mac=meta.get('mac', "").lower() and restart the container.

[Jainesh Nand (jainesh)]

Adding screenshot of changes.

[Evgeniy L (rustyrobot)]

Hmm, actually it's lowered further https://github.com/openstack/fuel-web/blob/stable/8.0/nailgun/nailgun/objects/node.py#L85

[Evgeniy L (rustyrobot)]

But comment https://bugs.launchpad.net/fuel/+bug/1650646/comments/4 is still valid.

[Jainesh Nand (jainesh)]

screenshot of changes number 2

[Jainesh Nand (jainesh)]

snapshot number 3

[Dmitry Sutyagin (dsutyagin)]

I am able to reproduce this bug in a test environment by setting identical macs on two arbitrary interfaces on two nodes. We need to use a different method of node identification instead of MACs, for example system UUID + S/N from ohai output.

[Jainesh Nand (jainesh)]

The UUID is difference on each Nodes in my environment.

The Mellanox Mac Address defaults to 00:00:00:00:00:01 when it is in Ethernet Mode. It would be nice for Mirantis Fuel can have the Mellanox switchable between IB and EN modes.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/414664

[Dmitry Sutyagin (dsutyagin)]

@Jainesh Nand
Please try the proposed fix and let us know if it resolves the issue. You will need to:
1. change /usr/lib/python2.7/site-packages/nailgun/objects/node.py as in the following review https://review.openstack.org/414664
2. restart nailgun (systemctl restart nailgun)
3. reboot bootstrapped nodes

[Dmitry Sutyagin (dsutyagin)]

@Jainesh Nand
The current version of the review (patch set 18) should resolve the issue, however the review is for master branch and might not apply directly on 8.0 or 9.0. If you you feel confident in Python please try to apply the changes similarly on your Fuel system and restart nailgun (if running 8.0 - do not restart the container if running 8.0 - this will revert any changes inside it; only restart nailgun service inside). Otherwise we need to wait for the change to be reviewed, and if approved we will need to make a backport.

[Dmitry Pyzhov (dpyzhov)]

Very invasive change and rare case. Moving out of 9.2 scope.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Dmitry Sutyagin (<email address hidden>) on branch: master
Review: https://review.openstack.org/414664
Reason: Approach disapproved

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/421551

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/421552

[Dmitry Sutyagin (dsutyagin)]

New proposed fix for master: https://review.openstack.org/#/c/421539

[OpenStack Infra (hudson-openstack)]

Change abandoned by Andreas Jaeger (<email address hidden>) on branch: stable/newton
Review: https://review.opendev.org/421552
Reason: This repo is retired now, no further work will get merged.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Andreas Jaeger (<email address hidden>) on branch: master
Review: https://review.opendev.org/421539
Reason: This repo is retired now, no further work will get merged.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Andreas Jaeger (<email address hidden>) on branch: stable/mitaka
Review: https://review.opendev.org/421551
Reason: This repo is retired now, no further work will get merged.