Incorrect order of eth* during bootstrap

[Sergey Galkin]

api: '1.0'
astute_sha: c15623d05ccdf7ac10873e7a90df954de8726280
auth_required: true
build_id: 2014-11-26_13-35-04
build_number: '10'
feature_groups:
- mirantis
fuellib_sha: 25eb629f3c6a6ff41cf187e260fe4ff456cfc4e4
fuelmain_sha: 465afb6479a0b3c677040fb978cc109dcf62f774
nailgun_sha: bf9ddb9f9d5dbb09c4b50201ce176635791d7d3e
ostf_sha: a35f516f1606b0d03d51ff63bfe3fbe23de4b622
production: docker
release: '6.0'

Steps to reproduce
1. Deploy CentOS on the same hw 99 nodes (SUPERMICRO SUPERSERVER 5037MR-H8TRF MICRO-CLOUD (with 2 Ethernet ports) + AOC-STGN-i2S - 2-port 10 Gigabit Ethernet SFP+)

1 node from 99 bootsrtaped node has incorrect interfaces order (eth0 and eth1 on the AOC-STGN-i2S Ethernet interfaces and eth2 and eth3 on internal interfaces)

In my case this is compute_52 (node-77)
Logs from Jenkins job

00:43:12.721 2014-11-28 16:01:02,986 - python.http - DEBUG - PUT [{'interfaces': [{u'name': u'eth0', u'state': u'up', u'mac': u'00:25:90:e3:37:2a', u'max_speed': 1000, u'current_speed': 1000, u'assigned_networks': [{u'id': 2, u'name': u'public'}], u'type': u'ether', u'id': 60}, {u'name': u'eth1', u'state': u'up', u'mac': u'00:25:90:e3:37:2b', u'max_speed': 1000, u'current_speed': 1000, u'assigned_networks': [], u'type': u'ether', u'id': 59}, {u'name': u'eth2', u'state': u'up', u'mac': u'0c:c4:7a:1d:f7:ea', u'max_speed': 10000, u'current_speed': 10000, u'assigned_networks': [{u'id': 1, u'name': u'fuelweb_admin'}, {u'id': 3, u'name': u'management'}, {u'id': 4, u'name': u'storage'}], u'type': u'ether', u'id': 58}, {u'name': u'eth3', u'state': u'up', u'mac': u'0c:c4:7a:1d:f7:eb', u'max_speed': 10000, u'current_speed': 10000, u'assigned_networks': [], u'type': u'ether', u'id': 57}], 'id': 15}] to http://172.16.44.11:8000/api/nodes/interfaces
=== Incorrect node ====
00:43:12.825 2014-11-28 16:01:03,090 - python.http - DEBUG - PUT [{'interfaces': [{u'name': u'eth0', u'state': u'up', u'mac': u'0c:c4:7a:1e:a8:58', u'max_speed': 10000, u'current_speed': 10000, u'assigned_networks': [{u'id': 2, u'name': u'public'}], u'type': u'ether', u'id': 308}, {u'name': u'eth1', u'state': u'up', u'mac': u'0c:c4:7a:1e:a8:59', u'max_speed': 10000, u'current_speed': 10000, u'assigned_networks': [], u'type': u'ether', u'id': 307}, {u'name': u'eth2', u'state': u'up', u'mac': u'0c:c4:7a:14:25:36', u'max_speed': 1000, u'current_speed': 1000, u'assigned_networks': [{u'id': 1, u'name': u'fuelweb_admin'}, {u'id': 3, u'name': u'management'}, {u'id': 4, u'name': u'storage'}], u'type': u'ether', u'id': 306}, {u'name': u'eth3', u'state': u'up', u'mac': u'0c:c4:7a:14:25:37', u'max_speed': 1000, u'current_speed': 1000, u'assigned_networks': [], u'type': u'ether', u'id': 305}], 'id': 77}] to http://172.16.44.11:8000/api/nodes/interfaces
===================
00:43:12.957 2014-11-28 16:01:03,222 - python.http - DEBUG - PUT [{'interfaces': [{u'name': u'eth0', u'state': u'up', u'mac': u'0c:c4:7a:14:30:96', u'max_speed': 1000, u'current_speed': 1000, u'assigned_networks': [{u'id': 2, u'name': u'public'}], u'type': u'ether', u'id': 292}, {u'name': u'eth1', u'state': u'up', u'mac': u'0c:c4:7a:14:30:97', u'max_speed': 1000, u'current_speed': 1000, u'assigned_networks': [], u'type': u'ether', u'id': 291}, {u'name': u'eth2', u'state': u'up', u'mac': u'0c:c4:7a:1e:41:d2', u'max_speed': 10000, u'current_speed': 10000, u'assigned_networks': [{u'id': 1, u'name': u'fuelweb_admin'}, {u'id': 3, u'name': u'management'}, {u'id': 4, u'name': u'storage'}], u'type': u'ether', u'id': 290}, {u'name': u'eth3', u'state': u'up', u'mac': u'0c:c4:7a:1e:41:d3', u'max_speed': 10000, u'current_speed': 10000, u'assigned_networks': [], u'type': u'ether', u'id': 289}], 'id': 73}]

[Dima Shulyak (dshulyak)]

Well, i thought we fixed it, but apparantly there is more issues. Workaround is:

1. Delete cobbler system
  dockerctl shell cobbler
  cobbler system remove --name node-77

2. reboot node

[Sergey Galkin (sgalkin)]

snapshot - https://drive.google.com/open?id=0B1LFPizteKUEZEtiNmR2SVhZOVk

[Dima Shulyak (dshulyak)]

It seems to me such issue is possible only in case of manual intervantion, i mean by using api directly in deployment
script. I think that it happens because you assume that every network are configured on same interface for every node.
So, probably, there is some kind of loop that assigns networks for each node on same interfaces.

But we can not guarantee that network cards will be discovered in same order on each boot, without udev rules