[nailgun] 'NodeBondInterface' object has no attribute 'ip_addr'

[Frank J. Cameron]

Whenever I configure the interfaces for one of my node types, the Fuel UI becomes completely nonresponsive for that cluster until I remove the node from the cluster from the CLI. From the nailgun api.log:

2014-06-09 15:45:53.341 DEBUG [7ffe521fc700] (logger) Request GET /api/nodes?cluster_id=1&_=1401891941175 from 172.16.106.127:44297
2014-06-09 15:45:53.489 ERROR [7ffe521fc700] (logger) Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/web/application.py", line 239, in process
    return self.handle()
  File "/usr/lib/python2.6/site-packages/web/application.py", line 230, in handle
    return self._delegate(fn, self.fvars, args)
  File "/usr/lib/python2.6/site-packages/web/application.py", line 420, in _delegate
    return handle_class(cls)
  File "/usr/lib/python2.6/site-packages/web/application.py", line 396, in handle_class
    return tocall(*args)
  File "<string>", line 2, in GET
  File "/usr/lib/python2.6/site-packages/nailgun/api/handlers/base.py", line 95, in content_json
    data = func(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/nailgun/api/handlers/node.py", line 87, in GET
    return self.collection.to_json(nodes)
  File "/usr/lib/python2.6/site-packages/nailgun/objects/base.py", line 312, in to_json
    iterable=iterable
  File "/usr/lib/python2.6/site-packages/nailgun/objects/base.py", line 295, in to_list
    use_iterable
  File "/usr/lib/python2.6/site-packages/nailgun/objects/base.py", line 294, in <lambda>
    lambda o: cls.single.to_dict(o, fields=fields),
  File "/usr/lib/python2.6/site-packages/nailgun/objects/node.py", line 600, in to_dict
    networks_grouped.get(instance.cluster_id, [])
  File "/usr/lib/python2.6/site-packages/nailgun/network/manager.py", line 625, in get_node_networks_optimized
    network_data.append(cls._get_admin_network(node_db))
  File "/usr/lib/python2.6/site-packages/nailgun/network/manager.py", line 869, in _get_admin_network
    'dev': node.admin_interface.name
  File "/usr/lib/python2.6/site-packages/nailgun/db/sqlalchemy/models/node.py", line 229, in admin_interface
    ip_addr = interface.ip_addr
AttributeError: 'NodeBondInterface' object has no attribute 'ip_addr'
2014-06-09 15:45:53.490 ERROR [7ffe521fc700] (logger) Response code '500 Internal Server Error' for GET /api/nodes?cluster_id=1&_=1401891941175 from 172.16.106.127:44297

The nodes in question have a quad-port card eth{0,1,2,3} and dual-port on-board eth{4,5}. The systems PXE boot on eth4 and eth{0,1,2} are added to an LACP bond on which all other VLANs are tagged. As soon as the interface configuration is submitted, the problem appears.

The other nodes have quad-port on-board eth{0,1,2,3}. Those systems PXE boot on eth3 and eth{0,1,2} are added to an LACP bond on which all other VLANs are tagged. Submitting the interface configuration for these nodes has no problems.

[Dima Shulyak (dshulyak)]

Hi, what version of fuel you used?

Also please attach diagnostic snapshot

[Frank J. Cameron (fjc)]

VERSION:
  mirantis: "yes"
  production: "docker"
  release: "5.0"
  api: "1.0"
  build_number: "26"
  build_id: "2014-05-27_05-51-41"
  astute_sha: "a7eac46348dc77fc2723c6fcc3dbc66cc1a83152"
  fuellib_sha: "2f79c0415159651fc1978d99bd791079d1ae4a06"
  ostf_sha: "a8b7660082a6f152794c610d6abe30d360fd577d"
  nailgun_sha: "bd09f89ef56176f64ad5decd4128933c96cb20f4"
  fuelmain_sha: "505741e4f431f85a8d0252fc42754d10c0326c1a"

[Frank J. Cameron (fjc)]

Uploading the diagnostic snapshot timed-out; I'll try again later. Thanks.

[Dima Shulyak (dshulyak)]

i think it will fail every time we select bond interface for master network

[Dima Shulyak (dshulyak)]

Can you describe order of operations that lead to such issue?

Did you created bonds with fuel-cli?
Or created bonds, then removed node from cluster, and it fails when you are trying to add it back again?

[Frank J. Cameron (fjc)]

1. Installed fuel
2. Using the web UI
3. Added 3 controller/osd nodes (4-nic models)
4. Added 4 compute nodes (two 4-nic, two 6-nic)
5. Configured osd storage
6. Configured interfaces for 4-nic models
    * Create bond eth{0,1,2}
    * Set type to LACP
    * Save and return to node list
7. Configured interfaces for 6-nic models
    * Create bond eth{0,1,2}
    * Set type to LACP
    * Save and return to node list
8. The node list page does not load
9. Remove the two 6-nic nodes using CLI
10. The node list page does load
11. Re-add one or both 6-nic node(s)
12. The node list page does not load

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/100935

[Frank J. Cameron (fjc)]

The patch has improved things:

Before:

1. Add 6-nic nodes
2. Configure bonding
3. Web UI node list doesn't load anymore
4. Remove nodes from CLI
5. Web UI node list loads again
6. Re-add the nodes
7. Web UI node list doesn't load anymore

After:

1. Add 6-nic nodes
2. Configure bonding
3. Web UI node list doesn't load anymore
4. Remove nodes from CLI
5. Web UI node list loads again
6. Re-add the nodes
7. Web UI node list still loads
8. Configure bonding
9. Web UI node list doesn't load anymore

nailgun api log:

2014-06-18 16:00:23.151 DEBUG [7f5e6ebfd700] (logger) Request GET /api/nodes?cluster_id=1&_=1402930426364 from 172.17.42.1:47692
2014-06-18 16:00:23.417 DEBUG [7f5e6ebfd700] (logger) Response code '200 OK' for GET /api/nodes?cluster_id=1&_=1402930426364 from 172.17.42.1:47692
2014-06-18 16:00:28.621 DEBUG [7f5e84dfa700] (logger) Request PUT /api/nodes/6/interfaces from 172.17.42.1:47693 [{"type":"bond","name":"ovs-bond0","mode":"active-backup","assigned_networks":[{"id":8,"name":"public"},{"id":10,"name":"storage"},{"id":9,"name":"management"}],"slaves":[{"name":"eth0"},{"name":"eth1"},{"name":"eth2"}]},{"name":"eth0","state":"up","mac":"a0:36:9f:2a:44:20","max_speed":1000,"current_speed":1000,"assigned_networks":[],"type":"ether","id":114},{"name":"eth1","state":"up","mac":"a0:36:9f:2a:44:21","max_speed":1000,"current_speed":1000,"assigned_networks":[],"type":"ether","id":113},{"name":"eth2","state":"up","mac":"a0:36:9f:2a:44:22","max_speed":1000,"current_speed":1000,"assigned_networks":[],"type":"ether","id":112},{"name":"eth3","state":"down","mac":"a0:36:9f:2a:44:23","max_speed":1000,"current_speed":null,"assigned_networks":[],"type":"ether","id":111},{"name":"eth4","state":"up","mac":"f0:1f:af:d8:ec:02","max_speed":1000,"current_speed":1000,"assigned_networks":[{"id":1,"name":"fuelweb_admin"}],"type":"ether","id":110},{"name":"eth5","state":"down","mac":"f0:1f:af:d8:ec:03","max_speed":1000,"current_speed":null,"assigned_networks":[],"type":"ether","id":109}]
2014-06-18 16:00:28.626 DEBUG [7f5e857fb700] (logger) Request PUT /api/nodes/7/interfaces from 172.17.42.1:47694 [{"type":"bond","name":"ovs-bond0","mode":"active-backup","assigned_networks":[{"id":8,"name":"public"},{"id":10,"name":"storage"},{"id":9,"name":"management"}],"slaves":[{"name":"eth0"},{"name":"eth1"},{"name":"eth2"}]},{"name":"eth0","state":"up","mac":"a0:36:9f:2a:40:90","max_speed":1000,"current_speed":1000,"assigned_networks":[],"type":"ether","id":120},{"name":"eth1","state":"up","mac":"a0:36:9f:2a:40:91","max_speed":1000,"current_speed":1000,"assigned_networks":[],"type":"ether","id":119},{"name":"eth2","state":"up","mac":"a0:36:9f:2a:40:92","max_speed":1000,"current_speed":1000,"assigned_networks":[],"type":"ether","id":118},{"name":"eth3","state":"down","mac":"a0:36:9f:2a:40:93","max_speed":1000,"current_speed":null,"assigned_networks":[],"type":"ether","id":117},{"name":"eth4","state":"up","mac":"f0:1f:af:d8:fa:82","max_speed":1000,"current_speed":1000,"assigned_networks":[{"id":1,"name":"fuelweb_admin"}],"type":"ether","id":116},{"name":"eth5","state":"down","mac":"f0:1f:af:d8:fa:83","max_speed":1000,"current_speed":null,"assigned_networks":[],"type":"e...

[Dima Shulyak (dshulyak)]

You cherry-picked only top patch? Or applied all chain (3 patches) ?

[Frank J. Cameron (fjc)]

Ah yes, I had only grabbed 'Remove bonds from configuration on node unnassign' not 'Ensure all nested attributes loaded once' nor 'Improve get_node_networks in network manager'.

[Frank J. Cameron (fjc)]

I had to make a couple of minor changes to apply the 3 patches cleanly to my version, but the problem appears to be resolved. Thanks.

The issues with applying the patch:
* api/v1/... vs api/...
* api/serializers vs objects/serializers
* two failed hunks

[Dima Shulyak (dshulyak)]

I will backport all this changes to 5.0 after they got merged into master.
So you will be be able to use 5.0.1/5.1 or upgrade fuel master to this versions

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/5.0
Review: https://review.openstack.org/102478

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/100935
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=3de47c11a0a31e5eedc0af547ca8439f70ff33aa
Submitter: Jenkins
Branch: master

commit 3de47c11a0a31e5eedc0af547ca8439f70ff33aa
Author: Dima Shulyak <email address hidden>
Date: Wed Jun 18 18:02:33 2014 +0300

    Remove bonds from configuration on node unnassign

    - Bonds should be removed from configuration when node
      unnasigned from cluster
    - If no admin network group is found, lookup admin interface
      by cidr only on node.nic_interfaces

    This bug partially fixed by previous refactoring

    Change-Id: Ibafab7d2aa2a3377a6c184a689a2607b4953ee09
    Closes-Bug: #1328163

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/102478
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=a5fed3fea6aa42306694d83a65cc05559ef08d0d
Submitter: Jenkins
Branch: stable/5.0

commit a5fed3fea6aa42306694d83a65cc05559ef08d0d
Author: Dima Shulyak <email address hidden>
Date: Wed Jun 18 18:02:33 2014 +0300

    Remove bonds from configuration on node unnassign

    - Bonds should be removed from configuration when node
      unnasigned from cluster
    - If no admin network group is found, lookup admin interface
      by cidr only on node.nic_interfaces

    This bug partially fixed by previous refactoring

    Change-Id: Ibafab7d2aa2a3377a6c184a689a2607b4953ee09
    Closes-Bug: #1328163

[Artem Panchenko (apanchenko-8)]

Verified on iso # 170 (5.0.1)