[System Tests] Nailgun returns 400 error when try to update nodes interfaces on old cluster after upgrade

Bug #1440687 reported by Andrey Sledzinskiy
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
Critical
Dmitry Tyzhnenko
6.0.x
Fix Committed
Critical
Dmitry Tyzhnenko

Bug Description

We got 400 error from nailgun trying to update added nodes interfaces:
2015-04-06 05:25:35.442 DEBUG [7fb7b0187740] (logger) Request PUT /api/nodes/interfaces from 10.108.0.1:34124 [{"interfaces": [{"name": "eth0", "driver": null, "state": "up", "mac": "64:e9:b6:58:65:88", "m
ax_speed": null, "current_speed": null, "assigned_networks": [{"id": 1, "name": "fuelweb_admin"}], "type": "ether", "id": 10, "bus_info": null}, {"name": "eth1", "driver": null, "state": "up", "mac": "64:e
7:0e:c2:f8:72", "max_speed": null, "current_speed": null, "assigned_networks": [{"id": 2, "name": "public"}], "type": "ether", "id": 9, "bus_info": null}, {"name": "eth2", "driver": null, "state": "up", "m
ac": "64:1a:e2:a4:8c:b8", "max_speed": null, "current_speed": null, "assigned_networks": [{"id": 3, "name": "management"}], "type": "ether", "id": 8, "bus_info": null}, {"name": "eth3", "driver": null, "st
ate": "up", "mac": "64:b7:64:b4:fa:6e", "max_speed": null, "current_speed": null, "assigned_networks": [{"id": 5, "name": "fixed"}], "type": "ether", "id": 7, "bus_info": null}, {"name": "eth4", "driver":
null, "state": "up", "mac": "64:b5:3e:e3:11:45", "max_speed": null, "current_speed": null, "assigned_networks": [{"id": 4, "name": "storage"}], "type": "ether", "id": 6, "bus_info": null}], "id": 2}]
2015-04-06 05:25:35.445 DEBUG [7fb7b0187740] (logger) Response code '400 Bad Request' for PUT /api/nodes/interfaces from 10.108.0.1:34124

Error Message

HTTP Error 400: Bad Request

Stacktrace

Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/case.py", line 331, in run
    testMethod()
  File "/usr/lib/python2.7/unittest/case.py", line 1043, in runTest
    self._testFunc()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/6.1.system_test.centos.ceph_multinode_compact.upgrade/UPGRADE/fuelweb_test/helpers/decorators.py", line 66, in wrapper
    return func(*args, **kwargs)
  File "/home/jenkins/workspace/6.1.system_test.centos.ceph_multinode_compact.upgrade/UPGRADE/fuelweb_test/tests/test_upgrade.py", line 179, in upgrade_ha_one_controller_delete_node
    cluster_id, {'slave-03': ['compute', 'ceph-osd']}, False, True)
  File "/home/jenkins/workspace/6.1.system_test.centos.ceph_multinode_compact.upgrade/UPGRADE/fuelweb_test/__init__.py", line 48, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/6.1.system_test.centos.ceph_multinode_compact.upgrade/UPGRADE/fuelweb_test/models/fuel_web_client.py", line 773, in update_nodes
    self.update_nodes_interfaces(cluster_id)
  File "/home/jenkins/workspace/6.1.system_test.centos.ceph_multinode_compact.upgrade/UPGRADE/fuelweb_test/__init__.py", line 48, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/6.1.system_test.centos.ceph_multinode_compact.upgrade/UPGRADE/fuelweb_test/models/fuel_web_client.py", line 904, in update_nodes_interfaces
    self.update_node_networks(node['id'], assigned_networks)
  File "/home/jenkins/workspace/6.1.system_test.centos.ceph_multinode_compact.upgrade/UPGRADE/fuelweb_test/__init__.py", line 48, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/6.1.system_test.centos.ceph_multinode_compact.upgrade/UPGRADE/fuelweb_test/models/fuel_web_client.py", line 802, in update_node_networks
    [{'id': node_id, 'interfaces': interfaces}])
  File "/home/jenkins/workspace/6.1.system_test.centos.ceph_multinode_compact.upgrade/UPGRADE/fuelweb_test/__init__.py", line 48, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/6.1.system_test.centos.ceph_multinode_compact.upgrade/UPGRADE/fuelweb_test/helpers/decorators.py", line 102, in wrapped
    response = func(*args, **kwargs)
  File "/home/jenkins/workspace/6.1.system_test.centos.ceph_multinode_compact.upgrade/UPGRADE/fuelweb_test/models/nailgun_client.py", line 185, in put_node_interfaces
    return self.client.put("/api/nodes/interfaces", data)
  File "/home/jenkins/workspace/6.1.system_test.centos.ceph_multinode_compact.upgrade/UPGRADE/fuelweb_test/helpers/http.py", line 83, in put
    return self._open(req)
  File "/home/jenkins/workspace/6.1.system_test.centos.ceph_multinode_compact.upgrade/UPGRADE/fuelweb_test/helpers/http.py", line 92, in _open
    return self._get_response(req)
  File "/home/jenkins/workspace/6.1.system_test.centos.ceph_multinode_compact.upgrade/UPGRADE/fuelweb_test/helpers/http.py", line 109, in _get_response
    return self.opener.open(req)
  File "/usr/lib/python2.7/urllib2.py", line 410, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 523, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 448, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 531, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 400: Bad Request

Revision history for this message
Andrey Sledzinskiy (asledzinskiy) wrote :
Revision history for this message
Aleksey Kasatkin (alekseyk-ru) wrote :

It is intended behaviour for now.
Please see:
https://bugs.launchpad.net/fuel/+bug/1428776
https://bugs.launchpad.net/fuel/+bug/1425901
https://review.openstack.org/169012

It was a mail thread were we agreed to deny interfaces changes during and after deployment.

Revision history for this message
Aleksey Kasatkin (alekseyk-ru) wrote :

from DB dump:

slave-02_compute_ceph-osd ready

Revision history for this message
Aleksey Kasatkin (alekseyk-ru) wrote :

It's all OK for new node #4:

slave-04_compute discover

2015-04-06 05:25:35.327 DEBUG [7fb7b0187740] (logger) Request PUT /api/nodes/interfaces from 10.108.0.1:34122 [{"interfaces": [{"name": "eth0", "driver": null, "state": "up", "mac": "64:5d:52:95:98:38", "max_speed": null, "current_speed": null, "assigned_networks": [{"id": 1, "name": "fuelweb_admin"}], "type": "ether", "id": 20, "bus_info": null}, {"name": "eth1", "driver": null, "state": "up", "mac": "64:57:b0:43:c2:fd", "max_speed": null, "current_speed": null, "assigned_networks": [{"id": 2, "name": "public"}], "type": "ether", "id": 19, "bus_info": null}, {"name": "eth2", "driver": null, "state": "up", "mac": "64:ce:cc:bc:9c:79", "max_speed": null, "current_speed": null, "assigned_networks": [{"id": 3, "name": "management"}], "type": "ether", "id": 18, "bus_info": null}, {"name": "eth3", "driver": null, "state": "up", "mac": "64:51:89:f1:8a:fd", "max_speed": null, "current_speed": null, "assigned_networks": [{"id": 5, "name": "fixed"}], "type": "ether", "id": 17, "bus_info": null}, {"name": "eth4", "driver": null, "state": "up", "mac": "64:0d:7c:10:56:6a", "max_speed": null, "current_speed": null, "assigned_networks": [{"id": 4, "name": "storage"}], "type": "ether", "id": 16, "bus_info": null}], "id": 4}]
2015-04-06 05:25:35.424 DEBUG [7fb7b0187740] (logger) Response code '200 OK' for PUT /api/nodes/interfaces from 10.108.0.1:34122

Revision history for this message
Andrey Sledzinskiy (asledzinskiy) wrote :

So all our tests where we are adding/removing nodes are affected

Changed in fuel:
importance: High → Critical
Revision history for this message
Andrey Sledzinskiy (asledzinskiy) wrote :
Changed in fuel:
assignee: Fuel QA Team (fuel-qa) → Dmitry Tyzhnenko (dtyzhnenko)
status: New → Confirmed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-qa (master)

Fix proposed to branch: master
Review: https://review.openstack.org/171225

Changed in fuel:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-qa (master)

Reviewed: https://review.openstack.org/171225
Committed: https://git.openstack.org/cgit/stackforge/fuel-qa/commit/?id=0dc194be96c39c3010bb27d0b796bc276f2a6a9f
Submitter: Jenkins
Branch: master

commit 0dc194be96c39c3010bb27d0b796bc276f2a6a9f
Author: Dmitry Tyzhnenko <email address hidden>
Date: Tue Apr 7 17:29:42 2015 +0300

    Update interface settings only in updated node

    Since Fuel has denied update interface config in ready nodes we should
    update configuration of interfaces only on bootstrap nodes

    Change-Id: I7d7d5769d20fa75dda2d979b5ec49572b77a23b7
    Closes-bug: #1440687

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-main (stable/6.0)

Fix proposed to branch: stable/6.0
Review: https://review.openstack.org/171721

Revision history for this message
Dmitry Tyzhnenko (dtyzhnenko) wrote :
Changed in fuel:
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-main (stable/6.0)

Reviewed: https://review.openstack.org/171721
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=8bf06bf4ba9b2a7d5e20062d227467f0d923a0bb
Submitter: Jenkins
Branch: stable/6.0

commit 8bf06bf4ba9b2a7d5e20062d227467f0d923a0bb
Author: Dmitry Tyzhnenko <email address hidden>
Date: Wed Apr 8 19:22:04 2015 +0300

    Update interface settings only in updated node

    Since Fuel has denied update interface config in ready nodes we should
    update configuration of interfaces only on bootstrap nodes

    Change-Id: I73c4957e06b459dec9d8dd3c94d7500331f0718e
    Closes-bug: #1440687

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.