Configuration of offloading types weren't applied for all nodes

[Alexander Zatserklyany]

Steps to reproduce
------------------
1. Deploy Fuel Master Node
2. Create new cluster with 5 slaves(controller, compute and cinder). Use Neutron with VLAN segmentation with 5 NIC architecture.
3. Switch on one offloading type for certain physical interface all node using Web UI
4. Run Network check
5. Deploy cluster
6. Run OSTF tests
7. Connect to testable node
8. Check offloading type for certain physical interface

Expected results:
-----------------
Offloading type for certain physical interface configured on all nodes

Actual results:
-----------------
Offloading type for certain physical interface configured only on node-1

[Bug Checker Bot (bug-checker)]

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

version

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

[Alexander Zatserklyany (zatserklyany)]

Version: fuel-9.0-mos-495-2016-06-16_18-18-00

[Julia Aranovich (jkirnosova)]

Steps 3 sounds unclear. We have Offloading Mode setting for interfaces in Fuel UI. Do you mean this setting?
What particular value did you set up fo the setting before deployment?
Could you please check if offloading mass-configuration was applied to all the nodes before a deployment? Please provide screenshots if possible.
Which particular values offloading had after deployment for all the nodes?

[Alexander Zatserklyany (zatserklyany)]

Step 3:
1) Check upper [Select All] at the Nodes tab page (See Step3-1.png);
2) Click [Interface Configuration] (See Step3-2.png);
3) Check enp0s3 (Admin PXE) and enp0s5 (Management) interfaces;
Note: Management interface must be tagged. See picture Step3-3.png;
4) Click [Bound];
5) Click [Apply] at the bottom of the page.

[Dmitriy Kruglov (dkruglov)]

Occurrences (constantly) on CI swarm runs:
https://product-ci.infra.mirantis.net/job/10.0.system_test.ubuntu.offloading/76/testReport/
https://product-ci.infra.mirantis.net/job/10.0.system_test.ubuntu.bonding_ha_one_controller/76/testReport/(root)/offloading_bond_neutron_vlan/offloading_bond_neutron_vlan/
https://product-ci.infra.mirantis.net/job/10.0.system_test.ubuntu.bonding_ha_one_controller/76/testReport/(root)/offloading_bond_neutron_vxlan/offloading_bond_neutron_vxlan/

[Ksenia Svechnikova (kdemina)]

I've inversigare the problem.

The necessary offloading type has appropriate status

root@node-1:~# ethtool --show-offload enp0s6 | grep generic-receive-offload
generic-receive-offload: off

https://product-ci.infra.mirantis.net/view/10.0_swarm/job/10.0.system_test.ubuntu.offloading/77/console
In Trace we can see the same , that enp0s6 is off on compute node:
AssertionError: Offload type generic-receive-offload is generic-receive-offload: off on slave-02_compute

The problem is in result parameter:

https://github.com/openstack/fuel-qa/blob/master/fuelweb_test/helpers/checkers.py#L1160

https://github.com/openstack/fuel-qa/blob/master/fuelweb_test/tests/test_offloading_types.py#L239

It's not just "off". it's "generic-receive-offload: off". And they are not equal

Assume that it's not the UI problem, but Nailgun. Reassign issue and add swarm-blocker tag

Additional I can see that after the deploy in /etc/astute.yaml on node-2

[Ksenia Svechnikova (kdemina)]

Two more failed test: https://product-ci.infra.mirantis.net/view/10.0_swarm/job/10.0.system_test.ubuntu.offloading/77/testReport/(root)/offloading_neutron_vlan/offloading_neutron_vlan/

https://product-ci.infra.mirantis.net/view/10.0_swarm/job/10.0.system_test.ubuntu.offloading/77/testReport/(root)/offloading_neutron_vxlan/offloading_neutron_vxlan/

[Alexander Zatserklyany (zatserklyany)]

About "It's not just "off". it's "generic-receive-offload: off""
It's not true, because https://github.com/openstack/fuel-qa/blob/master/fuelweb_test/helpers/checkers.py#L1160 returns just "off":
[root@nailgun ~]# ssh node-1 ethtool --show-offload enp0s3 | awk '/generic-receive-offload/ {{print $2}}'
off

and in log:
assert_equal(result, "off", "Offload type {0} is {1} on {2}".format(name, result, node['name']))

{1} is result ("off")

[Ksenia Svechnikova (kdemina)]

Check the CI response:
AssertionError: Offload type generic-receive-offload is "generic-receive-offload: off" on slave-02_compute <=>

                result = check_offload(node['ip'], iface2, name)
                assert_equal(result, "on",
                             "Offload type {0} is {1} on {2}".format(
                                     name, result, node['name']))

where result from test is "generic-receive-offload: off"

As you can see the test also want to get off from this offloading type, but after all is ok we write AssertionError: Offload type generic-receive-offload is "generic-receive-offload: off" on slave-02_compute

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/336468
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=60d575f25cb92d928cfb2430181693989d288c62
Submitter: Jenkins
Branch: master

commit 60d575f25cb92d928cfb2430181693989d288c62
Author: zatserklyany <email address hidden>
Date: Thu Jun 30 16:32:32 2016 +0300

    Fix offloading checker

    Modify verification of applyed changes

    Change-Id: Ia9188e4654bafc3189964707bc4d2516765ebc4f
    Closes-Bug: 1594788

[Alexander Zatserklyany (zatserklyany)]

1. Deploy Fuel Master Node
2. Create new cluster with 5 slaves(controller, compute and cinder). Use Neutron with VLAN segmentation with 5 NIC architecture.

Note, next all steps do Using Web UI

3. Select tab page "Networks", select Settings "Other" at "Public network assignment" check [Assign public network to all nodes] and click [Save Settings]
See Pic-1.png
4. Select tab page "Nodes", check [Select All] and click [Configure interfaces]
See Pic-2.png
5. For interface enp0s5 (Management) click on Offloading Modes: [Default] check "Enabled" for 'rx-all' and 'rx-vlan-offload'
See Pic-3.png
6. For interface enp0s6 (Private) click on Offloading Modes: [Default] check "Disabled" for 'generic-receive-offload', 'generic-segmentation-offload', 'tcp-segmentation-offload'
See Pic-4.png
7. Click [Apply]
8. Click [Back To Node List]
9. Uncheck [Select All]
10. Click on button [Settings] of any but node-1 and click [Configure interfaces]
See Pic-5

Expected results:
See Pic-6.png

Actual results:
See Pic-7.png

[Alexander Zatserklyany (zatserklyany)]

[root@nailgun ~]# shotgun2 short-report
cat /etc/fuel_build_id:
 450
cat /etc/fuel_build_number:
 450
cat /etc/fuel_release:
 10.0
cat /etc/fuel_openstack_version:
 newton-10.0
------------
root@nailgun ~]# fuel node
id | status | name | cluster | ip | mac | roles | pending_roles | online | group_id
---+--------+---------------------+---------+------------+-------------------+------------+---------------+--------+---------
 5 | ready | slave-01_controller | 1 | 10.109.0.4 | 64:e6:65:87:f1:58 | controller | | 1 | 1
 3 | ready | slave-04_compute | 1 | 10.109.0.7 | 64:64:ec:5f:a1:8d | compute | | 1 | 1
 4 | ready | slave-03_controller | 1 | 10.109.0.6 | 64:58:e1:4c:07:1d | controller | | 1 | 1
 2 | ready | slave-02_controller | 1 | 10.109.0.5 | 64:08:b6:4b:92:1c | controller | | 1 | 1
 1 | ready | slave-05_compute | 1 | 10.109.0.8 | 64:1c:74:e2:cd:58 | compute | | 1 | 1
 -------
 [root@nailgun ~]# for i in {1..5}; do ssh node-$i ethtool -k enp0s5 | grep -E 'rx-all|rx-vlan-offload'; doneWarning: Permanently added 'node-1' (ECDSA) to the list of known hosts.
rx-vlan-offload: on
rx-all: on
Warning: Permanently added 'node-2' (ECDSA) to the list of known hosts.
rx-vlan-offload: on
rx-all: off
Warning: Permanently added 'node-3' (ECDSA) to the list of known hosts.
rx-vlan-offload: on
rx-all: off
Warning: Permanently added 'node-4' (ECDSA) to the list of known hosts.
rx-vlan-offload: on
rx-all: off
Warning: Permanently added 'node-5' (ECDSA) to the list of known hosts.
rx-vlan-offload: on
rx-all: off
[root@nailgun ~]# for i in {1..5}; do ssh node-$i ethtool -k enp0s6 | grep -E 'generic-receive-offload|generic-segmentation-offload|tcp-segmentation-offload'; done
Warning: Permanently added 'node-1' (ECDSA) to the list of known hosts.
tcp-segmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: off
Warning: Permanently added 'node-2' (ECDSA) to the list of known hosts.
tcp-segmentation-offload: on
generic-segmentation-offload: on
generic-receive-offload: on
Warning: Permanently added 'node-3' (ECDSA) to the list of known hosts.
tcp-segmentation-offload: on
generic-segmentation-offload: on
generic-receive-offload: on
Warning: Permanently added 'node-4' (ECDSA) to the list of known hosts.
tcp-segmentation-offload: on
generic-segmentation-offload: on
generic-receive-offload: on
Warning: Permanently added 'node-5' (ECDSA) to the list of known hosts.
tcp-segmentation-offload: on
generic-segmentation-offload: on
generic-receive-offload: on

[Alexander Zatserklyany (zatserklyany)]

It affects only Fuel Web UI

[Vitaly Kramskikh (vkramskikh)]

This is Fuel UI bug, thus it cannot be swarm-blocker

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/367279

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/369406

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/367279
Committed: https://git.openstack.org/cgit/openstack/fuel-ui/commit/?id=7d287a8a0281f9926750ea346fd63da3d12e46c5
Submitter: Jenkins
Branch: master

commit 7d287a8a0281f9926750ea346fd63da3d12e46c5
Author: Julia Aranovich <email address hidden>
Date: Tue Sep 13 14:56:48 2016 +0300

    Fix configuration of offloading types applying

    Get limitations of the first node when updating interfaces of other nodes.

    Change-Id: I39287f8885658c37bcb8d66ddee0fc75c0a8cd7e
    Closes-Bug: #1594788

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/369406
Committed: https://git.openstack.org/cgit/openstack/fuel-ui/commit/?id=8a3924d6d94e4007de6239140727a9a12e09f035
Submitter: Jenkins
Branch: stable/mitaka

commit 8a3924d6d94e4007de6239140727a9a12e09f035
Author: Julia Aranovich <email address hidden>
Date: Tue Sep 13 14:56:48 2016 +0300

    Fix configuration of offloading types applying

    Get limitations of the first node when updating interfaces of other nodes.

    Change-Id: I39287f8885658c37bcb8d66ddee0fc75c0a8cd7e
    Closes-Bug: #1594788

[Alexey. Kalashnikov (akalashnikov)]

Verified on 9.1 snapshot #263

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/fuel-ui 10.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/fuel-ui 10.0.0 release.