Generic (recv/segm) offload on physical nics should be disabled

Bug #1275650 reported by Andrey Kirilochkin on 2014-02-03
32
This bug affects 5 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
High
Vladimir Kozhukalov
4.1.x
High
Sergey Vasilenko
5.0.x
High
Fuel Python (Deprecated)
5.1.x
High
Fuel Python (Deprecated)

Bug Description

I have environment with 10gigs broadcom bnx2x:
# ethtool eth1
Settings for eth1:
 Supported ports: [ FIBRE ]
 Supported link modes: 1000baseT/Full
                         10000baseT/Full
 Supported pause frame use: Symmetric Receive-only
 Supports auto-negotiation: No
 Advertised link modes: 10000baseT/Full
 Advertised pause frame use: No
 Advertised auto-negotiation: No
 Speed: 10000Mb/s
 Duplex: Full
 Port: Direct Attach Copper
 PHYAD: 16
 Transceiver: internal
 Auto-negotiation: off
 Supports Wake-on: d
 Wake-on: d
 Current message level: 0x00000000 (0)

 Link detected: yes

# ethtool -k eth1
Offload parameters for eth1:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: on

Dmesg: http://paste.gnu.su/show/26/
Interfaces: http://paste.gnu.su/show/27/

The situation:
Kernel can't turn interfaces in promisc mode, due to generic segmentation offload; http://paste.gnu.su/show/26/. Every time due this error agents migrates to another host, and instances leave their ip-addresses because they have no more access to dhcp server.
I don't know how to describe the situation, but i have solved the situation.
Solution:
ethtool -K eth1 gso off
ethtool -K eth1 gro off

I think, that we must use these options be default for 10 gig interfaces.

not the same bug

Changed in fuel:
importance: Undecided → High
status: New → Triaged
milestone: none → 4.1
assignee: nobody → Sergey Vasilenko (xenolog)
summary: - Broadcom bnx2x 10 gigs generic segmentation offload
+ Generic (recv/segm) offload on physical nics should be disabled
Changed in fuel:
status: Triaged → In Progress
Changed in fuel:
importance: High → Medium
Mike Scherbakov (mihgen) on 2014-02-21
Changed in fuel:
milestone: 4.1 → 5.0
Changed in fuel:
importance: Medium → High
Bogdan Dobrelya (bogdando) wrote :

Raised to high because the https://bugs.launchpad.net/fuel/+bug/1270439 duplicate bug has High priority (Neutron GRE network perfomance related)

Matthew Mosesohn (raytrac3r) wrote :

Easy, but important bug. Targeting for 4.1.1

tags: added: customer-found
Changed in fuel:
milestone: 5.0 → 4.1.1

Reviewed: https://review.openstack.org/75079
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=964808cdc692a1bf3e191f2f678bd313fdf11f45
Submitter: Jenkins
Branch: master

commit 964808cdc692a1bf3e191f2f678bd313fdf11f45
Author: Sergey Vasilenko <email address hidden>
Date: Fri Feb 14 17:12:39 2014 +0400

    Add ethtool support to the L23network module

    Change-Id: I5c74b503b13847eac5176b7e0561fcc0a95e0822
    Partial-bug: #1275650
    Implements: blueprint ethtool-at-boot-by-l23network

Mike Scherbakov (mihgen) wrote :

Will we introduce it into stable/4.1?

I do not see any reasons why not. But will be better backport it after QA
decision.

/sv

Vladimir Kuklin (vkuklin) wrote :

backport to 4.1.1, please

Changed in fuel:
milestone: 4.1.1 → 5.0
tags: added: backports-4.1.1
Changed in fuel:
milestone: 5.0 → 4.1.1
Sergey Vasilenko (xenolog) wrote :

I think we should make a "disable offload" checkbox in UI and passing the relevant options to the astute.yaml if need.

Reviewed: https://review.openstack.org/83315
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=328497e21b36f428f9d51f7f09c9dd41457fcb68
Submitter: Jenkins
Branch: stable/4.1

commit 328497e21b36f428f9d51f7f09c9dd41457fcb68
Author: Sergey Vasilenko <email address hidden>
Date: Fri Feb 14 17:12:39 2014 +0400

    Add ethtool support to the L23network module

    backport to 4.1

    Change-Id: I5c74b503b13847eac5176b7e0561fcc0a95e0822
    Partial-bug: #1275650
    Implements: blueprint ethtool-at-boot-by-l23network

Changed in fuel:
status: In Progress → Fix Committed
Mike Scherbakov (mihgen) on 2014-05-08
tags: added: release-notes
Meg McRoberts (dreidellhasa) wrote :

Marked as Fixed Issue in 5.0 Release Notes

Matthew Mosesohn (raytrac3r) wrote :

Sergey's comment #10 indicates we should have a checkbox in Fuel UI. Passing to fuel-python team

Mike Scherbakov (mihgen) wrote :

Why do we need a checkbox for it? Don't we know for sure that it is needed for best performance on certain interfaces?

Mike Scherbakov (mihgen) on 2014-11-15
no longer affects: fuel/6.1.x
no longer affects: fuel/6.0.x
Dmitry Pyzhov (dpyzhov) wrote :

Checkbox is open question. Moving back to library team =)

Changed in fuel:
assignee: Fuel Python Team (fuel-python) → Fuel Library Team (fuel-library)
Dmitry Pyzhov (dpyzhov) wrote :

Quick solution is to add checkbox "Disable generic offload on physical nics" on openstack settings page.

Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Fuel Python Team (fuel-python)
Dmitry Pyzhov (dpyzhov) wrote :

Checkbox must be set by default

Changed in fuel:
assignee: Fuel Python Team (fuel-python) → Vladimir Kozhukalov (kozhukalov)

Fix proposed to branch: master
Review: https://review.openstack.org/136003

Changed in fuel:
status: Triaged → In Progress

Reviewed: https://review.openstack.org/136003
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=c1335346fcc155d4e58aa2e37e71148ab554c93b
Submitter: Jenkins
Branch: master

commit c1335346fcc155d4e58aa2e37e71148ab554c93b
Author: Vladimir Kozhukalov <email address hidden>
Date: Thu Nov 20 17:27:09 2014 +0300

    Added 'disable generic offload' checkbox

    By default, we don't touch physical nic settings.
    However, it is better to disable generic segmentation offload (gso)
    and generic receive offload (gro). See ethtool man for details.

    Related fuel-library patch: I69fa29757557ed5337239031bf22d127e4a4edf6

    DocImpact
    Change-Id: I24fdb63bab9166720862b4f6c96b90f6cd1d5f8c
    Closes-Bug: 1275650

Changed in fuel:
status: In Progress → Fix Committed

Reviewed: https://review.openstack.org/136030
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=582d1aaf31b990ab6069e3c89e08ddec7c963de4
Submitter: Jenkins
Branch: master

commit 582d1aaf31b990ab6069e3c89e08ddec7c963de4
Author: Vladimir Kozhukalov <email address hidden>
Date: Thu Nov 20 18:35:01 2014 +0300

    Added disable_offload parameter

    By default, we don't touch physical nic settings.
    However, it is better to disable generic segmentation offload (gso)
    and generic receive offload (gro). See ethtool man for details.

    Related fuel-web patch: I24fdb63bab9166720862b4f6c96b90f6cd1d5f8c

    DocImpact
    Change-Id: I69fa29757557ed5337239031bf22d127e4a4edf6
    Closes-bug: 1275650

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers