StarlingX

dnsmasq.conf misconfigs mgmt MTU to 1500 even if it is configered to 9216

Bug #1811404 reported by Yang Liu on 2019-01-11

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	StarlingX	Fix Released	Medium	Teresa Ho

Bug Description

Brief Description
-----------------
/etc/dnsmasq.conf is setting a default of 1500 for the mgmt interface even when system is configured with 9216 mgmt MTU

Severity
--------
Major

Steps to Reproduce
------------------
1. Install and configure a system with large mgmt MTU
2. check /etc/dnsmasq.conf on a compute host
3. grep MTU /var/log/kern.log on a compute host

Expected Behavior
------------------
1. mgmt MTU is 9216 in system host-if-list <host>
2/3. mgmt MTU should not be set to 1500

Actual Behavior
----------------
1. as expected
2. mgmt mtu is set to 1500 in dnsmasq.conf
3. mgmt MTU on compute was set to 9216 by ifcfg, it is then got reset to 1500 after DHCPDISCOVER
compute-1:/home/wrsroot# grep MTU /var/log/kern.log
    2019-01-08T20:15:05.820 compute-1 kernel: info [ 120.136433] ixgbe 0000:83:00.1 ens801f1: changing MTU from 1500 to 9216
    2019-01-08T20:15:13.249 compute-1 kernel: info [ 127.565644] ixgbe 0000:83:00.1 ens801f1: changing MTU from 9216 to 1500
    2019-01-09T16:46:57.116 compute-1 kernel: info [ 38.450255] ixgbe 0000:83:00.1 ens801f1: changing MTU from 1500 to 9216
    2019-01-09T16:47:04.038 compute-1 kernel: info [ 45.372024] ixgbe 0000:83:00.1 ens801f1: changing MTU from 9216 to 1500
    2019-01-10T19:32:28.146 compute-1 kernel: info [ 38.470733] ixgbe 0000:83:00.1 ens801f1: changing MTU from 1500 to 9216
    2019-01-10T19:32:38.194 compute-1 kernel: info [ 48.517861] ixgbe 0000:83:00.1 ens801f1: changing MTU from 9216 to 1500
    2019-01-10T21:36:43.011 compute-1 kernel: info [ 38.362522] ixgbe 0000:83:00.1 ens801f1: changing MTU from 1500 to 9216

Reproducibility
---------------
Reproducible

System Configuration
--------------------
Dedicated storage (probably also Multi-node system)

Branch/Pull Time/Commit
-----------------------
master as of 2019-01-06_20-18-00

Timestamp/Logs
--------------
compute-1:/home/wrsroot# grep MTU /var/log/kern.log
    2019-01-08T20:15:05.820 compute-1 kernel: info [ 120.136433] ixgbe 0000:83:00.1 ens801f1: changing MTU from 1500 to 9216
    2019-01-08T20:15:13.249 compute-1 kernel: info [ 127.565644] ixgbe 0000:83:00.1 ens801f1: changing MTU from 9216 to 1500
    2019-01-09T16:46:57.116 compute-1 kernel: info [ 38.450255] ixgbe 0000:83:00.1 ens801f1: changing MTU from 1500 to 9216
    2019-01-09T16:47:04.038 compute-1 kernel: info [ 45.372024] ixgbe 0000:83:00.1 ens801f1: changing MTU from 9216 to 1500
    2019-01-10T19:32:28.146 compute-1 kernel: info [ 38.470733] ixgbe 0000:83:00.1 ens801f1: changing MTU from 1500 to 9216
    2019-01-10T19:32:38.194 compute-1 kernel: info [ 48.517861] ixgbe 0000:83:00.1 ens801f1: changing MTU from 9216 to 1500
    2019-01-10T21:36:43.011 compute-1 kernel: info [ 38.362522] ixgbe 0000:83:00.1 ens801f1: changing MTU from 1500 to 9216

Tags:

Revision history for this message

Ghada Khalil (gkhalil) wrote on 2019-01-11:

Marking as release gating -- Issue introduced by Generalized Network config feature; unclear why this bug didn't cause any issues before.

Changed in starlingx:
importance:	Undecided → Medium
assignee:	nobody → Teresa Ho (teresaho)
status:	New → Triaged
tags:	added: stx.2019.03 stx.networking

Ken Young (kenyis) on 2019-01-18

tags:

added: stx.2019.05
removed: stx.2019.03

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-02-01: Fix proposed to stx-config (master)

Fix proposed to branch: master
Review: https://review.openstack.org/634415

Changed in starlingx:
status:	Triaged → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-02-01: Fix merged to stx-config (master)

Reviewed: https://review.openstack.org/634415
Committed: https://git.openstack.org/cgit/openstack/stx-config/commit/?id=fda0dcd604b6f7f3d613e6a7a3a637a27a8fdf29
Submitter: Zuul
Branch: master

commit fda0dcd604b6f7f3d613e6a7a3a637a27a8fdf29
Author: Teresa Ho <email address hidden>
Date: Thu Jan 31 11:51:44 2019 -0500

Fix mtu configuration for dns and nfs

    The MTU value was set to the default value instead of the configured
    value for the interface.
    This commit sets the MTU value to the configured for the interface.

The mtu value is also used to calculate the read/write size for nfs
mounts. The commit also ensures that it is set to the correct value.

    Closes-Bug: 1811404
    Change-Id: Ieb3c6d989726075af205823dc1d570c00a24c28d
    Signed-off-by: Teresa Ho <email address hidden>

Changed in starlingx:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-02-01: Fix proposed to stx-config (f/stein)

Fix proposed to branch: f/stein
Review: https://review.openstack.org/634475

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-02-01: Fix merged to stx-config (f/stein)

Download full text (6.3 KiB)

Reviewed: https://review.openstack.org/634475
Committed: https://git.openstack.org/cgit/openstack/stx-config/commit/?id=bc22ce210b8d587dbb21a42469f7160328b19cc3
Submitter: Zuul
Branch: f/stein

commit fda0dcd604b6f7f3d613e6a7a3a637a27a8fdf29
Author: Teresa Ho <email address hidden>
Date: Thu Jan 31 11:51:44 2019 -0500

Fix mtu configuration for dns and nfs

    The MTU value was set to the default value instead of the configured
    value for the interface.
    This commit sets the MTU value to the configured for the interface.

The mtu value is also used to calculate the read/write size for nfs
mounts. The commit also ensures that it is set to the correct value.

    Closes-Bug: 1811404
    Change-Id: Ieb3c6d989726075af205823dc1d570c00a24c28d
    Signed-off-by: Teresa Ho <email address hidden>

commit 6e0e3f2e2fb05ab8d1a6710e8b9a655e145b35eb
Author: Matt Peters <email address hidden>
Date: Wed Jan 30 15:44:28 2019 -0500

Fix multi-net interface configuration

    On the worker and storage nodes, the mgmt interface should be
    set to DHCP and the cluster host alias should be set to use static IP.
    However, the recent fix for this added DHCP to an alias interface which
    is not supported. This was causing the management network to not be
    configured after the host was rebooted and the interfaces were
    reconfigured by Puppet.

    The interface configuration has been updated to ensure that DHCP
    networks are configured against the parent device and all additional
    static addresses will be configured as alias interfaces.

    There is also a few additional issues that have been fixed:
      - multinetting with different address families
      - traffic controls applied to parent interface
      - address scope for aliases on AIO-SX with dedicated interfaces
      - setting of the gateway address on multi-net interfaces

Story: 2004273
Task: 27826

Change-Id: If8e354bf41cbc1c11966a4b43b8a058c33580c49
Signed-off-by: Matt Peters <email address hidden>

commit 5624c74062f97c2cfaddda6d5d5cb04a10411db4
Author: Tyler Smith <email address hidden>
Date: Thu Jan 10 11:37:27 2019 -0500

Trigger application reapply on host unlock/delete

    - Triggers an application reapply (including override
      regeneration) on node unlock to pick up a new node being added
      or configuration changes being made.
    - The reapply also triggers on node delete if the host
      had a compute node label to remove any per-host overrides.
    - Turned on the restriction that nodes must be locked to modify
      labels. Added an audit task to sync any labels made before
      a node is unlocked and missing from k8s.
    - Modified k8s puppet manifest to only launch kubelet on the
      initial configuration, after that the service file is
      modified to have a dependancy on the config gate.
      This is to avoid PLEG errors in kubernetes due to the node
      being overwhelmed during boot.

    Change-Id: I1d9ca92f451aa322765da43ffcbb1d95f97f92f2
    Story: 2004520
    Task: 28826
    Signed-off-by: Tyler Smith <email address hidden>

c...

Reviewed:  https://review.openstack.org/634475
Committed: https://git.openstack.org/cgit/openstack/stx-config/commit/?id=bc22ce210b8d587dbb21a42469f7160328b19cc3
Submitter: Zuul
Branch:    f/stein

commit fda0dcd604b6f7f3d613e6a7a3a637a27a8fdf29
Author: Teresa Ho <teresa.ho@windriver.com>
Date:   Thu Jan 31 11:51:44 2019 -0500

Fix mtu configuration for dns and nfs
    
    The MTU value was set to the default value instead of the configured
    value for the interface.
    This commit sets the MTU value to the configured for the interface.
    
    The mtu value is also used to calculate the read/write size for nfs
    mounts. The commit also ensures that it is set to the correct value.
    
    Closes-Bug: 1811404
    Change-Id: Ieb3c6d989726075af205823dc1d570c00a24c28d
    Signed-off-by: Teresa Ho <teresa.ho@windriver.com>

commit 6e0e3f2e2fb05ab8d1a6710e8b9a655e145b35eb
Author: Matt Peters <matt.peters@windriver.com>
Date:   Wed Jan 30 15:44:28 2019 -0500

Fix multi-net interface configuration
    
    On the worker and storage nodes, the mgmt interface should be
    set to DHCP and the cluster host alias should be set to use static IP.
    However, the recent fix for this added DHCP to an alias interface which
    is not supported.  This was causing the management network to not be
    configured after the host was rebooted and the interfaces were
    reconfigured by Puppet.
    
    The interface configuration has been updated to ensure that DHCP
    networks are configured against the parent device and all additional
    static addresses will be configured as alias interfaces.
    
    There is also a few additional issues that have been fixed:
      - multinetting with different address families
      - traffic controls applied to parent interface
      - address scope for aliases on AIO-SX with dedicated interfaces
      - setting of the gateway address on multi-net interfaces
    
    Story: 2004273
    Task: 27826
    
    Change-Id: If8e354bf41cbc1c11966a4b43b8a058c33580c49
    Signed-off-by: Matt Peters <matt.peters@windriver.com>

commit 5624c74062f97c2cfaddda6d5d5cb04a10411db4
Author: Tyler Smith <tyler.smith@windriver.com>
Date:   Thu Jan 10 11:37:27 2019 -0500

Trigger application reapply on host unlock/delete
    
    - Triggers an application reapply (including override
      regeneration) on node unlock to pick up a new node being added
      or configuration changes being made.
    - The reapply also triggers on node delete if the host
      had a compute node label to remove any per-host overrides.
    - Turned on the restriction that nodes must be locked to modify
      labels.  Added an audit task to sync any labels made before
      a node is unlocked and missing from k8s.
    - Modified k8s puppet manifest to only launch kubelet on the
      initial configuration, after that the service file is
      modified to have a dependancy on the config gate.
      This is to avoid PLEG errors in kubernetes due to the node
      being overwhelmed during boot.
    
    Change-Id: I1d9ca92f451aa322765da43ffcbb1d95f97f92f2
    Story: 2004520
    Task: 28826
    Signed-off-by: Tyler Smith <tyler.smith@windriver.com>

commit 7e281afa397146321c3ee157aaa54ac13a65a823
Author: Chris Friesen <chris.friesen@windriver.com>
Date:   Mon Jan 28 15:46:09 2019 -0600

Add support for galera based mariadb replication
    
    Building on top of the previous commit which added the helm chart
    for the galera arbitrator, this does essentially three things:
    
    1) Add the galera arbitrator (aka "garbd") helm chart to the
    stx-openstack application manifests as part of the mariadb
    chart group.
    
    2) Modify the mariadb system overrides to specify a number of
    replicas equal to the number of controller nodes.
    
    3) Modify sysinv to remove the garbd helm chart from the mariadb
    chart group if there is only one controller node (for the AIO-SX
    use-case).
    
    The rationale for this is that galera requires three DB nodes to
    allow for a quorum if a Kubernetes node goes down.  The mariadb
    replicas use StatefulSets and are therefore pinned to specific
    Kubernetes nodes which is problematic for the third DB node--if
    its host node goes down it won't be moved to the remaining
    Kubernetes node. By running a galera arbitrator as the third DB
    node, we avoid this problem (because the arbitrator is stateless
    and can be restarted on the remaining Kubernetes node).
    
    The third item was a bit tricky and required testing a few different
    approaches.  Just setting the chart manifests to "disabled" should
    have worked, but Armada got stuck waiting for resources that didn't
    exist.  Newer Armada supports explicitly setting an empty list of
    resources to wait for, but that would require an Armada upgrade and
    would still require a meta-override to specify the wait parameters.
    Ultimately the path chosen was to add the ability for each helm
    class in sysinv to specify meta-overrides as well as regular
    values overrides.  This allows them to change non-values portions
    of the "chart" section of the manifest, as well as the "chartgroup"
    section or even the main "manifest" section.
    
    Change-Id: I1fe66985a4af18d624aa54d45e91db7c121333f5
    Story: 2004712
    Task: 28743
    Task: 29055
    Task: 29056
    Signed-off-by: Chris Friesen <chris.friesen@windriver.com>

commit 860c97767b1b9f970c526f8161337a28bc5d0eb7
Author: Chris Friesen <chris.friesen@windriver.com>
Date:   Mon Jan 28 10:05:01 2019 -0600

Add helm chart for galera arbitrator
    
    As part of the work to enable clustered database, a new helm chart is
    needed to define how to run the galera arbitrator.  This chart makes
    use of the newly-created stx-mariadb Docker image which has the galera
    arbitrator added to the upstream openstack-helm mariadb Docker image.
    
    Since we have a new system chart, add sysinv support to handle any
    system overrides.  Initially this just means specifying the Docker
    image.
    
    When we go to merge this upstream, we should try to add the galera
    arbitrator to the existing upstream Docker image and the existing
    mariadb helm chart.  One upstream dev has said that they'd be willing
    to accept this.
    
    Change-Id: If0362916d3b575adabf9d6c8cc467e488b249b7b
    Story: 2004712
    Task: 29054
    Signed-off-by: Chris Friesen <chris.friesen@windriver.com>

tags:

added: in-f-stein

Ken Young (kenyis) on 2019-04-05

tags:

added: stx.2.0
removed: stx.2019.05

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.