Enable multus logs for better debugging

Bug #1900065 reported by Ghada Khalil
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
StarlingX
Fix Released
Low
Cole Walker

Bug Description

Brief Description
-----------------
There have been some reports of SR-IOV configuration (interface or vlan) missing intermittently on containerized application upgrade. Debugging this issue is proving challenging as the frequency of occurrence is low and there aren't enough logs to investigate after the fact. This LP will track enabling the multus logs by default so that more data is generated to help with debugging in the future.

Severity
--------
Minor

Steps to Reproduce
------------------
One instance reported is on upgrading a containerized application which uses a vlan SR-IOV interface

Expected Behavior
------------------
The interface/vlan is created in the pod after the application upgrade

Actual Behavior
----------------
The vlan is missing from the pod after the application upgrade

Reproducibility
---------------
Rare. Reported twice, but not reproducible since.

System Configuration
--------------------
AIO-SX

Branch/Pull Time/Commit
-----------------------
Likely an issue w/ stx.3.0 & stx.4.0. However, due to low frequency of occurrence, the recommendation is to turn the logs on in stx master only for now.

Last Pass
---------
N/A

Timestamp/Logs
--------------
Not required

Test Activity
-------------
Lab Testing

Workaround
----------
Delete and re-create the pod

Ghada Khalil (gkhalil)
Changed in starlingx:
importance: Undecided → Low
status: New → Triaged
assignee: nobody → Cole Walker (cwalops)
tags: added: stx.5.0 stx.networking
Changed in starlingx:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ansible-playbooks (master)

Fix proposed to branch: master
Review: https://review.opendev.org/758872

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to config-files (master)

Reviewed: https://review.opendev.org/758871
Committed: https://git.openstack.org/cgit/starlingx/config-files/commit/?id=4083754296b19e2350b7ba03ced46079cc69fcb3
Submitter: Zuul
Branch: master

commit 4083754296b19e2350b7ba03ced46079cc69fcb3
Author: Cole Walker <email address hidden>
Date: Mon Oct 19 17:43:31 2020 -0400

    Add logrotate config for multus logging

    This change provides the logrotate configuration to handle the log files
    being generated by the multus-cni container at /var/log/multus.log. The
    config file is packaged in an rpm and installed to
    /etc/logrotate.d/multus.lograte during system installation.

    Closes-Bug: 1900065

    Change-Id: I01142f7fca9da0d31f69facd569086d96bbde323
    Signed-off-by: Cole Walker <email address hidden>

Changed in starlingx:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ansible-playbooks (master)

Reviewed: https://review.opendev.org/758872
Committed: https://git.openstack.org/cgit/starlingx/ansible-playbooks/commit/?id=2903c81b6f737a5040c6db6bb5c0c9256975d79f
Submitter: Zuul
Branch: master

commit 2903c81b6f737a5040c6db6bb5c0c9256975d79f
Author: Cole Walker <email address hidden>
Date: Mon Oct 19 18:06:35 2020 -0400

    Enable multus logging in configmap

    Provides the required parameters to enable multus debug logging to a file
    on the host machine where the multus-cni container runs. Logs are saved
    to /var/log/multus.log

    Updates the name of the configmap with a version indicator to ensure
    that the multus daemonset is restarted when the new configmap is
    applied.

    A related review has been submitted for the logrotate configuration
    to manage the size and lifetime of the log file.

    Closes-Bug: 1900065

    Depends-On: https://review.opendev.org/#/c/758871/
    Change-Id: Id712637955a7421e1ecad55fb2c5747ed41b5c45
    Signed-off-by: Cole Walker <email address hidden>

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.