worker nodes applying puppet manifest before unlock

Bug #1853329 reported by Joseph Richard on 2019-11-20
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
StarlingX
Medium
Joseph Richard

Bug Description

Brief Description
-----------------
Puppet manifests are being applied on a worker node before it is unlocked. The hieradata appears to have been generated by the sriov interface configuration, before the mgmt interface is configured.

Severity
--------
<Major: System/Feature is usable but degraded>

Steps to Reproduce
------------------
Bring up system
configure sriov on worker nodes before management

Expected Behavior
------------------
Puppet manifest on worker nodes will not apply until after unlock

Actual Behavior
----------------
Puppet manifest on worker nodes applies before unlock

Reproducibility
---------------
<Reproducible/Intermittent/Seen once>
This is consistently reproducible on one lab

System Configuration
--------------------
Multi-node with sriov on worker nodes

Branch/Pull Time/Commit
-----------------------
###
### StarlingX
### Built from master
###

OS="centos"
SW_VERSION="19.09"
BUILD_TARGET="Host Installer"
BUILD_TYPE="Formal"
BUILD_ID="20191119T000000Z"

JOB="STX_build_master_master"
<email address hidden>"
BUILD_NUMBER="324"
BUILD_HOST="starlingx_mirror"
BUILD_DATE="2019-11-19 00:00:00 +0000"

Last Pass
---------
Not previously tested with the same config in that lab.

Timestamp/Logs
--------------
Attach the logs for debugging (use attachments in Launchpad or paste.openstack.org)
Provide a snippet of logs here and the timestamp when issue was seen.
Please indicate the unique identifier in the logs to highlight the problem

Joseph Richard (josephrichard) wrote :

Added logs for controllers. compute node logs aren't collected, as they failed to come up and became unreachable due to this bug. Most relevant information is hieradata for worker nodes.

Ghada Khalil (gkhalil) wrote :

stx.3.0 / high priority - worker nodes don't recover if sriov i/fs are configured first

description: updated
Changed in starlingx:
assignee: nobody → Joseph Richard (josephrichard)
importance: Undecided → High
status: New → Triaged
tags: added: stx.3.0 stx.config stx.networking
Matt Peters (mpeters-wrs) wrote :

The issue is being caused by the changes made under the following commit.
https://opendev.org/starlingx/config/commit/f8fc051a9bc49474251cb475bb36654174edf643

Ghada Khalil (gkhalil) wrote :

This issue can be avoided by configuring the interfaces in order. Lowering the priority and moving out of stx.3.0

tags: added: stx.4.0
removed: stx.3.0
Changed in starlingx:
importance: High → Medium
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers