Bug #1853329 “worker nodes applying puppet manifest before unloc...” : Bugs : StarlingX

Revision history for this message

Joseph Richard (josephrichard) wrote on 2019-11-20:

#1

ALL_NODES_20191119.233853.tar Edit (32.0 MiB, application/x-tar)

Added logs for controllers. compute node logs aren't collected, as they failed to come up and became unreachable due to this bug. Most relevant information is hieradata for worker nodes.

Revision history for this message

Ghada Khalil (gkhalil) wrote on 2019-11-20:

#2

stx.3.0 / high priority - worker nodes don't recover if sriov i/fs are configured first

description:	updated
Changed in starlingx:
assignee:	nobody → Joseph Richard (josephrichard)
importance:	Undecided → High
status:	New → Triaged
tags:	added: stx.3.0 stx.config stx.networking

Revision history for this message

Matt Peters (mpeters-wrs) wrote on 2019-11-21:

#3

The issue is being caused by the changes made under the following commit.
https://opendev.org/starlingx/config/commit/f8fc051a9bc49474251cb475bb36654174edf643

Revision history for this message

Ghada Khalil (gkhalil) wrote on 2019-11-26:

#4

This issue can be avoided by configuring the interfaces in order. Lowering the priority and moving out of stx.3.0

tags:	added: stx.4.0 removed: stx.3.0
Changed in starlingx:
importance:	High → Medium

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-12-13: Fix proposed to config (master)

#5

Fix proposed to branch: master
Review: https://review.opendev.org/698816

Changed in starlingx:
status:	Triaged → In Progress

Ghada Khalil (gkhalil) on 2019-12-17

description:

updated

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-12-17: Change abandoned on config (master)

#6

Change abandoned by Joseph Richard (<email address hidden>) on branch: master
Review: https://review.opendev.org/698816

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-12-18: Fix proposed to config (master)

#7

Fix proposed to branch: master
Review: https://review.opendev.org/699734

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-01-07: Fix merged to config (master)

#8

Reviewed: https://review.opendev.org/699734
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=a9686eccf93dbbb89f8e460aa0fd449e688f957d
Submitter: Zuul
Branch: master

commit a9686eccf93dbbb89f8e460aa0fd449e688f957d
Author: Joseph Richard <email address hidden>
Date: Fri Dec 13 15:01:58 2019 -0500

Prevent provisioning sriov without mgmt iface

    This commit adds a semantic check to prevent provisioning an sriov
    interface without a mgmt interface configured. This is necessary to
    prevent an invalid network config being generated and applied, which
    causes the node to lose connectivity over mgmt and become unreachable.

    Closes-bug: 1853329
    Change-Id: I783447a2214a1b0f4d698ac20037f8f1e8083958
    Signed-off-by: Joseph Richard <email address hidden>

Changed in starlingx:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-02-04: Fix proposed to config (f/centos8)

#9

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/705837

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-02-05: Fix merged to config (f/centos8)

#10

Download full text (35.0 KiB)

Reviewed: https://review.opendev.org/705837
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=8ac6ec70cb8a787a274fd7227eb34d2b7bcd5f5b
Submitter: Zuul
Branch: f/centos8

commit 7995dd436954b92f1c4e3f760a7609af670c84c8
Author: Jessica Castelino <email address hidden>
Date: Mon Feb 3 12:07:26 2020 -0500

Unit test cases for helm charts

    Test cases added for API endpoints used by:
     1. helm-override-delete
     2. helm-override-show
     3. helm-override-list
     4. helm-override-update
     5. helm-chart-attribute-modify

    Story: 2007082
    Task: 38012
    Change-Id: I86763496bb41084c006f2486702c3b15bde039d2
    Signed-off-by: Jessica Castelino <email address hidden>

commit 7e2fda010299f7305b630d6db97bbe1e169a38b1
Author: Angie Wang <email address hidden>
Date: Wed Jan 29 21:18:18 2020 -0500

Finish kubernetes networking upgrade support

    The commit completes the RPC kube_upgrade_networking
    in sysinv-conductor to run ansible playbook
    upgrade-k8s-networking.yml to upgrade networking pods
    and also updates the networking upgrade function called
    as part of sysinv-conductor startup to provide a current
    kubernetes version when running the upgrade playbook.

    The second control plane upgrade can only be performed
    after the networking upgrade is done, fix the semantic
    check in sysinv api.

    Change-Id: I8dcf5a2baedfaefb0a7ca037eb47bf7cacd686f8
    Story: 2006781
    Task: 37584
    Depends-On: https://review.opendev.org/#/c/705310/
    Signed-off-by: Angie Wang <email address hidden>

commit 52c37a35d2cd62fa1cc1933765c76c1ba8616864
Author: Jerry Sun <email address hidden>
Date: Fri Jan 31 16:10:25 2020 -0500

Add Unit Tests for Dex Sysinv Changes

Add unit tests for the dex helm chart changes under the same story
and task

Story: 2006711
Task: 37857

Depends-On: https://review.opendev.org/#/c/705297/

Change-Id: I3a0e1c490e56188adfbd614fd6ebb21bfdddf49e
Signed-off-by: Jerry Sun <email address hidden>

commit 144587a6ac9fc48b9249be99abadd35dfa49e7a7
Author: Teresa Ho <email address hidden>
Date: Fri Jan 31 15:35:04 2020 -0500

Tox tests for OIDC client helm overrides

Added some tox tests for OIDC client helm overrides.

Story: 2006711
Task: 38481

Change-Id: If4aeaf0010c7076d1d83bacd00d6fd0122d4ffad
Signed-off-by: Teresa Ho <email address hidden>

commit 763ddeadd4e83af6cebf752d693ee3e7d3b005b1
Author: Thomas Gao <email address hidden>
Date: Wed Jan 29 16:30:40 2020 -0500

Fixed errors in address deletion

Allowed address deletion despite missing associated interface or host.

Enabled relevant unit test.

Closes-Bug: 1860186

Change-Id: Ie6e6358aa75091e92914a8b581b4d5203a596f56
Signed-off-by: Thomas Gao <email address hidden>

commit 61463608169e75601b8a4f9db7c98190788d6f6a
Author: Thomas Gao <email address hidden>
Date: Tue Jan 28 15:32:58 2020 -0500

Fixed broken sysinv address get-all api call

Removed unexpected keyword argument that caused the error....

Reviewed:  https://review.opendev.org/705837
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=8ac6ec70cb8a787a274fd7227eb34d2b7bcd5f5b
Submitter: Zuul
Branch:    f/centos8

commit 7995dd436954b92f1c4e3f760a7609af670c84c8
Author: Jessica Castelino <jessica.castelino@windriver.com>
Date:   Mon Feb 3 12:07:26 2020 -0500

Unit test cases for helm charts
    
    Test cases added for API endpoints used by:
     1. helm-override-delete
     2. helm-override-show
     3. helm-override-list
     4. helm-override-update
     5. helm-chart-attribute-modify
    
    Story: 2007082
    Task: 38012
    Change-Id: I86763496bb41084c006f2486702c3b15bde039d2
    Signed-off-by: Jessica Castelino <jessica.castelino@windriver.com>

commit 7e2fda010299f7305b630d6db97bbe1e169a38b1
Author: Angie Wang <angie.wang@windriver.com>
Date:   Wed Jan 29 21:18:18 2020 -0500

Finish kubernetes networking upgrade support
    
    The commit completes the RPC kube_upgrade_networking
    in sysinv-conductor to run ansible playbook
    upgrade-k8s-networking.yml to upgrade networking pods
    and also updates the networking upgrade function called
    as part of sysinv-conductor startup to provide a current
    kubernetes version when running the upgrade playbook.
    
    The second control plane upgrade can only be performed
    after the networking upgrade is done, fix the semantic
    check in sysinv api.
    
    Change-Id: I8dcf5a2baedfaefb0a7ca037eb47bf7cacd686f8
    Story: 2006781
    Task: 37584
    Depends-On: https://review.opendev.org/#/c/705310/
    Signed-off-by: Angie Wang <angie.wang@windriver.com>

commit 52c37a35d2cd62fa1cc1933765c76c1ba8616864
Author: Jerry Sun <jerry.sun@windriver.com>
Date:   Fri Jan 31 16:10:25 2020 -0500

Add Unit Tests for Dex Sysinv Changes
    
    Add unit tests for the dex helm chart changes under the same story
    and task
    
    Story: 2006711
    Task: 37857
    
    Depends-On: https://review.opendev.org/#/c/705297/
    
    Change-Id: I3a0e1c490e56188adfbd614fd6ebb21bfdddf49e
    Signed-off-by: Jerry Sun <jerry.sun@windriver.com>

commit 144587a6ac9fc48b9249be99abadd35dfa49e7a7
Author: Teresa Ho <teresa.ho@windriver.com>
Date:   Fri Jan 31 15:35:04 2020 -0500

Tox tests for OIDC client helm overrides
    
    Added some tox tests for OIDC client helm overrides.
    
    Story: 2006711
    Task: 38481
    
    Change-Id: If4aeaf0010c7076d1d83bacd00d6fd0122d4ffad
    Signed-off-by: Teresa Ho <teresa.ho@windriver.com>

commit 763ddeadd4e83af6cebf752d693ee3e7d3b005b1
Author: Thomas Gao <Thomas.Gao@windriver.com>
Date:   Wed Jan 29 16:30:40 2020 -0500

Fixed errors in address deletion
    
    Allowed address deletion despite missing associated interface or host.
    
    Enabled relevant unit test.
    
    Closes-Bug: 1860186
    
    Change-Id: Ie6e6358aa75091e92914a8b581b4d5203a596f56
    Signed-off-by: Thomas Gao <Thomas.Gao@windriver.com>

commit 61463608169e75601b8a4f9db7c98190788d6f6a
Author: Thomas Gao <Thomas.Gao@windriver.com>
Date:   Tue Jan 28 15:32:58 2020 -0500

Fixed broken sysinv address get-all api call
    
    Removed unexpected keyword argument that caused the error.
    
    Enabled relevant unit test.
    
    Closes-Bug: 1860176
    
    Change-Id: Ic5957da9613cfecd788a9652ede734443b4460c4
    Signed-off-by: Thomas Gao <Thomas.Gao@windriver.com>

commit a3c9e2e6547bc1c59b91b3f9d862016fd0f65d33
Author: Teresa Ho <teresa.ho@windriver.com>
Date:   Fri Jan 17 09:11:18 2020 -0500

Add OIDC client as a platform managed application
    
    Add system overrides for the OIDC client application for
    Windows active directory authentication support.
    The application will be uploaded automatically on unlock
    of controller-0.
    
    Story: 2006711
    Task: 38481
    
    Depends-On: https://review.opendev.org/#/c/703187/
    Change-Id: I3f9cbb4359f042ed51378f3912400f1d76a0743f
    Signed-off-by: Teresa Ho <teresa.ho@windriver.com>

commit 1230cb792de4f8e4addfdb825ea6346353c4344b
Author: Joseph Richard <joseph.richard@windriver.com>
Date:   Fri Jan 17 09:24:06 2020 -0500

Add sysinv api address unit tests
    
    Story: 2007082
    Task: 37999
    Change-Id: I7596a8de794a823e568288d682a1e5772c380b1b
    Signed-off-by: Joseph Richard <joseph.richard@windriver.com>

commit de23dcfd0540a4ec20b3683f94250d1035e60901
Author: Jim Somerville <Jim.Somerville@windriver.com>
Date:   Mon Jan 27 14:25:14 2020 -0500

Security: Allow disabling of spectre v1 swapgs mitigation
    
    Most of the v1 mitigation is baked into the kernel and not
    optional.  The swapgs barriers are, however, optional.
    They have a negative performance impact so we disable them
    by using the nospectre_v1 kernel bootarg.
    
    Change-Id: Ia5938249ad0f0a53435251e505dac843b923ad62
    Closes-Bug: 1860193
    Signed-off-by: Jim Somerville <Jim.Somerville@windriver.com>

commit fb20b0ef5cd41d9122c288b383d16dda647a7117
Author: Thomas Gao <Thomas.Gao@windriver.com>
Date:   Mon Jan 27 15:12:43 2020 -0500

Forbid pxeboot network deletion
    
    Updated the list of network types that are checked prior to network
    deletion. This disallows pxeboot type network to be deleted after
    initial system config.
    
    Enable the unit test to ensure semantic check forbids pxeboot network
    delete.
    
    Closes-Bug: 1859855
    
    Change-Id: I9a9d25abadc1e4f0cade2a4efef3c8adf3f0262b
    Signed-off-by: Thomas Gao <Thomas.Gao@windriver.com>

commit fccb70302c686d09a7e0f079e7ffcc3b95a1f178
Author: John Kung <john.kung@windriver.com>
Date:   Thu Jan 23 17:49:15 2020 -0500

Update stx-monitor to not require storage provisioner in subcloud
    
    In a subcloud configuration, the stx-monitor app does not have a
    dependency on storage provisioner.  In such case, do not perform
    operations with storage provisioner secrets.
    
    Change-Id: I04fe2f3ec5fae089ed715eba69afaf8db0c4cb35
    Story: 2007064
    Task: 38459
    Signed-off-by: John Kung <john.kung@windriver.com>

commit f34731ac2694a63935025f57b74a735f2895f340
Author: Jessica Castelino <jessica.castelino@windriver.com>
Date:   Thu Jan 16 16:48:02 2020 -0500

Unit test case for DNS API
    
    Story: 2007082
    Task: 38007
    Change-Id: I5ca089237ca17d287146795f587720a7aa7fd039
    Signed-off-by: Jessica Castelino <jessica.castelino@windriver.com>

commit b37576034d05b5c8ca4c9a0a7f6cd553de6db8ca
Author: Jessica Castelino <jessica.castelino@windriver.com>
Date:   Wed Jan 15 17:02:19 2020 -0500

Add unit test for NTP api
    
    Story: 2007082
    Task: 38026
    Change-Id: I40c5f6779ad28df858d4837b0f01c5bab6523d38
    Signed-off-by: Jessica Castelino <jessica.castelino@windriver.com>

commit 3c1eb398c8294465b05b45bd35a4b5a359c170b4
Author: David Sullivan <david.sullivan@windriver.com>
Date:   Wed Jan 15 23:47:59 2020 -0500

Install secondary controller nodes with kubeadm join
    
    Kubeadm init is no longer supported for installing secondary nodes in an
    HA kubernetes cluster. kubeadm join with the --controller-plane option
    should be used.
    
    Change-Id: I57d2d01335739fb3dce5a9c4cd749b99eafbf5ee
    Partial-Bug: 1846829
    Signed-off-by: David Sullivan <david.sullivan@windriver.com>

commit 59bc9f7a87dd6dc634e9b513e6152b658e968fc3
Author: Jerry Sun <jerry.sun@windriver.com>
Date:   Wed Jan 15 10:29:48 2020 -0500

Add Dex as a platform managed application
    
    Add system overrides for the dex application for Windows active
    directory authentication support. The application will be uploaded
    automatically on unlock of controller-0.
    
    Story: 2006711
    Task: 37857
    
    Depends-On: https://review.opendev.org/#/c/700180/
    Depends-On: https://review.opendev.org/#/c/703828/
    Change-Id: I17875b741b4f5d10661f99c527676c9fbac13aab
    Signed-off-by: Jerry Sun <jerry.sun@windriver.com>

commit 171eecff2ffccaac157503164a265270c40e9e8f
Author: Abdelfattah Saafan <abdelfattah.saafan@windriver.com>
Date:   Wed Jan 22 13:48:44 2020 -0500

Enable metricbeat kubernetes state_statefulset
    
    metricbeat kubernetes module is missing state_statefulset metricset.
    This change will add the missing metricset.
    
    Depends-On: https://review.opendev.org/#/c/703854/
    Change-Id: I27efa6ca7a2c8f73dab087a6ea7cc5ad4376b0f1
    Closes-Bug: 1858221
    Signed-off-by: Abdelfattah Saafan <abdelfattah.saafan@windriver.com>

commit 09f30594b69ab2f70e84a3c285104bab1adaf9f0
Author: Joseph Richard <joseph.richard@windriver.com>
Date:   Wed Jan 15 11:51:59 2020 -0500

Add unit tests for oam network api
    
    Story: 2007082
    Task: 38024
    
    Change-Id: I849c182c387c387f4419dc2161431c2f15305725
    Signed-off-by: Joseph Richard <joseph.richard@windriver.com>

commit 387a20ab23b000b99692abab494c42bc6b6a76cb
Author: Andy Ning <andy.ning@windriver.com>
Date:   Wed Jan 22 09:11:09 2020 -0500

Populate barbican region_name during bootstrap
    
    During DC System Controller deployment, the ansible script
    intermittently fails during the barbican credential creation
    on ansible re-play. Even in the re-play case, it is a random
    failure depending on the order of the endpoints in the service
    catalog.
    
    The reason for this to happen is that, during the initial play, the
    barbican secrets are created prior to initial system configuration
    population so endpoints for SystemController region are not created.
    Barbican will use the RegionOne keystone endpoint. But after initial
    play finished, endpoints for SystemController region are created thus
    there are two keystone endpoints (RegionOne and SystemController).
    
    With two region keystone endpoints during re-play, Barbican may pickup
    SystemController region keystone endpoint during credential creation.
    Yet the service for SystemController region (dcorch identity proxy) has
    not started, causing the credential creation to fail.
    
    The fix is to explicitly configure Barbican region_name to RegionOne
    during bootstrap so re-play will use RegionOne keystone endpoint. Then
    update Barbican region_name after service endpoints reconfiguration to
    make region_name consistent with keystone service catalog, so requests
    to Barbican will always succeed.
    
    Change-Id: I7afda2806aad6437f746ca8ff39adee2d29571cf
    Closes-Bug: 1859726
    Signed-off-by: Andy Ning <andy.ning@windriver.com>

commit 6d53febf34e46fb44a5783718fd5cf28931f8bea
Author: Kristal Dale <kristal.dale@intel.com>
Date:   Wed Jan 15 14:42:24 2020 -0800

Update landing pages for docs, api-ref, and release notes:
    
    - Use updated project name in titles/text
    - Correct text for link to Storyboard (docs)
    - Correct capitalization in section headings
    - Correct formatting for section headings
    - Update project name in link to release notes, api-ref
    - Update project name in config for docs/releasenotes/api-ref
    
    Story:2007193
    Task:38336
    
    Change-Id: Ia01582558ab27accb68ccdf00d492f5e79689f04
    Signed-off-by: Kristal Dale <kristal.dale@intel.com>

commit f5e43997e398f0a18f4d10cd314161f1361826b9
Author: Angie Wang <angie.wang@windriver.com>
Date:   Thu Jan 9 16:45:22 2020 -0500

Clean up dead code in controllerconfig
    
    This commit updates to clean up the obsolete code in
    controllerconfig as a result of switch to Ansible.
    
    The cleanup includes:
      - the whole config_controller with its relevant files
        (ie..initial configuration, backup/restore, image clone...)
      - config_subcloud
      - config_region
    
    Tests conducted:
     - fresh installation of AIO-SX, AIO-DX, standard
       and storage labs
     - platform sanity performed on AIO-SX, AIO-DX,
       standard and storage labs
     - stx-openstack app apply/remove performed on AIO-SX
       and AIO-DX
     - stx-monitor app apply/remove on standard and
       storage labs
     - backup&restore performed on standard lab
     - distributed cloud was deployed
    
    Change-Id: I0d1eb91188dfb24dca68a19a497b9e4edb0b0ef0
    Partial-Bug: 1834218
    Depends-On: https://review.opendev.org/#/c/703517/
    Signed-off-by: Angie Wang <angie.wang@windriver.com>

commit 6b4e8efc26b092b1d17caadcfd5c742df2d28d11
Author: Al Bailey <Al.Bailey@windriver.com>
Date:   Mon Jan 20 10:41:11 2020 -0600

Updates for cgtsclient to work with python3
    
    Deprecate cgtsclient.openstack.common files since
    those files were mostly replaced by oslo.
    
    Fix a wrapping_formatters import which was using a relative path.
    
    A recursion error with deepcopy has been fixed which allows
    the py36 unit tests to be enabled.
    
    Newer flake8, hacking and bugbear are now enabled since python3
    is able to be tested now.
    
    Change-Id: I4978ab33b09e13b933b7f0fc9da22ed97e671938
    Story: 2007082
    Task: 38314
    Signed-off-by: Al Bailey <Al.Bailey@windriver.com>

commit 7d4120b240b6735ec6c3947bc3d33bc0ca9c15aa
Author: Zhipeng Liu <zhipengs.liu@intel.com>
Date:   Fri Jan 17 02:42:07 2020 +0800

Add unit tests for inumas_update_by_ihost RPC EP.
    
    Story: 2007082
    Task: 38139
    
    Change-Id: Iaddc0572261813f5117ba8495dbd15ff4c7e1b30
    Signed-off-by: Zhipeng Liu <zhipengs.liu@intel.com>

commit f421b9bf3c80ec465754f43ac60209e7bb251268
Author: Zhipeng Liu <zhipengs.liu@intel.com>
Date:   Thu Jan 16 02:41:03 2020 +0800

Add unit tests for pci_device_update_by_host RPC EP.
    
    Change-Id: Icf899feb3d9e877fcc57cbf417941ccc85a755f2
    Story: 2007082
    Task: 38108
    Signed-off-by: Zhipeng Liu <zhipengs.liu@intel.com>

commit 52b2e021f3d206f420cbd764f5024ddf73f586a8
Author: Lin Shuicheng <shuicheng.lin@intel.com>
Date:   Sun Jan 19 01:49:59 2020 +0000

Revert "Revert "configure kubernetes to use containerd as CRI""
    
    This reverts commit be3208886a2a2c69248e923eab38e4c73e35c43d.
    
    Depends-On: https://review.opendev.org/703263
    Change-Id: Id834b878099f6db49d18cbc06e97cdb2669cb352
    Signed-off-by: Shuicheng Lin <shuicheng.lin@intel.com>

commit befc4cb96cc6baf9baa0cff2b72726f65f176fb2
Author: zhipengl <zhipengs.liu@intel.com>
Date:   Wed Jan 15 00:14:24 2020 +0800

Add unit tests for get_ihost_by_hostname RPC EP.
    
    Change-Id: I99898dd16909e8ca19ab7c8612b7fc69b721f387
    Story: 2007082
    Task: 38105
    Signed-off-by: Zhipeng Liu <zhipengs.liu@intel.com>

commit 1bc2f5a8948683b6fc5cf79c1a3d24fed39e1d32
Author: zhipengl <zhipengs.liu@intel.com>
Date:   Tue Jan 14 20:02:57 2020 +0800

Add unit tests for get_ihost_by_macs RPC EP.
    
    Story: 2007082
    Task: 38097
    Change-Id: I0f2a243099e506fd1743a9eaf64fdc6dd3828a1e
    Signed-off-by: Zhipeng Liu <zhipengs.liu@intel.com>

commit 7dbe9962c41a5b1f1530acdae0fb1c9daabac0a5
Author: Robert Church <robert.church@windriver.com>
Date:   Thu Jan 16 03:10:34 2020 -0500

Use platform::network::routes::runtime to update routes
    
    Use the new route update runtime puppet class to make route updates for
    a specific host interface.
    
    This class has been introduced to only perform the minimum amount of
    work specific to route manipulation.
    
    Update the route, address, and sriov resource classes to match up with
    the network.pp manifest restructuring done to introduce the class.
    
    Change-Id: I56943f382b5ebb5ae52865aad3b05c1720b6d794
    Depends-On: Ieba501a6bd86164599eff97b9fe73d847740df68
    Story: 2007101
    Task: 38153
    Signed-off-by: Robert Church <robert.church@windriver.com>

commit 51563b30d3b78fa6d22ff17418d5d4a40dfddd65
Author: Robert Church <robert.church@windriver.com>
Date:   Thu Jan 16 02:58:09 2020 -0500

Provide the host UUID when updating a route on a host interface
    
    Optimize updating routes on a host's network interface by sending the
    host UUID with the config data when requesting the route to be updated.
    
    Without providing the host UUID all personalities will receive the route
    update request and apply a manifest regardless of if the interface is
    actually present on the host.
    
    With the host UUID provided the sysinv agent can skip the puppet apply
    if the host's UUID doesn't match.
    
    Change-Id: I3ed0fd657a1d0a4fafd4ff15c3b81da0cb8503e0
    Story: 2007101
    Task: 38145
    Signed-off-by: Robert Church <robert.church@windriver.com>

commit 11929c129e6417b60297e36ab80d0836ce205366
Author: John Kung <john.kung@windriver.com>
Date:   Thu Jan 16 15:57:47 2020 -0500

Allow versioned Armada operators to be loaded
    
    Enable the uploading of newer version of Armada manifest operator
    for an application by checking for a suffix and overriding with the
    latest numerically versioned suffix.
    
    Change-Id: Ie1e2d9f68743236194ae184668cfbe037f6c415e
    Partial-Bug: 1859882
    Signed-off-by: John Kung <john.kung@windriver.com>

commit c22e5787c8308b61bb2d5feccd12e35110dcb558
Author: Thomas Gao <Thomas.Gao@windriver.com>
Date:   Thu Jan 16 14:38:32 2020 -0500

Verified types of errors for sriov test suites.
    
    Verification is done by checking whether certain messages
    are shown in the response.json['error_message']. The original
    error message "Unable to provision pci-sriov interface
    without configured mgmt interface" is promptly caught, causing
    the test to fail, as expected.
    
    Since the intended error message is supposed to be something
    that's not "Unable to provision pci-sriov interface without
    configured mgmt interface," I added mgmt interface
    configuration so the test won't fail.
    
    Closes-Bug: 1860030
    
    Change-Id: I6a2f11d6cf5b37166fc44b722deab47e696f9fd6
    Signed-off-by: Thomas Gao <Thomas.Gao@windriver.com>

commit eda1dc8de4984dbd7e9ee3ad0cf3184468b89b6f
Author: Al Bailey <Al.Bailey@windriver.com>
Date:   Thu Nov 21 14:57:31 2019 -0600

Update sysinv flake8 to use python3
    
    Running flake8 in python3 exposes bugs that will
    impact the conversion to python3.
    
    This also adjusts the unit tests to run py36
    rather than py35.
    
    Some of the notable issues are:
     - cmp operator has been deprecated
     - file operator has been deprecated
        (replaced by call to open)
    
    The behaviour in python2 has not changed.
    
    For python3 unit tests should be added for:
     -  sysinv/agent/lldp/plugin.py  (Key)
    sort in python3 expects a different method (__lt__)
    
     -  sysinv/conductor/manager.py (evaluate_app_reapply)
    The contents from file open when passed to the md5 routine
    should fail unless opened with utf-8 encoding
    ie:
    TypeError: Unicode-objects must be encoded before hashing
    
    Change-Id: Ifcec7d5539cba2350649f56057db0f5e78ef279d
    Story: 2007082
    Task: 38130
    Signed-off-by: Al Bailey <Al.Bailey@windriver.com>

commit 73b4a423b5e8b8f9852646a7bed5105620974152
Author: Mingyuan Qi <mingyuan.qi@intel.com>
Date:   Tue Jan 14 13:03:15 2020 +0800

Add unit test for is_app_aborted in AppOperator
    
    Change-Id: Id33170b45ac2713300dd0befee6d916e001b5276
    Story: 2007082
    Task: 38099
    Depends-On: https://review.opendev.org/702341
    Signed-off-by: Mingyuan Qi <mingyuan.qi@intel.com>

commit 27cccd6eaa5eef0ff8569d6cd64d8aef17acf6e9
Author: Bin Yang <bin.yang@intel.com>
Date:   Tue Jan 14 17:26:53 2020 +0800

Add unit tests for platform_interfaces RPC EP.
    
    Story: 2007082
    Task: 38109
    
    Change-Id: I5015f3c28ad02a787886b13dd31bd94d9ea15073
    Signed-off-by: Bin Yang <bin.yang@intel.com>

commit 371c2e2c561d7859122663922df3ca1a98f9da69
Author: Bin Yang <bin.yang@intel.com>
Date:   Tue Jan 14 15:35:18 2020 +0800

Add unit tests for ilvg_get_nova_ilvg_by_ihost RPC EP.
    
    Story: 2007082
    Task: 38106
    
    Change-Id: I7fc0ae94eece53ce85617db8b034589b27c378f3
    Signed-off-by: Bin Yang <bin.yang@intel.com>

commit 4da9854454303632d166545ccf9e8153b966dcaa
Author: Bin Yang <bin.yang@intel.com>
Date:   Tue Jan 14 14:22:47 2020 +0800

Add unit tests for mtc_host_add RPC EP.
    
    Story: 2007082
    Task: 38104
    
    Change-Id: I51685ff372b8f95c2f8272524271d0ddbaf6a759
    Signed-off-by: Bin Yang <bin.yang@intel.com>

commit 7cd87ab8db3be81a1971e58dd99b293277f0f5f6
Author: Bin Yang <bin.yang@intel.com>
Date:   Tue Jan 14 13:45:36 2020 +0800

Add unit tests for vim_host_add RPC EP.
    
    Story: 2007082
    Task: 38101
    
    Change-Id: I59ca30d159c2779f27f8cfc3cb3d080b07c52ffd
    Signed-off-by: Bin Yang <bin.yang@intel.com>

commit 314f2a9a8c909781fd54409153c1286bae70223a
Author: Sun Austin <austin.sun@intel.com>
Date:   Wed Jan 15 15:12:05 2020 +0800

Add unit test for reapply in AppOperator
    
    Story: 2007082
    Task: 38122
    
    Change-Id: Ie9eea11ba384c96f0f2f78a5d7f99ee222fafa2c
    Signed-off-by: Sun Austin <austin.sun@intel.com>

commit 546e232848a1e6b133b519a6a79dbf11ca338bfc
Author: Sun Austin <austin.sun@intel.com>
Date:   Tue Jan 14 15:56:10 2020 +0800

Add unit test for get_appname and is_app_active in AppOperator
    
    Change-Id: I4e035370c5962aa445cc20b2afbe1ed8d229b1c9
    Story: 2007082
    Task: 38103
    Signed-off-by: Sun Austin <austin.sun@intel.com>

commit 343d1d89f2f1b3f21d610a06b9f21c1dff75c0ab
Author: Mingyuan Qi <mingyuan.qi@intel.com>
Date:   Tue Jan 14 16:06:59 2020 +0800

Add unit test for deactivate in AppOperator
    
    Change-Id: Ia3665a0ef9775431fe183395d2b1ca1c923b679c
    Story: 2007082
    Task: 38107
    Depends-On: https://review.opendev.org/#/c/702341/
    Signed-off-by: Mingyuan Qi <mingyuan.qi@intel.com>

commit 421e1ff65847e9cd79cdf03c5b2deddef6a3f32d
Author: Al Bailey <Al.Bailey@windriver.com>
Date:   Tue Jan 14 14:15:14 2020 -0600

Refactor unit test API example to use unit test mixins
    
    The test_pv file is being used as a reference when writing
    new API unit tests, and has been refactored to use the mixin
    classes that other unit tests utilize.
    
    This should allow other API configurations and permutations
    to be more easily testable.
    
    Change-Id: I3f1c730459cc3e08ba9ce98e28afa1ec8260db16
    Story: 2007082
    Task: 38112
    Signed-off-by: Al Bailey <Al.Bailey@windriver.com>

commit 140d0aa47ac158b3a0475bfc0840a1e57d8da8c3
Author: Joseph Richard <joseph.richard@windriver.com>
Date:   Mon Jan 13 11:00:08 2020 -0500

Add unit tests for network api
    
    Story: 2007082
    Task: 38023
    Change-Id: Iee6f3ded006c29e41f9ca0928e3bcf261c25a70c
    Signed-off-by: Joseph Richard <joseph.richard@windriver.com>

commit ca310aaa7b38e4e79f64882d3e410d6cae5cc7f7
Author: Mingyuan Qi <mingyuan.qi@intel.com>
Date:   Tue Jan 14 15:44:14 2020 +0800

Add unit test for activate in AppOperator
    
    Change-Id: I5e5b0a19222a214225c3ea24d523f458f8f3c14e
    Story: 2007082
    Task: 38100
    Depends-On: https://review.opendev.org/#/c/702341/
    Signed-off-by: Mingyuan Qi <mingyuan.qi@intel.com>

commit 399713200318a97fd6977f35f79a17168edf113b
Author: Mingyuan Qi <mingyuan.qi@intel.com>
Date:   Tue Jan 14 12:10:11 2020 +0800

Add test class for kube_app AppOperator
    
    Add test class and setup method.
    Mock method/operator will be added in individual commits which use
    that method or operator.
    
    Story: 2007082
    Task: 37998
    
    Change-Id: I60a4b235417f288adb4a13577b9779d79169a233
    Signed-off-by: Mingyuan Qi <mingyuan.qi@intel.com>

commit be3208886a2a2c69248e923eab38e4c73e35c43d
Author: Don Penney <don.penney@windriver.com>
Date:   Tue Jan 14 20:37:52 2020 +0000

Revert "configure kubernetes to use containerd as CRI"
    
    This reverts commit c229d8ac7e893ecce406cb15c8d1e35988e101e2.
    
    Reverting due to https://bugs.launchpad.net/starlingx/+bug/1859686
    
    Change-Id: I6534fb83c202827aaf1365d92ea6acbf4e531d84

commit 6258c07dde04934a085b7b5ccb5a33980b561e91
Author: Al Bailey <Al.Bailey@windriver.com>
Date:   Fri Jan 10 14:05:19 2020 -0600

Adding pylint checking to the sysinv unit tests
    
    pylint checks were excluding the unit tests folder.
    It is now included.
    
    The following pylint errors needed to be fixed in the unit tests
     - wildcard import (tests.db)
     - duplicate-key (fornodeid was listed twice in one test structure)
     - function-redefined (test_storage_tier.py)
     - dangerous-default-value (using a list as a default)
     - no-member (rename a class to end in Mixin to avoid these checks)
    
    Change-Id: I1095d54542e1e7b3d07865e1e7b4cd46f14dc407
    Story: 2007082
    Task: 38081
    Signed-off-by: Al Bailey <Al.Bailey@windriver.com>

commit 745aec251eb1247ba75d63465f734e49421db24b
Author: abdelfattah saafan <abdelfattah.saafan@windriver.com>
Date:   Mon Jan 13 09:47:09 2020 -0600

stx-monitor app: enhanced missing elastic label error message
    
    When attempting to apply stx-monitor app without the required elastic
    labels, user will get an error message detailing what is missing.
    This label was enhanced to be more readable and provide instructions
    on how to apply the missing labels.
    
    Moved utility methods up to the super class LabelTestCase. Also
    moved host definitions to each of the test classes since
    LabelAssignTestCase require hosts to be locked and
    StxAppLabelsTest require hosts to be unlocked.
    
    Added the following enhanced logic:
    1. All controller nodes are required to have all 4 elastic labels
    2. At least one worker node should have elastic-master label.
    3. Add label value validation and print error message for invalid
    values.
    
    Change-Id: I0b2bc7b87d2e8bf78bfbba455d499c52e1a72349
    Closes-Bug: 1856763
    Signed-off-by: abdelfattah saafan <abdelfattah.saafan@windriver.com>

commit 238b196d3a9d244ce6e86f0fc7a2d14c2b3878d7
Author: Sun Austin <austin.sun@intel.com>
Date:   Thu Nov 21 21:23:00 2019 +0800

Additional condition is comparison when doing config check out of date
    
    This will fix below scenario:
    Controller-0 is installed and some configs were applied. Config
    f5d83753-c485-4221-b4aa-e986f5421d6d is the target config and requests
    boot.
    After controller-0 is unlocked and reboot.another request with uuid
    29c89b7e-51fe-40cb-8700-e22c83ed5fa9 is coming,then the config_target is
    set to a9c89b7e-51fe-40cb-8700-e22c83ed5fa9;
    Once 29c89b7e-51fe-40cb-8700-e22c83ed5fa9 was applied,the config should
    be cleared, but because 29c89b7e-51fe-40cb-8700-e22c83ed5fa9 is not same
    as a9c89b7e-51fe-40cb-8700-e22c83ed5fa9,the alarm can not be cleared.
    
    In this fix, tracking list for reboot config and clear it once config is
    applied. the out of date config will be clear if reboot config tracking
    is empty and config_applied w/ reboot config flag is equal config_target
    
    Closes-Bug: 1851874
    Change-Id: Iabeab338bc3fb4615cefcff9e4ae9402e4216321
    Signed-off-by: Sun Austin <austin.sun@intel.com>

commit af308b40d528db6c575880655158ec2254de1959
Author: Bart Wensley <barton.wensley@windriver.com>
Date:   Fri Jan 10 07:39:58 2020 -0600

Add sample unit test for helm data generation
    
    Adding unit test for nova_api_proxy helm data generation.
    
    Change-Id: Icd62ad584e2f3bb0332455ab8458cd8fa585ae09
    Story: 2007082
    Task: 37994
    Signed-off-by: Bart Wensley <barton.wensley@windriver.com>

commit 89e2975eb85132c4c1b2619319c5060deef0c3ee
Author: Al Bailey <Al.Bailey@windriver.com>
Date:   Wed Jan 8 11:42:52 2020 -0600

Add missing requirements for sysinv
    
    The sysinv code has imports for python components but does
    not have all of those specified as rpm dependencies.
    
    Adding:
     python-jsonpatch (used by several sysinv/api/controllers/v1/*)
     python-keystoneauth1 (used by sysinv/common/fm)
     python-keystonemiddleware (used by sysinv/api/acl)
     python-oslo-serialization (used by sysinv/conductor/manager)
     python-oslo-service (used by sysinv/common/wsgi_service)
     python-paste (used by api config file)
     python-psutil (used by sysinv/api/controllers/v1/host)
     python-requests (used by sysinv/common/ceph)
     python-retrying (used by sysinv/agent/manager)
     python-stevedore (used by sysinv/puppet)
     python-webob (used by sysinv/api/hooks)
    
    This is a protective commit, since this only works because other
    packages are pulling in the dependencies, and those packages can be
    updated or removed in the future, thus breaking sysinv.
    
    Change-Id: I9457e3817ae8d219ef0c582b0b0177510a8ec4b0
    Story: 2004515
    Task: 37985
    Signed-off-by: Al Bailey <Al.Bailey@windriver.com>

commit a6b821acad5f873ef3ca192e621ff5d46d591b49
Author: Kevin Smith <kevin.smith@windriver.com>
Date:   Wed Jan 8 14:34:04 2020 -0500

Elastic: lower elasticsearch curator limits.
    
    Lower the elasticsearch curator limits to take into account the
    elasticsearch cluster.routing.allocation.disk settings and increase
    the cronjob frequency to run every half hour.
    
    Change-Id: I7eee7d6ad8a1eb2210a4a9c10c8ff8782783c192
    Closes-Bug: 1858436
    Signed-off-by: Kevin Smith <kevin.smith@windriver.com>

commit b1d0e2caf392e90aed0819beca84cff4d2d8e888
Author: Al Bailey <Al.Bailey@windriver.com>
Date:   Mon Jan 6 11:49:19 2020 -0600

Deprecate sysinv.openstack.common.lockutils file
    
    Removing sysinv.openstack.common.lockutils
    The newer lockutils file is located in oslo_concurrency.
    Only unit tests were using the older lockutils.
    
    The following pylint error codes are no longer suppressed:
      E0611: no-name-in-module
      E1124: redundant-keyword-arg
    
    Change-Id: I060523571e94857beb7a65cef72bbb367aa1bbc5
    Story: 2006796
    Task: 37962
    Signed-off-by: Al Bailey <Al.Bailey@windriver.com>

commit bbbcb55d12748566012bc217d3ae60515109d948
Author: Al Bailey <Al.Bailey@windriver.com>
Date:   Fri Jan 3 09:26:00 2020 -0600

Remove sqlalchemy-migrate 0.7.3 patching
    
    This is similar to upstream submissions in 2013 such as
    https://review.opendev.org/#/c/56705/
    
    We currently build, test and ship with a newer version
    of sqlalchemy-migrate 0.11.0 so we do not need to handle
    0.7.3 workarounds and patches.
    
    This review also fixes: tests.db.sqlalchemy.test_migrations
    which would fail when tested in isolation, due to its environment
    not being initialized properly, but working if certain other tests
    were run first.
    
    Some skipped tests are now also enabled.
    
    Change-Id: I6ce73491f7a2471f25c7691dab8b53dc354619a2
    Story: 2004515
    Task: 37954
    Signed-off-by: Al Bailey <Al.Bailey@windriver.com>

commit c337d55590fc6fc93037baeb62200ca96879acdd
Author: marvin <weifei.yu@intel.com>
Date:   Thu Nov 21 18:42:45 2019 +0800

Monitor the datanetwork for non-OpenStack work node
    
    Update the lmon to support datanetwork interface monitoring
    and use collectd to control the alarm information. Now lmon
    will obtain the list of interfaces from /etc/lmon/lmon.conf
    which can be generated by puppet.
    
    Change-Id: Id886c935d54158eca44d6685f1e282c20367b9a4
    Story: #2002948
    Task: #37326
    Depends-on: https://review.opendev.org/#/c/694927
    Signed-off-by: marvin <weifei.yu@intel.com>

commit 063e398afe5abf81735eb4e43c70315b50a7b36a
Author: Joseph Richard <joseph.richard@windriver.com>
Date:   Wed Dec 18 14:15:01 2019 -0500

Skip generating hieradata for uninventoried host
    
    When _config_apply_runtime_manifest is run with force=True, it will
    generate hieradata for all hosts, even hosts that have yet to come
    online and are still uninventoried.  Because the node is still
    uninventoried, the hieradata will have an incomplete network config
    with only the loopback interface configured.
    When those hosts come online for the first time, they will mount
    /opt/platform and check for their hieradata, and because it exists,
    apply their puppet manifests with that hieradata, which will bring down
    all non-loopback interfaces and make the node go offline and become
    unreachable.
    This commit adds a check to only generate hieradata for inventoried
    hosts, even when force=True.
    
    Closes-bug: 1856617
    Change-Id: I37c611e9a1047d5651844c71351f40d939977b6f
    Signed-off-by: Joseph Richard <joseph.richard@windriver.com>

commit a9686eccf93dbbb89f8e460aa0fd449e688f957d
Author: Joseph Richard <joseph.richard@windriver.com>
Date:   Fri Dec 13 15:01:58 2019 -0500

Prevent provisioning sriov without mgmt iface
    
    This commit adds a semantic check to prevent provisioning an sriov
    interface without a mgmt interface configured.  This is necessary to
    prevent an invalid network config being generated and applied, which
    causes the node to lose connectivity over mgmt and become unreachable.
    
    Closes-bug: 1853329
    Change-Id: I783447a2214a1b0f4d698ac20037f8f1e8083958
    Signed-off-by: Joseph Richard <joseph.richard@windriver.com>

commit b5c78afe287ae5373a3ea8e7fe749960a7ffb4eb
Author: Bart Wensley <barton.wensley@windriver.com>
Date:   Fri Dec 20 10:50:33 2019 -0600

Add kubernetes health checks
    
    Adding kubernetes health checks when a kubernetes upgrade is
    started to ensure the kubernetes cluster is healthy. Also
    adding a new "system health-query-kube-upgrade" command to
    do the health checks without starting a kubernetes upgrade.
    
    Change-Id: Ib3a491e25a6cb523cfd9f08e74879e5c0a25d4f7
    Story: 2006781
    Task: 37583
    Depends-On: https://review.opendev.org/#/c/700197/
    Signed-off-by: Bart Wensley <barton.wensley@windriver.com>

commit 4501e6ee8683821480feb6feffd663f93fb806b8
Author: Al Bailey <Al.Bailey@windriver.com>
Date:   Tue Dec 17 14:55:59 2019 -0600

Removing mox from sysinv in StarlingX
    
    Mox is only being used by unit tests.
    
    There is an openstack governance directive to remove mox:
        https://storyboard.openstack.org/#!/story/2001546
    
    Many unit tests were able to be updated based on
    similar work in ironic. Ex:
        https://review.opendev.org/#/c/50729/
    
    Removed two oslo incubator files under sysinv.openstack.common.fixture:
        mockpatch (unused)
        moxstubout
    
    Change-Id: Ie90a9c9e5a3028575d016f6eccf0ad118495d482
    Story: 2006796
    Task: 37851
    Signed-off-by: Al Bailey <Al.Bailey@windriver.com>

commit e04886ffb9f01e2f217f6516729dd631c50588c9
Author: Stefan Dinescu <stefan.dinescu@windriver.com>
Date:   Mon Dec 9 17:19:48 2019 +0200

Remove glance storage backend
    
    The glance storage-backend was still present for legacy
    reasons. Since openstack is now a containerized application,
    this glance backend is no longer needed.
    
    Change-Id: Ie00b7c136ce3da043fe3b034b02230de022f8123
    Closes-bug: 1848198
    Signed-off-by: Stefan Dinescu <stefan.dinescu@windriver.com>

commit c229d8ac7e893ecce406cb15c8d1e35988e101e2
Author: Shuicheng Lin <shuicheng.lin@intel.com>
Date:   Fri Sep 27 23:43:40 2019 +0800

configure kubernetes to use containerd as CRI
    
    docker and containerd have different local filesystem for container
    image. For containerd, crictl is used to pull the image instead of
    docker client for docker. Due to containerd/crictl doesn't support
    push image to registry, docker client is kept to push image to
    local registry. And image pulled by docker need be removed after
    push to registry.
    
    Story: 2006145
    Task: 36747
    Depends-On: https://review.opendev.org/685211
    
    Change-Id: Id376c7e41a92aa69571b391020505934b6cdc23b
    Signed-off-by: Shuicheng Lin <shuicheng.lin@intel.com>

tags:

added: in-f-centos8

StarlingX

worker nodes applying puppet manifest before unlock

Bug Description

Other bug subscribers

Bug attachments

Remote bug watches