Upgrades are not able to add new keystone users/services/endpoints

Bug #1886064 reported by Bart Wensley
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
StarlingX
Fix Released
Medium
Andy

Bug Description

Brief Description
-----------------
When we upgrade to a new release of StarlingX, we often introduce new services which require new keystone users/services/endpoints. In past releases, the new keystone users/services/endpoints could be added during the upgrade by updating the openstack::keystone::upgrade class in stx-puppet/puppet-manifests/src/modules/openstack/manifests/keystone.pp. However, this was broken several releases ago (see the comments in this class) and now there is no way to do this.

Severity
--------
Critical: No way to add new keystone users/services/endpoints during an upgrade.

Steps to Reproduce
------------------
See above

Expected Behavior
------------------
See above

Actual Behavior
----------------
See above

Reproducibility
---------------
Reproducible

System Configuration
--------------------
All configurations

Branch/Pull Time/Commit
-----------------------
All StarlingX releases to date are impacted.

Last Pass
---------
This was possible prior to the initial StarlingX open source release.

Timestamp/Logs
--------------
N/A

Test Activity
-------------
Developer Testing

Workaround
----------
None

CVE References

tags: added: stx.update
Ghada Khalil (gkhalil)
tags: added: stx.5.0
Changed in starlingx:
importance: Undecided → Medium
status: New → Triaged
Ghada Khalil (gkhalil)
Changed in starlingx:
assignee: nobody → Andy (andy.wrs)
Revision history for this message
Andy (andy.wrs) wrote :

When upgrade.pp is applied during controller-1 upgrade, keystone::upgrade is run and fail. In latest puppet.log on controller-1 we can see:

8053 2020-11-17T10:04:33.272 ^[[0;36mDebug: 2020-11-17 10:04:33 +0000 Exec[upgrade token issue](provider=posix): Executing 'openstack --os-username admin --os-password Li69nux* --os-auth-url http://127.0.0.1:5000/v3 --os-project-name admin --os-user-domain-name Default --os-project-domain-name Default --os-interfaceinternal --os-identity-api-version 3 token issue -c id -f value > /etc/keystone/upgrade_token'^[[0m
8054 2020-11-17T10:04:33.274 ^[[0;36mDebug: 2020-11-17 10:04:33 +0000 Executing: 'openstack --os-username admin --os-password Li69nux* --os-auth-url http://127.0.0.1:5000/v3 --os-project-name admin --os-user-domain-name Default --os-project-domain-name Default --os-interface internal --os-identity-api-version 3 token issue -c id -f value > /etc/keystone/upgrade_token'^[[0m
8055 2020-11-17T10:04:35.650 ^[[mNotice: 2020-11-17 10:04:35 +0000 /Stage[main]/Keystone/Exec[upgrade token issue]/returns: The request you have made requires authentication. (HTTP 401) (Request-ID: req-e4834fcc-953c-489b-b7f1-e80738e12fef)^[[0m
8056 2020-11-17T10:04:35.653 ^[[1;31mError: 2020-11-17 10:04:35 +0000 openstack --os-username admin --os-password Li69nux* --os-auth-url http://127.0.0.1:5000/v3 --os-project-name admin --os-user-domain-name Default --os-project-domain-name Default --os-interface internal --os-identity-api-version 3 token issue -c id -f value > /etc/keystone/upgrade_token returned 1 instead of one of [0]

When checking keystone database, it is found that the DB is empty, there are not users, projects etc.

The cause of this is that:
In /usr/lib64/python2.7/site-packages/controllerconfig/upgrades/management.py:

 44 if sysinv_constants.SERVICE_TYPE_IDENTITY not in shared_services: ====> identity is no longer a shared service in DC
 45 UPGRADE_DATABASES += ('keystone',)
 46 UPGRADE_DATABASE_SKIP_TABLES.update({'keystone': ('token',)})

The checking prevents identity db to be backed up. So data-migration doesn't migrate any data into keystone DB running on controller-1.

This issue is being tracked by: https://bugs.launchpad.net/starlingx/+bug/1904675

Andy (andy.wrs)
Changed in starlingx:
status: Triaged → In Progress
Revision history for this message
Andy (andy.wrs) wrote :
Changed in starlingx:
status: In Progress → Fix Released
Revision history for this message
Andy (andy.wrs) wrote :

The other review for this LP: https://review.opendev.org/c/starlingx/stx-puppet/+/763857
merged on 2020-11-25

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to stx-puppet (f/centos8)

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/c/starlingx/stx-puppet/+/792009

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on stx-puppet (f/centos8)

Change abandoned by "Chuck Short <email address hidden>" on branch: f/centos8
Review: https://review.opendev.org/c/starlingx/stx-puppet/+/792009

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to stx-puppet (f/centos8)

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/c/starlingx/stx-puppet/+/792013

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on stx-puppet (f/centos8)

Change abandoned by "Chuck Short <email address hidden>" on branch: f/centos8
Review: https://review.opendev.org/c/starlingx/stx-puppet/+/792013

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to stx-puppet (f/centos8)

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/c/starlingx/stx-puppet/+/792018

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on stx-puppet (f/centos8)

Change abandoned by "Chuck Short <email address hidden>" on branch: f/centos8
Review: https://review.opendev.org/c/starlingx/stx-puppet/+/792018

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to stx-puppet (f/centos8)

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/c/starlingx/stx-puppet/+/792029

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to stx-puppet (f/centos8)
Download full text (48.0 KiB)

Reviewed: https://review.opendev.org/c/starlingx/stx-puppet/+/792029
Committed: https://opendev.org/starlingx/stx-puppet/commit/2b026190a3cb6d561b6ec4a46dfb3add67f1fa69
Submitter: "Zuul (22348)"
Branch: f/centos8

commit 3e3940824dfb830ebd39fd93265b983c6a22fc51
Author: Dan Voiculeasa <email address hidden>
Date: Thu May 13 18:03:45 2021 +0300

    Enable kubelet support for pod pid limit

    Enable limiting the number of pids inside of pods.

    Add a default value to protect against a missing value.
    Default to 750 pids limit to align with service parameter default
    value for most resource consuming StarlingX optional app (openstack).
    In fact any value above service parameter minimum value is good for the
    default.

    Closes-Bug: 1928353
    Signed-off-by: Dan Voiculeasa <email address hidden>
    Change-Id: I10c1684fe3145e0a46b011f8e87f7a23557ddd4a

commit 0c16d288fbc483103b7ba5dad7782e97f59f4e17
Author: Jessica Castelino <email address hidden>
Date: Tue May 11 10:21:57 2021 -0400

    Safe restart of the etcd SM service in etcd upgrade runtime class

    While upgrading the central cloud of a DC system, activation failed
    because there was an unexpected SWACT to controller-1. This was due
    to the etcd upgrade script. Part of this script runs the etcd
    manifest. This triggers a reload/restart of the etcd service. As this
    is done outside of the sm, sm saw the process failure and triggered
    the SWACT.

    This commit modifies platform::etcd::upgrade::runtime puppet class
    to do a safe restart of the etcd SM service and thus, solve the
    issue.

    Change-Id: I3381b6976114c77ee96028d7d96a00302ad865ec
    Signed-off-by: Jessica Castelino <email address hidden>
    Closes-Bug: 1928135

commit eec3008f600aeeb69a42338ed44332228a862d11
Author: Mihnea Saracin <email address hidden>
Date: Mon May 10 13:09:52 2021 +0300

    Serialize updates to global_filter in the AIO manifest

    Right now, looking at the aio manifest:
    https://review.opendev.org/c/starlingx/stx-puppet/+/780600/15/puppet-manifests/src/manifests/aio.pp
    there are 3 classes that update
    in parallel the lvm global_filter:
    - include ::platform::lvm::controller
    - include ::platform::worker::storage
    - include ::platform::lvm::compute
    And this generates some errors.

    We fix this by adding dependencies between the above classes
    in order to update the global_filter in a serial mode.

    Closes-Bug: 1927762
    Signed-off-by: Mihnea Saracin <email address hidden>
    Change-Id: If6971e520454cdef41138b2f29998c036d8307ff

commit 97371409b9b2ae3f0db6a6a0acaeabd74927160e
Author: Steven Webster <email address hidden>
Date: Fri May 7 15:33:43 2021 -0400

    Add SR-IOV rate-limit dependency

    Currently, the binding of an SR-IOV virtual function (VF) to a
    driver has a dependency on platform::networking. This is needed
    to ensure that SR-IOV is enabled (VFs created) before actually
    doing the bind.

    This dependency does not exist for configuring the VF rate-limits
    however. There is a cha...

tags: added: in-f-centos8
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.