Reviewed: https://review.opendev.org/c/starlingx/distcloud/+/796528 Committed: https://opendev.org/starlingx/distcloud/commit/4c5344f8765b372cb84d2b1181589c16db2ae6e4 Submitter: "Zuul (22348)" Branch: f/centos8 commit cb979811017bd193fc1f06e53bb7830fd3184859 Author: Yuxing Jiang Date: Wed Jun 9 11:11:27 2021 -0400 Format the IP addresses in payload before adding a subcloud The IPv6 addresses can be represented in multiple formats. As IP addresses are stored as text in database, ansible inventory and overrides, this commit converts the IP addresses in payload to standard text format of IPv6 address during adding a new subcloud. Tested with installing and bootstrapping a new subcloud(RVMC configured) with the correct IPv6 address values, but with unrecommended upper case letters and '0'. The addresses are converted to standard format in database, ansible inventory and overrides files. Partial-Bug: 1931459 Signed-off-by: Yuxing Jiang Change-Id: I6c26e749941f1ea2597f91886ad8f7da64521f0d commit 2cf5d6d5cef0808c354f7575336aec34253993b3 Author: albailey Date: Thu May 20 14:19:24 2021 -0500 Delete existing vim strategy from subcloud during patch orch When dcmanager creates a patch strategy, if a subcloud has an existing vim patch strategy, it will attempt to re-use that strategy during its patching phase, which may result in an error. This commit deletes the existing vim patch strategy in a subcloud, if it exists, so it can be re-created. If the strategy cannot be deleted, orchestration fails. Change-Id: Id35ef26ed3ddae6d71874fc6bac11df147f72323 Closes-Bug: 1929221 Signed-off-by: albailey commit 9e14c83f0162549a2a94cb8bc1e73dbc4f4d4887 Author: albailey Date: Tue Jun 1 14:37:14 2021 -0500 Adding activation retry to upgrade orchestration When performing an activation, the keystone endpoints may not be accessible in the subcloud due to the asyncronous way that cert-mon can trigger a restart of keystone. This would have occasionally resulted in the upgrade activation failing to be initiated, and orchestration needing to be invoked again to resume. This 'hack' adds retries and sleeps to the initial activation action. Change-Id: Ic757521dec7bdc248a51a70b5463caafe7927360 Partial-Bug: 1927550 Signed-off-by: albailey commit bb604c0a9b872efd65fa45f1e2269995818c6262 Author: Tee Ngo Date: Thu May 27 22:17:16 2021 -0400 Fix subcloud show --detail command related issues If the subcloud is offline, the command stalls and eventually returns the "ERROR (app)" output. If the subcloud is online, the oam_floating_ip info is excluded from the output when the subcloud id instead of subcloud name is specified. This commit fixes both of the above issues. Closes-Bug: 1929893 Change-Id: I995591368564539b0e6af185b1adba2db73e0e46 Signed-off-by: Tee Ngo commit f6a7ef1262d7e62e98cbcf1cefa45ce635b04e20 Author: albailey Date: Thu Apr 22 09:01:41 2021 -0500 Cleanup Distributed Cloud Orchestration unit tests We have a DB setup that we can populate, so the subcloud DB values should not be mocked, and instead we should verify they are being properly set. Added unit tests for these states that had no tests: - kube upgrade 'updating patches' - kube upgrade 'deleting vim strategy' - kube upgrade 'upgrade simplex' Relocate a unit test that was in an old directory and was not being executed: - upgrade 'deleting load' Some tests were incorrectly changing the name of an object rather than its state attribute. hacking was incorrectly being specified directly within tox.ini rather than the test-requirements.txt file. FakeSubcloud is no longer being used anywhere. Story: 2008137 Task: 42409 Change-Id: I9d43a2b0f1923cbd7d469965968062503a58234d Signed-off-by: albailey commit d7986567d0111adeb175a9c106ad1f72b2e0c052 Author: Jessica Castelino Date: Thu May 20 13:43:33 2021 -0400 DC dcorch log cleanup In order to increase retention period of dcorch logs, some info logs are moved to debug level. Change-Id: I8091b7fea6f0f8c3834802d03c438252529137ae Signed-off-by: Jessica Castelino Closes-Bug: 1929111 commit 9f5b85c41c2d8aec2e550f187a8b40a7525033b0 Author: albailey Date: Wed May 19 14:12:07 2021 -0500 Specify the nodeset for zuul jobs The py2.7 jobs need to specify xenial The py3.6 jobs need to specify bionic The focal zuul nodes only have python 3.8 installed in them Zuul targets that invoke a generic python3 interpreter such as pep8 is not specified. The copyright date was updated in order to trigger the zuul jobs, as a no-delta type of change. Change-Id: If5352228a018f3147ae78172ae625066150c0a95 Partial-Bug: 1928978 Signed-off-by: albailey commit 6b87e180298b19f5cfc441620cfdc546696ffb58 Author: Jessica Castelino Date: Wed May 19 10:02:25 2021 -0400 Add support for k8s-upgrade-completed event notification This will be used by sysinv to notify dcmanager that the kubernetes upgrade has been completed so that it can trigger a kubernetes audit of all subclouds. Change-Id: I2ff48c939e808c3576a5739fa1dc37906e6a3a4e Signed-off-by: Jessica Castelino Closes-Bug: 1928864 Depends-On: https://review.opendev.org/c/starlingx/config/+/792043 commit 077ab926e388d477a788e3cf306e2b7443c3927e Author: albailey Date: Fri May 14 13:48:23 2021 -0500 Updating some of the dcmanager and audit logs On startup of any process, the entire configuration was being logged, which was we were logging the entire configuration to the file (almost 200 lines per process). - Lowering that log to DEBUG. - Added an extra 'Starting...' log entry to help indicate when a process start/restart occurs. The update_subcloud_endpoint_status log was not including enough information, as it is called several times in a row for each endpoint. Two audit logs that are created every 30 seconds have been reduced from info to debug. Closes-Bug: 1928335 Change-Id: I7ada5abf87c2f28f5826c02345f8dd3197eae665 Signed-off-by: albailey commit 84dc98f5df97a8dba050a96e4cedace0e9419d8f Author: Melissa Wang Date: Wed May 5 08:51:22 2021 -0400 Fix for subcloud upgrade activation failure This change adds exception-handling in case the haproxy is reset during upgrade activation so that it will retry the activation step Closes-Bug:1927550 Change-Id: I678d50df3d8866518356d8da95b4742f0072c7fe Signed-off-by: Melissa Wang commit 8d16d245504039b7e59b5d7a083b3b9d28b358e9 Author: albailey Date: Thu May 13 08:36:52 2021 -0500 Reset the force patch audit flag after it is triggered The force patch audit flag could be set, but never cleared. This would result in patch audits being run during every audit loop, as well as any derived audits such as software load. This change ensures that the force flag is cleared when a patch audit is invoked. The force patch audit flag is used by other audits such as the audit_load, so it needs to be reset after those evaluations. Change-Id: I833d157a616916a9ee37553b1f4cf3582caa8862 Closes-Bug: 1928333 Signed-off-by: albailey commit 06d48e2ede3991d623089455e513918e07cf7ba4 Author: Yuxing Jiang Date: Thu May 6 13:51:32 2021 -0400 Extend the ansible timeout of the rehome playbook On a sub cloud that can no longer talk to its system controller, ansible cannot escalate the root privilege in the sub cloud until connecting to the LDAP server times out. This commit overrides the ansible default timeout(10s) to 180s of the re-home playbook, to prevent the re-home playbook failed by privilege escalation timeout before correcting the LDAP server address in a sub cloud. Tested: Re-homed a subcloud after powering off the previous system controller and deleting the static route to the system controller. Story: 2008774 Task: 42408 Change-Id: I65f9e0b7e242ef53a6fadcbab801ff39778b3d8d Signed-off-by: Yuxing Jiang commit b57f5ef65bea69db8563683c13d2299cccc35845 Author: Jessica Castelino Date: Wed May 5 10:50:58 2021 -0400 Re-initialize RegionOne keystone client properly after subcloud add When a new subcloud is added, we expect IndexErrors while trying to fetch the subclouds identity auth URL. At this time, the RegionOne keystone client is also refreshed (by setting it to None). The next subcloud that tries to access the master keystone client cannot properly handle this and its availability is set to None for a short duration. This commit fixes the issue described above. Change-Id: I3139108b119aa6b6ba20277d829c14cc61e9f88c Signed-off-by: Jessica Castelino Closes-Bug: 1927229 commit 17b5505d9ea9b149cf28236be3c1b4c263a89ffb Author: Tao Liu Date: Mon May 3 12:32:53 2021 -0400 Fix Sub clouds going offline due to auth failure This update contains the following changes that prevent subclouds going offline due to authentication failure: 1. The os region client cache is cleared when a new keystone client is created. The os region client will be re-created using the new keystone session. 2. When the user's access info (such as role id) is changed create new keystone client and os region clients. This could happen after system controller keystone role ids were synced to subclouds 3. Remove get_admin_backup_session that was only required when upgrading to stx 4.0. 4. Increase AVAIL_FAIL_COUNT_TO_ALARM to 2 as we don't want to alarm first failure since there are cases where we expect a transient failure in the subcloud (e.g. haproxy process restart to update certificates) Tested on DC-6: 1. Adding 50 subclouds twice 2. Soaking the fix over the weekend Closes-Bug: 1927007 Signed-off-by: Tao Liu Change-Id: I86fdc9a2f062409e704bdfac2119dc488123f7de commit 8acd9699acec23c6ce4fe97bb1ea790d38032951 Author: Jessica Castelino Date: Wed Apr 28 16:36:08 2021 -0400 Fix: Subcloud shows (patching) out-of-sync on the system controller The patching logic is modified to mark the subcloud as in-sync when the patches that are applied on the system controller have been committed on a subcloud. Change-Id: Ic56f0490cc0f7728ace721a2436d602a03ca1328 Signed-off-by: Jessica Castelino Closes-Bug: 1926635 commit 70a3475c73c851c4179d809d8e1b28ba46508683 Author: Jessica Castelino Date: Mon Apr 19 17:18:18 2021 -0400 Handling retry attempts of duplex subcloud upgrade strategy This commit supports retry attempts for duplex subcloud upgrade strategy when upgrade states are "data-migration", "data-migration-complete" or "data-migration-failed". Change-Id: I919228126a4960f712092c26b3abbf12e2de8ccb Signed-off-by: Jessica Castelino Story: 2008055 Task: 42304 commit f5890513348e889824b61fda8867b14c7d0c4cde Author: Tee Ngo Date: Mon Apr 19 12:22:22 2021 -0400 Handle upgrade failure following unlock timeout In this commit: - Set the subcloud deploy status to DEPLOY_STATE_DATA_MIGRATION_FAILED if the subcloud host unlock/reboot times out so that the subcloud can be audited and handled correctly in pre_check upon upgrade retry. - Update pre_check to advance to activating step if deploy status is DEPLOY_STATE_DATA_MIGRATION_FAILED but the subcloud is online and passes health check. - Separate simplex and duplex pre_check logics for readability. - Correct a dcorch log that was not updated when the platform admin userid was changed from wrsroot to sysadmin. Closes-Bugs: 1925053 Change-Id: Ie7f23a4f8de4e5dbcacfd7d625e15fe25d30e499 Signed-off-by: Tee Ngo commit a9157c51d5562eb395815d4653d8e7d39b38920e Author: Tee Ngo Date: Fri Apr 16 11:00:26 2021 -0400 Filter out skippable alarms in precheck Subcloud online checks are now skipped if the subcloud is already in the 'migrated' state. The precheck also skips upgrade alarm as well as the host lock alarm if upgrade has started. In addition, the commit includes the code to handle bmc_password with None value which the previous commit (69a74499884a8a73a3b705ce074f46202c4aa278) did not handle. Closes-Bug: 1924774 Change-Id: Ifdfe1e44a34c2c2561c299a82d7178dce6063daf Signed-off-by: Tee Ngo commit 6c986880b7caa6687ffceaa1f3888d989756eb3e Author: albailey Date: Wed Apr 14 10:03:30 2021 -0500 Kube audit should report out-of-sync if a kube upgrade exists A subcloud can have all its kubernetes components upgraded, but there can still be a kubernetes upgrade object. Audit will now report the existance of that object as meaning the subcloud is out of sync. Similar logic already exists for regular upgrades. Change-Id: If84c24bdd213c18fefde3752b1798f67aad560ae Story: 2008137 Task: 42265 Signed-off-by: albailey commit ef0123d0b48136edcef794ec03cad776100603e3 Author: Tee Ngo Date: Tue Apr 13 21:25:29 2021 -0400 Add restore related deploy states to auditable list In this commit, relevant restore related deploy states are added to auditable list so the subcloud can be set as offline while undergoing remote restore. Closes-Bug: 1923688 Change-Id: Ib415327478c5ab45b61726f3ce2d6e88be0654cd Signed-off-by: Tee Ngo commit de96cf465b65661a60a8070f4293348fb962055d Author: Tee Ngo Date: Mon Apr 12 16:33:32 2021 -0400 Restore upgrade retry of offline simplex subcloud Code changes to support upgrade orchestration for duplex subclouds broke the upgrade retry functionality for offline simplex subclouds. This commit fixes it. Closes-Bug: 1923498 Change-Id: I92d31df7ae8396293b302d357655470b447d143c Signed-off-by: Tee Ngo commit 119dbaf87bb13a37e62b32d14dbd00776a1919db Author: Yuxing Jiang Date: Mon Feb 1 13:45:17 2021 -0500 Add a migrate option for rehoming a subcloud This commit adds a "migrate" option in dcmanager subcloud add API. With this option, the dcmanager can invoke a rehome_subcloud playbook to migrate a subcloud under another central cloud. The playbook will be installed under: /usr/share/ansible/stx-ansible/playbooks/rehome_subcloud.yml. The subcloud info will store in dcmanager database. The keystone password of "smapi" is required during the subcloud migration, so it is requested from keyring and added into the subcloud's overrides file. Tested: Happy path: On a system controller, trigger a successful run of the rehome_subcloud playbook with command: dcmanager subcloud add --migrate --bootstrap-address --bootstrap-values bootstrap-values.yml In the boostrap-values.yml the systemcontroller_gateway_address should be set under the management subnet. The subcloud info is added into the dcmanager database. And its deploy_status is changed in the sequence: pre-rehome-> rehoming-> complete Unhappy path: 1 On a system controller, trigger a failed run of the rehome_subcloud playbook with the correct bootstrap values. The subcloud info is also stored in the dcmanager database. But the deploy_status is changed into rehome-failed in the end. 2 The systemcontroller_gateway_address is not under the management subnet. The rehome_subcloud playbook is not triggered, and a notice indicates the problem is prompted. 3 Create an error during creating the intermediate ca cert in cert-manager, subcloud's deploy status changes to "rehome-prep-failed" in the end. Depends-On: https://review.opendev.org/c/starlingx/ansible-playbooks/+/780101 Story: 2008774 Task: 42150 Change-Id: I758e102d386a63c7ed318bcc91d66e11d3f56299 Signed-off-by: Yuxing Jiang commit 69a74499884a8a73a3b705ce074f46202c4aa278 Author: Tee Ngo Date: Thu Apr 8 14:40:30 2021 -0400 Encode bmc_password received from the subcloud host If the subcloud is configured to store bmc secret in its barbican database, ensure the password is base64 encoded before passing the install data to rvmc. Closes-Bug: 1923073 Change-Id: I9639c8e09acb2d78013932d4123511b354bb04fb Signed-off-by: Tee Ngo commit 8dcb56bb493078f34fec3c4f07786112cfbfacdb Author: Jessica Castelino Date: Fri Mar 19 17:18:29 2021 -0400 Reduce keystone load for offline subclouds When a large number of subclouds are offline (e.g. powered down), the dcmanager-audit generates significant load on the keystone services in the System Controller when it periodically (roughly once per minute) creates a keystone token and then retrieves the endpoints. This causes CPU usage to spike. In order to address the issue, RegionOne client is cached in EndpointCache. Change-Id: I17db5cd0107457f07fe14febfec23da5290153de Story: 2007267 Task: 42082 Signed-off-by: Jessica Castelino commit 4a49932841abd3ca4f70eca500b37e60ba3ed746 Author: albailey Date: Mon Apr 5 15:01:53 2021 -0500 Only install yamllint in py3 env A new yamllint was released which dropped py2 support. Zuul servers that have updated their caches will attempt to install that version, as it is not clamped by upper constraints. This also attempts to not use site packages. Partial-Bug: 1922590 Change-Id: I8b2776f4c12447a66743a8b2626aa914969dcd2a Signed-off-by: albailey commit b0b3cfe07b591648a12b669e36e6cc218d5328c3 Author: Teresa Ho Date: Thu Apr 1 15:38:30 2021 -0400 Add API retries in finishing firmware update In firmware update orchestration, after the subcloud is unlocked, the runtime manifests on the subcloud may take some time to complete causing the sysinv API calls to query the subcloud to fail. This update is to add retry attempts during the finishing firmware update strategy step. Closes-Bug: 1922281 Change-Id: I678eaf701d9c0cf40907fb8c378a5a2c531a88ae Signed-off-by: Teresa Ho commit d5fbf8913f0990f5d0e3b99edb79a8d8da801b12 Author: Jessica Castelino Date: Fri Mar 12 13:43:49 2021 -0500 Objectify firmware audit data Recently merged commit broke firmware audit and caused firmware_sync_status to be unknown even after the device image was uploaded on system controller. This commit fixes firmware audit by objectify the firmware audit data similar to patch and kubernetes audit data. Change-Id: Ibb53641b9ead78767523612a1cff8c8993c874dd Story: 2007267 Task: 42048 Signed-off-by: Jessica Castelino commit 36497a9bf518c52338e5f46d2f541124d742a8c2 Author: Tee Ngo Date: Wed Mar 10 09:05:19 2021 -0500 Add unit tests for subcloud restore Update API and manager unit tests to include subcloud restore tests. Story: 2008573 Task: 42028 Change-Id: I0e16f9914fb5a4d0b73ea9837a75931e2a4432d6 Signed-off-by: Tee Ngo commit 7625b448de19d0a66ed3c98793e254671404e801 Author: Jessica Castelino Date: Mon Mar 8 16:51:09 2021 -0500 Handle in-progress subcloud operations after dcmanager-manager restarts When a subcloud is added it can go through several phases - install, bootstrap and deploy. If the dcmanager-manager process is restarted while the subcloud is being added, the DB would never be updated and subsequent phases are not completed. This commit fixes this issue by making dcmanager audit for subclouds in transitory states on startup and update the deploy state to the appropriate failure state. Story: 2007267 Task: 42026 Change-Id: I90a57cef89f1cc04a07fa6482ff2c29edf7be965 Signed-off-by: Jessica Castelino commit 1348af4772414959561e40faa5d5a97e3cea11ce Author: Tee Ngo Date: Wed Mar 10 13:28:13 2021 -0500 Update dcmanager API doc for subcloud restore Document the dcmanager subcloud restore API. Story: 2008573 Task: 42034 Change-Id: I38505518b367b275a95f040cee67e6a183cf9606 Signed-off-by: Tee Ngo commit c4a208325c6dd8403c9a93b74c50f1e7801097b0 Author: David Sullivan Date: Wed Mar 10 10:19:51 2021 -0600 Remove existing dc-vault load data during import During a distributed cloud upgrade, an admin may need to delete and import a new load. If the load-delete command is issued without the --os-region-name SystemController parameter the load will be deleted from sysinv but will still be present in the dc-vault. To resolve this we will always remove any existing dc-vault data during a load import. Closes-Bug: 1918451 Signed-off-by: David Sullivan Change-Id: I143e2c084205c105f084d96714e8ebef94dbd130 commit 8e26a6731ae97b05f399dc52709b50eaa59054b8 Author: Chris Friesen Date: Tue Mar 2 13:25:14 2021 -0600 add support for event notifications to dcmanager Add support for generic event notifications to dcmanager. Initially this will be used by sysinv to notify dcmanager that the platform upgrade has been completed so that it can trigger a load audit of all subclouds Story: 2007267 Task: 41968 Change-Id: I9edb00a385ee5f617ac4415aa7e4675f52e28139 Signed-off-by: Chris Friesen commit b6638ce9a1055e5131a494d70897cbf37ce2a8c1 Author: albailey Date: Tue Nov 17 15:47:58 2020 -0600 Distributed Cloud Kubernetes Upgrade Orchestration This commit introduces DC Kubernetes upgrade orchestration. - Added a new kubernetes endpoint for DC audit The audit compares that the active-target-version in the subcloud for kubernetes matches the system controller. - Wiring the kubernetes strategy to the kube orch thread - Add protections to orchestrator shutdown for when 'start' encounters a problem. - Added 'kubernetes' as a valid strategy type for: create / abort / delete / show - Update the endpoint flag file name to trigger updating the new endpoint - Added a kubernetes orch thread similar to sw upgrade orch thread with these states: - updating kube patches - creating vim patch strategy - applying vim patch strategy - deleting vim patch strategy - creating vim kube upgrade strategy - applying vim kube upgrade strategy Patch vim orchestration steps will be skipped if there are no patches that need to be applied in the subcloud. Change-Id: Ic508ed1a504dba177c100a9caf783e87bc10193a Story: 2008137 Task: 41177 Depends-On: https://review.opendev.org/c/starlingx/nfv/+/767421 Signed-off-by: albailey commit 1602fc70b271c8986c0ba414300a9b27ff5f2e9d Author: David Sullivan Date: Fri Feb 26 08:57:11 2021 -0600 Support background runtime manifests during upgrade-activate Update dcmanager orchestrator to accept the new activating-hosts state. Story: 2008055 Task: 41947 Signed-off-by: David Sullivan Change-Id: I032d82ed489b1ae0652429f91adfa5fa43fc3d3f commit af14a6f65cfa13746ef9bed29a59e9640b4d1092 Author: Tee Ngo Date: Tue Feb 23 00:09:57 2021 -0500 DCManager update for Redfish subcloud restore In this commit, dcmanager api and manager are updated to support the restore of a Redfish capable subcloud from backup data. Unit tests will be added in a separate commit. Tests: - Restore request without install (not yet supported) - Restore request for a subcloud that is currently in 'managed' state - Load is not in dc-vault - Mandatory restore value (backup_filename) is not present - Specified backup file cannot be found - Subcloud restore with backup data on the target (i.e. backup tarball is under /opt/platform-backup) - Subcloud restore with backup data on the system controller (i.e. on_box_data = false) - Simulate install failure - Simulate check target failure - Batch subcloud restore - Subcloud restore retry - Subcloud restore with patches Task: 41725 Story: 2008573 Depends-On: https://review.opendev.org/c/starlingx/ansible-playbooks/+/777046 Change-Id: I8134c535e39231837727811475b0f01b2ccddb63 Signed-off-by: Tee Ngo commit 69b815af7d1e20eebc6fef2fe4b514a3355359ad Author: Adriano Oliveira Date: Thu Feb 11 21:02:41 2021 -0500 Add orchestrator unlock retry logic During distributed cloud orchestrator upgrade, a SR-IOV VFs configuration change may take several minutes to apply because the runtime manifest may have delayed for several minutes. While not applied, if an unlock is attempted, it will fail because sysinv prevent it to happen since DB was not updated yet. Code has been added to detect host unlock failure and implement a retry logic that can take up to 20 minutes. Partial-Bug: 1914836 Signed-off-by: Adriano Oliveira Change-Id: I51b1eba8cf7cba1368e159f9dbf0b4d6ee858285 commit c446fb2f2ff58dc3f4c4708ff5aae259d75ff4d1 Author: Chris Friesen Date: Tue Feb 16 17:11:58 2021 -0600 dcmanager update availability on first failed audit In order to improve the response time in the face of failures, change the dcmanager aduit code to update the availability (offline/online) on the first failed audit rather than waiting for a second failure. This does mean that an intermittent failure will result in dcmanager treating the subcloud as "offline". Story: 2007267 Task: 41336 Signed-off-by: Chris Friesen Change-Id: If71d51e82a0147ba38153bf06a6fa8f54d38ae80 commit 0a6ce1a956e6d9474c69a26d34ec733716b821b4 Author: Chris Friesen Date: Thu Feb 25 10:05:56 2021 -0600 Trigger all sub-audits when subcloud goes from offline to online When we detect a subcloud going from offline to online, we can't assume anything about it anymore so let's trigger all the sub-audits so that on the next audit we'll know whether it's in sync or not. (I also re-raised a couple of exceptions to get better error handling. Shouldn't ever hit this but you never know...) Closes-Bug: 1916784 Signed-off-by: Chris Friesen Change-Id: Ie252272ff75077e21a9942f136ebfbd95420c175 commit 3b2fb67cfe1556fb468363c5196056d54c8aad6b Author: Chris Friesen Date: Mon Feb 22 13:18:24 2021 -0600 trigger subcloud audit when subcloud becomes managed When we change a subcloud from unmanaged to managed, trigger all of the various subcloud audits for that subcloud. Previously we triggered them for *all* subclouds, but now we have the ability to do it for just one. Also change it to only trigger the audits if we successfully change the subcloud to "managed". No point in triggering them if we weren't successful or if it was already managed. Story: 2007267 Task: 41889 Signed-off-by: Chris Friesen Change-Id: I4494c80080dacb59d330162035bd6b595d8026d2 commit 0b38bccc5f13ed4bf661d7b47a799c2eb2bbe1e2 Author: Chris Friesen Date: Tue Feb 16 11:04:03 2021 -0600 add API to run all subcloud audits for one subcloud Add a SubcloudAuditManager API to trigger all of the subcloud audits for a specific subcloud. This would be useful when adding a new subcloud. Story: 2007267 Task: 41336 Signed-off-by: Chris Friesen Change-Id: I015c7c5762a0fca1f08ff6e1aa92ca7a738ddb82 commit 843545184a89c1ae4c4907c7b8169a9eb60e29c7 Author: Chris Friesen Date: Fri Feb 12 12:23:48 2021 -0600 Trigger audits on dcmanager-audit startup If we swact controllers or patch dcmanager-audit, the audit processes can potentially be killed while there are audits in progress. Rather than wait for the full timeout, we trigger an "audit fixup" on dcmanager-audit startup, and also trigger all of the sub-audits in case we were doing any of them when the audit processes were killed. Story: 2007267 Task: 41336 Signed-off-by: Chris Friesen Change-Id: Ic069b6d10cae77613571faaba10869367182c118 commit fe05f2ea9bb6eba394229ad019b23f54eac31b69 Author: Chris Friesen Date: Tue Jan 26 17:38:17 2021 -0600 fix up stale subcloud audits If something happens and an audit is interrupted after it was started but before it completed, the timestamps will be messed up in the DB and a new audit won't be triggered because the audit manager will think that there's an audit already in progress. Add a new check that looks for subclouds where the audit has been "in progress" for long enough that we can be pretty sure something has gone wrong. Once we find them, reset the audit finish time so that a new audit will be triggered at the next audit start time. Story: 2007267 Task: 41336 Signed-off-by: Chris Friesen Change-Id: I390a99ba2bb06915b60fcfb8865096e1ede36cd3 commit c71703128e99bdd47532524716684908d2a734ff Author: Chris Friesen Date: Mon Nov 30 17:46:11 2020 -0600 Move subcloud audit to new worker processes In order to allow for faster subcloud audits, introduce a new "audit-worker" process. By default there will be four worker processes to perform the actual subcloud audits. The main audit process will scan the DB for subclouds which need auditing based on the new audit start/end timestamps. It will then send out RPC messages to the "audit-worker" processes to request an audit for a list of subcloud IDs. When the "audit-worker" process receives the "audit_subclouds" RPC message, it loops over each of the specified subclouds. For each subcloud it updates the DB to indicate that the audit is starting for that subcloud, then basically does the exact same audit that is currently done in the audit process, then updates the DB to indicate that the audit has been completed for that subcloud. Story: 2007267 Task: 41336 Signed-off-by: Chris Friesen Change-Id: Ifb3dd363fd337d24f2c3f7aaa3549624fffaceca commit 8417ce7737a9798c0938a6e2d378db2961cb4636 Author: Chris Friesen Date: Mon Nov 30 17:38:07 2020 -0600 Add subcloud audit DB table In order to allow for subcloud audits to be handled by multiple processes, add a new DB table to track subcloud audit information including start and finish times for the various audits as well as whether a special audit has been requested. On DB upgrade we populate the new table with entries for each non-deleted subcloud. The subcloud creation code is modified to also create a corresponding subcloud_audit entry. Story: 2007267 Task: 41336 Signed-off-by: Chris Friesen Change-Id: Id2f2b49eb5b8b1f20b1c128bce4811479bcb96b9 commit a6cb0594193ce9cec4934a5b1b588b58d7f36082 Author: Jessica Castelino Date: Mon Sep 28 17:57:47 2020 -0400 Upgrade Orchestration for Standard subclouds This commit enhances upgrade orchestration to support Standard subclouds. Strategy Steps: - Pre-checks to verify subcloud is ready for an upgrade and to recover from previous failures - Install release N+1 license in subcloud - Download/import the release N+1 load in subcloud - Update and apply N+1 patches to release N+1 load in subcloud - Finishing patch strategy - Start duplex upgrade - Lock controller-1 - Upgrade and re-install controller-1 - Unlock controller-1 - Swact to controller-1 - Create VIM upgrade strategy - Apply VIM upgrade strategy - Swact to controller-0 - Activating Upgrade - Completing Upgrade - Delete release N load Change-Id: I6fb60912ec1e90baa593cf98e502f4c91f599f42 Story: 2008055 Task: 41105 Signed-off-by: Jessica Castelino commit aab68918c9f307330692fa2cbff26a558566f066 Author: Chris Friesen Date: Wed Dec 16 14:36:53 2020 -0600 cleanup audit interval constants This converts the hardcoded patch interval to make use of an already-defined but unused config option, and tweaks the default audit interval. Story: 2007267 Task: 41336 Signed-off-by: Chris Friesen Change-Id: Ie6f023d0e46accdb79bd854e9833158e0a33e4b3 commit 527a48513a2534ed73755f758e01992414cabe21 Author: Bart Wensley Date: Fri Jan 22 14:32:18 2021 -0600 Turn off the legacy pip resolver for distcloud The requirements that had conflicts have been updated. The local upper-constraints is no longer being used by tox since two constraints files with different values for the same requirements are not supported. To enable this change, some unnecessary code that required an older version of oslo.messaging is removed. Bugbear is also being disabled due to brokeness that can cause test failures. Change-Id: If007b7e0ad3c04b78b619542162267cc7d502460 Story: 2004515 Task: 41670 Signed-off-by: Bart Wensley commit 13092996ca4d3224354bd8dd39a9c50e256d5646 Author: John Kung Date: Thu Jan 14 09:20:02 2021 -0600 dcorch-engine endpoint type handling improvements Handle the endpoint type remove and add operations. This is invoked for optional applications such as stx-openstack. Cleanup unused SubCloudEngine class. Tests Performed with multiple subclouds: Bootstrap, Installation and Deployment Initial manage and sync Resource update and sync via proxy and audit Add and remove endpoint type Unmanage and manage subclouds Change-Id: I5c7033e7bed91a7bcc9b2c989daf333b76013b8c Story: 2007267 Task: 41589 Signed-off-by: John Kung commit 43138e6aa59bf057ae4c7d2cda623ed042fc2fc1 Author: Yuxing Jiang Date: Tue Jan 5 10:27:34 2021 -0500 Specified RVMC image tag for subcloud install In local docker registry, it is possible that there are multiple versions/tags of Redfish Virtual Media Controller(RVMC) image, caused by upgrade/downgrade platform for example. Currently, the subcloud RVMC install picks only the first returned image tag which may not compatible. This commit specifies the required RVMC image tag. And this image will be downloaded/pushed to the local docker registry as a static image after a fresh install or upgrade by the ansible playbook. Closes-Bug: 1908100 Change-Id: Ic2a593ce81952e4b43f41ee9908b59ba51fe92ab Signed-off-by: Yuxing Jiang commit 547b164fa3e72267c47c9ee344a75306d2f88460 Author: Tao Liu Date: Mon Jan 4 10:25:53 2021 -0500 Orchestration Support for a Single Subcloud Group This update adds an option to create an orchestration strategy for a single subcloud group. It encompasses all types of DC orchestration - patch, upgrade, firmware. Test: Create patch-strategy with or without --group option Create upgrade-strategy with or without --group option Create fw-update-strategy with or without --group option Story: 2007267 Task: 41524 Signed-off-by: Tao Liu Change-Id: Icda88d1bbfa537f3e9e399d7fabb48e2b2b6a7fa commit ec8dba8a53b1beb0562bdd4228860f42e8d055e2 Author: Takamasa Takenaka Date: Wed Dec 23 09:10:33 2020 -0300 Remove entries related to host-based snmp According to the host-based SNMP removal, remove community and trapdest related processes Story: 2008132 Task: 41394 Change-Id: Icad7f82d5e20149a49cbb1540486b882856ac189 Depends-On: https://review.opendev.org/765381 Signed-off-by: Takamasa Takenaka commit 181bd0c86d77dd783d0d1c82acdc0063732ed073 Author: John Kung Date: Tue Dec 8 10:43:50 2020 -0500 Enable multiple dcorch-engine worker processes Multiple worker processes are created for distributing the dcorch sync and audit work of the dcorch-engine on the active platform controller host. The database sync lock is introduced as a synchronization mechanism for dcorch-engine workers and stores subcloud, endpoint and action locks. The subcloud sync status table is introduced to track subcloud/endpoint sync request, audit status, and report status. In-memory caching of subcloud objects and tracking info such as last audit and reported sync are removed and the dcorch DB is referenced instead. The GenericSyncManager and sync_threads are restructured and use thread pools for work to be performed on subclouds. The infrastructure allows for sync_threads to be allocated to a dcorch-engine worker. In this commit, each worker process can handle each subcloud. Robustness audits ensures no stale sync_lock and dcorch-engine worker process file limits are increased. Tests Performed with multiple subclouds: Bootstrap, Installation and Deployment Initial manage and sync Resource update and sync via proxy and audit Unmanage and manage subclouds Verify sync_lock on controller swact and process termination Dead Office Recovery Co-Authored-By: Tao Liu Co-Authored-By: Teresa Ho Change-Id: I179279dea5babaf57084b28adea86a03b330924a Story: 2007267 Task: 41374 Task: 38662 Signed-off-by: John Kung commit 37959399c9d1e5a8915bf0df80ba51136a5e91f4 Author: albailey Date: Wed Dec 9 12:13:20 2020 -0600 Fix bugs with assigning a group for a subcloud The subcloud CLI and API support passing a group by name or ID, however the code was only working if an ID was passed. Furthermore, it could bypass the validator and allow setting zero as the group id, even though there is no group with an ID of zero. This submission includes unit tests for updating the group by ID, name, and with a number of rejection tests. Change-Id: I53d20a090cfe031c55802104012cea1cb4ff6cb6 Closes-Bug: 1907488 Signed-off-by: albailey commit 9f0c5426cc54775ec4f9d2f0ff89b709ae95b5fc Author: albailey Date: Tue Dec 8 16:07:51 2020 -0600 Fix tox failures from new resolver in pip New pip was released on Dec 3, 2020 which breaks downloads from pypi for some URLs. New virtualenv was released Dec 7 which includes the new pip. The new pip uses a different resolver which fails with many of the components installed by pip. These components may need to be fixed by later commits, since the legacy resolver is being removed completely in early 2021. Change-Id: Ifcbd8965e2ef0b53dae0c878c2a0f2bd91b572c2 Closes-Bug: 1907125 Signed-off-by: albailey commit 164e8be8dd972031547127b4c1a820dc27a4886c Author: Bart Wensley Date: Thu Dec 3 10:12:49 2020 -0600 Fix pylint errors Fixing remaining pylint errors and removing from pylint disable list so they cannot be re-introduced in the future: E1102: not-callable E1120: no-value-for-parameter E1128: assignment-from-none Change-Id: I0f59fdcd0fd20e9d0a8042ba3f7f2a5a20a701db Story: 2007082 Task: 41335 Signed-off-by: Bart Wensley commit a5ad984ba2990a4cf42479c45bd4a246e4ac778e Author: Yuxing Jiang Date: Mon Nov 30 11:27:41 2020 -0500 Add a specific counter to handle activation failure We intend to retry activation 10 times if it fails(in a total of 45 minutes) in the DC orchestration. As the audit and retry are sharing the same counter, if the activation fails after 10 minutes, the orchestrator will not re-attempt the activation. To handle the activation failure, this commit adds a specific counter to activate retry in case the activate fails after 10 minutes. After this commit, the orchestrator will wait up to 45 minutes if the activation is in progress, and the retries will be limited to either the audit reaches DEFAULT_MAX_QUERIES or itself reaches MAX_FAILED_RETRIES. Tested by activating an AIOSX subcloud by orchestration. Closes-Bug: 1904934 Signed-off-by: Yuxing Jiang Change-Id: Ic37c14ee316e02f690b55b082862a85fc54501bc commit 326b8e33ddfc5854704a49b930f05229faf2124b Author: Bart Wensley Date: Mon Nov 30 13:30:50 2020 -0600 Fix pylint E1101 (no-member) errors Fixing pylint E1101 errors and removing from pylint disable list so they cannot be re-introduced in the future. Some of the fixes included deleting or commenting out code that was no longer in use and was causing these errors. Change-Id: I5c5a7f9e90cc5cb9cf4510f578bb6e31cea1f5ef Story: 2007082 Task: 41335 Signed-off-by: Bart Wensley commit f7728125e1284edb064d471a92388b7dea074b80 Author: Tee Ngo Date: Thu Nov 19 17:08:41 2020 -0500 Fix a logic flaw in orchestration pre-check If the subcloud is online but its deploy status is install-failed and the subcloud host is locked, the upgrading simplex step must have failed early in the previous upgrade attempt. The pre-check should transition directly to upgrading simplex step in the retry. Closes-Bug: 1904938 Change-Id: I73815939526310d04d7b50dc8b85d33b5d502d25 Signed-off-by: Tee Ngo commit b466efdb7e7e9ae2e45350a8d4cbe322383cb47c Author: Tee Ngo Date: Thu Sep 24 17:51:02 2020 -0400 Disable ansible logging when invoked via subprocess In this commit: - Overwrite the default ansible log path in dcmanager. The output of ansible playbook executed by dcmanager is already directed to /var/log/dcmanager directory. The default ansible logging to the playbook user's home directory (/root in this case) is unnecessary and can exhaust rootfs over time. - Write playbook output to /var/log/dcmanager/ansible directory. - Set separate logrotate rules for ansible output. Tests: - deploy subclouds and verify that a) no ansible output is written to /root b) ansible output (remote install, bootstrap, deploy) is written to /var/log/dcmanager/ansible, one single log per subcloud c) each instance of ansible output in the log file follows the following format: playbook command with timestamp followed by the output of the executed playbook d) subcloud ansible output log is rotated with delayed compression to ensure logging continuity for the on-going ansible play e) rotation rules (size, count) are respected - orchestrate a subcloud upgrade and repeat the above verifications Closes-Bug: 1895607 Change-Id: Idbf87e4472df26c844526328aeacd9e55325bf0c Signed-off-by: Tee Ngo commit e88f3ef6f504c8b72fb4a3f9650a7f71e8ae6fb9 Author: Lu Yao Chen Date: Mon Nov 9 15:03:23 2020 -0500 Flag secret to mask dcmanager logs Using oslo_config flag and setting secret=True on sensitive info tested on running controller-0 system, api, audit and engine logs are masked by redacting the info Partial-Bug: 1896116 Change-Id: I201ef344902ace391f6fa028ce81fd3fe96b92f3 Signed-off-by: Lu Yao Chen commit 83155e5e9f94bc1ab2e137f9b025b64119283e6a Author: Bart Wensley Date: Tue Nov 3 20:23:39 2020 -0600 Update unit tests to use database instead of mocking Updating the distributedcloud unit tests to use the database during unit testing instead mocking all database access. This improves code coverage and simplifies the unit tests. Also fixing a minor bug in SubcloudManager.add_subcloud that resulted in an incorrect deploy_status being returned to the caller of the method (uncovered by unit test changes). Change-Id: I68eb9b98d440ae0c5f6661b17a590e6d7b1ff414 Story: 2007082 Task: 41210 Signed-off-by: Bart Wensley commit 561b24b285bbe5d3e651b34bc4136bc206f42af3 Author: Bart Wensley Date: Thu Oct 29 11:33:31 2020 -0500 Remove distributedcloud dependency on sysinv The distributedcloud package uses a single constant from the sysinv package. Removing this dependency - use of sysinv should only be through the REST API or the client - no sysinv code should be imported. Also changing tox config to move the tox work directory out of the source tree. Change-Id: I9973f16ae66a7339d2df958b5f2afe047ee980d2 Story: 2007267 Task: 41176 Signed-off-by: Bart Wensley commit 12b81ff452ce79d6dc1547c0a315284309e95b7f Author: albailey Date: Tue Sep 1 08:02:57 2020 -0500 Delete subcloud VIM strategies when DC fpga strategy deleted As part of an fpga upgrade strategy in a subcloud by the VIM, that strategy needs to be deleted. Previously it was being deleted as part of a the final steps of a successful orchestration, but there can be scenarios where that orchestration is incomplete, and therefore these VIM strategies need to be handled. A new DC orchestration cannot be created until the old one is deleted, so this change ensures fpga vim strategies are deleted if possible when the DC orchestration strategy is deleted. A bug in applying vim strategy has been fixed, and the section removed since a vim auth exception cannot be raised since a new vim client is constructed per loop. Unit tests to invoke the delete strategy have been added. Tox coverage now generates an html report. Change-Id: I4f3d7bc3550e12c430d1b1c0ce3dffb298fe31b4 Closes-Bug: 1891521 Signed-off-by: albailey commit 797574962804762a562a088b08b114f929ed6512 Author: Don Penney Date: Tue Oct 13 09:45:52 2020 -0400 Setup distcloud logfile configuration This update creates and packages distcloud-specific logfile configuration for syslog and logrotate, moving it out of the central syslog-ng-config package. Change-Id: Ia6d38fe7d39ff68178441db8844c00232f7597f6 Story: 2008251 Task: 41099 Depends-On: https://review.opendev.org/757947 Signed-off-by: Don Penney commit c659a7f684ca06e8104a66e95047d61966c865c2 Author: Yuxing Jiang Date: Mon Aug 17 17:42:08 2020 -0400 Create an API to reinstall a subcloud This commit creates a REST API to reinstall a subcloud. In SubcloudController, the tasks includes: 1. Read the subcloud info from dcmanager db, subclouds table. 2. JSONify the data_install from db, generate the new install values. In SubcloudManager: 1. Check the subcloud availability, software version before reinstall 2. Check the image value, if image doesn't exists, update the image with iso in dc-vault 3. Run the install command and apply command API format: PATCH /v1.0/subclouds//reinstall ----- Tests: ----- Happy path: 1. dcmanager subcloud add: a new subcloud with bootstrap values, deploy config and install values. 2. dcmanager subcloud delete: an existing subcloud. 3. dcmanager subcloud reconfig: an existing subcloud with deploy config. 4. dcmanager subcloud reinstall: an existing offline subcloud, this subcloud has image path in data_install in db dcmanager. After the tasks of reinstall and bootstrap succeed, reconfig this subcloud with proper deploy config, this subcloud will be online. 5. upload an image to dc-vault using: system --os-region-name SystemController load-import -a . Then using dcmanager subcloud reinstall to an existing offline subcloud, this subcloud has no image path in data_install in db dcmanager. After the tasks of reinstall and bootstrap succeed, reconfig this subcloud with proper deploy config, this subcloud will be online. Unhappy path: 1. dcmanager subcloud reinstall: an existing online subcloud, reinstall fails. 2. dcmanager subcloud reinstall: an existing subcloud without data_install value in db, reinstall fails. 3. dcmanager subcloud reinstall: an existing subcloud with data_install in db, but missing mandatory install value, reinstall fails. 4. dcmanager subcloud reinstall: an existing subcloud, but its sw version in db doesn't meet the sw version with system controllers', reinstall fails. 5. dcmanager subcloud reinstall: an existing subcloud with data_install in db, but has no image path in data_install, and also has no right versioned image in dc-vault in system controller, reinstall fails. Story: 2007267 Task: 40732 Change-Id: I8be6d8d11e6b4ee02bbcca499ba8869ba76bffaa Signed-off-by: Yuxing Jiang commit 19790859f060c35b28e18b482a2406729c9a5915 Author: Tee Ngo Date: Thu Sep 24 09:48:09 2020 -0400 Use cached keystone client in add subcloud workflow Leverage keystone client caching work done for dcorch to reduce load to keystone during batch subcloud deployment. Tests: - Deploy 1000 subclouds in 50-subcloud increments in a simulated DC environment. Story: 2007267 Task: 40955 Change-Id: I716b202005b943a1f389d577ce035371d7ab938a Signed-off-by: Tee Ngo commit eb97f4c8b6b3c1cce6c162c76b5309751cb7a924 Author: Jessica Castelino Date: Mon Aug 24 13:19:14 2020 -0400 Move dcmanager orchestration to a separate process 1) Remove DC manager orchestration from dcmanager-manager process 2) Create dcmanager-orchestrator process and associated files 3) Add new RPC calls for dcmanager-orchestrator process to notify dcmanager 4) Create/update unit tests, to verify the implementation changes Story: 2007267 Task: 40734 Change-Id: Ibbbae77558a8a8fd95b636fa6c3aebb1dfefb514 Signed-off-by: Jessica Castelino commit 64caf6de7c0ba23fa0f5c92facc19400a11a3c4e Author: David Sullivan Date: Mon Aug 24 14:12:18 2020 -0400 Import load metadata for SX upgrades For SX upgrades use the load-import-metadata API to import the load. Do not sync patches for loads that were imported with metadata only. Change-Id: If106711c976c1514419b120ba933662014a9bd7a Story: 2007403 Task: 40807 Depends-on: https://review.opendev.org/#/c/749413/ Signed-off-by: David Sullivan commit 4c29aa748cc1d4b1540cbbfe8096d35855119a65 Author: David Sullivan Date: Wed Aug 26 15:30:30 2020 -0400 Skip patching during SX data migration For SX upgrades we will skip applying the patches present in the upgrade_data. The N+1 patches are not applied after load-import so would not be present in the upgrade_data. With https://review.opendev.org/#/c/745355/ all N+1 patches applied to the system controller will be present after the subcloud install. Change-Id: I8f174211d8a4ddf4d84502d9c81736aaa22bc0c8 Story: 2007403 Task: 40759 Depends-on: https://review.opendev.org/#/c/748288/ Signed-off-by: David Sullivan commit 8a14160c5396a190a4dc75db7f8e65f81ddbd3c8 Author: John Kung Date: Tue Aug 25 09:05:58 2020 -0400 Update upgrades and patching get client to ensure token valid The client access for upgrades and patching are updated to get the client with cached keystone token on each invocation. This ensures that the cached keystone token expiry is checked each time, and if necessary, refreshed. Tests Performed: Distributed Cloud patch orchestration Distributed Cloud upgrades orchestration Verify by shortening and forcing token expiry on long running tasks Change-Id: I4f69c16e3a61bdbdfd717f135681b82e9d860425 Story: 2007267 Task: 40741 Signed-off-by: John Kung commit fa09faf22994b37ee90c30a1e68a9567c835e074 Author: albailey Date: Mon Aug 24 15:13:30 2020 -0500 Update DC subcloud API unit tests for proper validation The DC subcloud API tests were mocking away many of the validators, and were testing generic failure for most invalid inputs. Now the unit tests for subclouds API endpoint will check success and failure scenarios more accurately. Change-Id: Ic0fd4c1d9f3552601f95bf5031f94982985019c3 Story: 2007267 Task: 40800 Signed-off-by: albailey commit b442f2afcdadb4e10893b66c085a9690005f5946 Author: John Kung Date: Wed Aug 19 12:06:28 2020 -0400 Enable subcloud install from image in dc vault The subcloud install is updated to allow reference to the vault load iso. When the image is not specified in the install values, or the image specified matches the vault load location, the image input iso is updated to reference the System Controller load vault. In such case, the bootimage iso is no longer duplicated, thus reduces the storage requirements. The -a' or '--active' flag, enables the upload of an active image:` system --os-region-name SystemController \ load-import -a Tests Performed: Import active image into dc vault Import active image into dc vault with new image Subcloud Install from image in dc vault Change-Id: I9faab0a5c9c7be7861ce2b4bc785cd44b9d4771a Story: 2007267 Task: 40695 Depends-On: https://review.opendev.org/#/c/746972 Signed-off-by: John Kung commit 7f25a7141d08718c111aad54141b26d34555be19 Author: Bin Qian Date: Fri Jul 24 21:22:23 2020 -0400 Trigger cert-mon audit from subcloud state changes When subcloud is managed, or becomes online, inform cert-mon via RPC call, so that cert-mon to perform endpoint certificate audit. Story: 2007347 Task: 40442 Depends-on: https://review.opendev.org/#/c/742468 Change-Id: I733d2e22c2a4f7089b903b428025d87cee5123c4 Signed-off-by: Bin Qian commit ed2ffb4f6b280cdaf0cec1d5625ed5c88d5ebdfb Author: Tee Ngo Date: Wed Aug 12 08:43:01 2020 -0400 Delete N load as part of DC orchestration In this commit, a new step is added to remove the N load after the subcloud has been upgraded to N+1 release. The importing load step is also updated to remove an old load in the subcloud should one exist prior to importing the N+1 load. Story: 2007403 Task: 40656 Change-Id: If015b94354d47931c2ac9090b6c10c29ee051c3c Signed-off-by: Tee Ngo commit fa3e10c7e4667cfd70c9dbc069e1a55b7a15f8fb Author: Tee Ngo Date: Thu Aug 13 18:19:20 2020 -0400 Add subcloud system health check to pre_check Subcloud upgrade can only start if system health check passes, i.e. subcloud node(s) are all unlocked, enabled, patch and config current; all critical components are healthy and there are no management affecting alarms. The commit also includes the fix for a minor bug related to no data install value. Story: 2007403 Task: 40665 Change-Id: I6464a20ea82f5ad0d6ac5e2bdc93bbcb28e08b6c Signed-off-by: Tee Ngo commit a93baec6fe472de8da5fa01aae42153d4087c6b5 Author: Don Penney Date: Wed Aug 19 21:07:24 2020 -0400 Add support for rd.net.timeout.ipv6dad to subcloud install The rd.net.timeout.ipv6dad boot option instructs dracut in the initrd to use the specified timeout for its IPv6 Duplicate Address Detection, rather than its default of 50 seconds. This allows the user to configure an increased timeout for a subcloud that may take longer for interface networking setup to be complete. Story: 2007267 Task: 40712 Change-Id: Ic7fe8eca975625f8634fff310de3d3550d7f5450 Signed-off-by: Don Penney commit 3360a8d3cb142ceb35a3bb17f691baa8a60929d2 Author: Teresa Ho Date: Thu Aug 13 13:59:17 2020 -0400 Fix system commands in dcorch-sysinv-api-proxy The request body for the system commands was not accessible when processing the response. The fix is to make a copy of the request and its body prior to processing the request. Closes-Bug: 1891546 Change-Id: I2a6202311d0f9f6fbb91ae9581e8818c69c78587 Signed-off-by: Teresa Ho commit cc43aa6205a4adfffb289556304602be64db52af Author: Jessica Castelino Date: Thu Aug 13 13:15:40 2020 -0400 Exception handling when subcloud not found When subcloud alarms are fetched for a subcloud name that doesn't exist, an exception is raised, but this exception is not generated correctly. This makes debugging the issue more difficult. A fix has been made to generate the error properly. Change-Id: I7f2efb94d3063980e1d7edc8aa6b9a19eea2f5e1 Closes-Bug: 1884575 Signed-off-by: Jessica Castelino commit 82fa2e9b2fa70712146de4ba563c91b74d31b2f4 Author: albailey Date: Tue Aug 11 11:44:02 2020 -0500 Set subcloud firmware in-sync immediately after subcloud completes When a subcloud completed, it might show out-of-sync until the remaining subclouds were also orchestrated since that is when the audit is explicitly called. Now we update the status as each subcloud completes. Change-Id: I0694c7aacfb781aae1a7d22dd777fa75dff2bdfc Closes-Bug: 1890952 Signed-off-by: albailey commit 589ab369f398c8a8283a64d19b742fefa19a9f22 Author: Jessica Castelino Date: Fri Jul 31 11:20:20 2020 -0400 Fixes to support DC FPGA Orchestration When dcmanager-audit is restarted, it ensures that all endpoints exist for all subclouds. If a subcloud without the endpoint is encountered during startup, it adds an entry for it without requiring the subcloud to be re-added. A flag is also added on a replicated filesystem to avoid these checks from being executed more than once on the system controller. Change-Id: I049404c6a3e881623f3ff905a859a94f03bcf8e1 Story: 2007875 Task: 40471 Signed-off-by: Jessica Castelino commit 663fcf0f8868c57540e682d47b93aab3d7a0c12c Author: albailey Date: Fri Jul 31 16:12:46 2020 -0500 Fixes for DC firmware orchestration related to labels and upload The applied image labels on the system controller should be the labels used when applying the image on the subcloud. The stage that removes images now uses all the applied labels for the image when removing them. The code that converts image fields for upload was passing an integer for the revoke_key_id. It now converts to a string to prevent an encode exception. This commit also enables some unit tests for DC firmware orchestration that were being skipped. Closes-Bug: 1890355 Closes-Bug: 1890340 Closes-Bug: 1890324 Change-Id: I1a730b01c0bbaeb5f3e1f6ecfb19f649911da453 Signed-off-by: albailey commit 5aab8597d40856f115e238f18a283d2964588e39 Author: Jessica Castelino Date: Thu Aug 6 16:06:01 2020 -0400 Trigger firmware audit immediately once subcloud is managed When a subcloud is managed, patch audit and firmware audit are triggered. However, the patch audit on completion resets the force_firmware_audit variable, which prevents the firmware audit from immediately being triggered and keeps the subcloud out-of-sync until the next firmware audit cycle (Once in 1 hour). Thus, a fix is added to address this issue. Change-Id: Ie1aa2c7b4cb706d32b17f33bffa17d1004721887 Closes-Bug: 1890294 Signed-off-by: Jessica Castelino commit 749f7dc3d96f2842cd80b59d8022ae433e376494 Author: Yuxing Jiang Date: Fri Jul 24 10:52:23 2020 -0400 Add REST API timeout to patching and sysinv clients The REST API calls from the distributed cloud should timeout to prevent process to hang forever. Change-Id: Iea7f05018c348a817758f6bc920489127242ed11 Closes-Bug: 1886037 Signed-off-by: Yuxing Jiang commit 537d2a9749337fdec501d834f1a84021e5bbd56a Author: Yuxing Jiang Date: Fri Jul 31 09:23:38 2020 -0400 Extend default periodic interval for resource sync audit This commit extends the periodic time interval for resource sync audit to prevent the elapsed time of the dcorch audit exceeding the previous 10 minute period along with the increment of a large number of subclouds. Change-Id: I96c93aef28c6b6359a03f652c6bb4618220a28db Closes-Bug: 1889506 Signed-off-by: Yuxing Jiang commit 8dcf8967da9f856e49b5038e5372d6b55d4326c6 Author: albailey Date: Thu Jul 30 17:57:14 2020 -0500 Fix issues in DC firmware orchestration Labels were not being properly passed in to the apply image API. If removing labels, a syntax error could cause the importing firmware state to fail. Change-Id: Ia5c378f373668c9b9e82b0b422af15deddd42849 Story: 2007875 Task: 40468 Signed-off-by: albailey commit 2b953fd0c7d8d4973d179df34d5497492a56c9f7 Author: Tee Ngo Date: Thu Jul 30 19:07:31 2020 -0400 Add --force option when managing a subcloud When managing a subcloud, allow a --force option to be specified. The force option will allow an offline subcloud to be managed. This is necessary in cases where the user has (mistakenly) unmanaged a subcloud that is offline and the subcloud cannot be recovered while it is unmanaged. One example is a failure during DC upgrade orchestration that leaves the subcloud offline. To rerun upgrade orchestration and recover the subcloud, the subcloud must be managed. Story: 2007403 Task: 40466 Change-Id: I864de63b40b1065ffb1c6a025b5b221b793fbaf7 Signed-off-by: Tee Ngo commit 74027ac73986dc9eb8806ba3ebd836c6ba5a5194 Author: albailey Date: Wed Jul 8 15:44:13 2020 -0500 Add fw-update strategy type support for DC orch Firmware update states are: - 'importing firmware' - 'creating vim strategy' - 'applying vim strategy' - 'finishing fw update' Change-Id: I6f39f45a8b0f9d6b1ed8d33abb49715bb11a8496 Story: 2007875 Task: 40421 Co-Authored-By: Jessica Castelino Signed-off-by: albailey commit 54ef00192d59b43c104d94229b4d66ea0a11b27e Author: Gerry Kopec Date: Thu Jul 23 08:06:07 2020 -0400 Add master reauthentication for identity data sync This fixes Issue #4 from Bug 1870999 where some subclouds failed to sync an identity resource change. This was due to unauthorized request error on the master cloud due to expired token. The current error handling only covers reauthentication of subcloud dbs client. In addition, the number of sync tries is increased from 2 to 3 to accommodate the case were data sync fails authentication on master and subcloud. Also fix exception in dcdbsync on subcloud due to uninitialized variable when handling change to project resource from dcorch audit on system controller. Change-Id: Ife8d86481a64ed841f5231a6df055028ee6597b0 Partial-Bug: 1870999 Signed-off-by: Gerry Kopec commit 598178e36ed6039d2064a81cbd3adde3a51edddd Author: Teresa Ho Date: Thu Jul 16 14:10:25 2020 -0400 Support device image commands in dcorch-api-proxy The dcorch-api-proxy is updated to support the device image upload, apply, remove and delete operations. Story: 2007875 Task: 40371 Depends-On: https://review.opendev.org/#/c/742457 Change-Id: Id5506226297e723f19b7d03ca18cc47ddfa8af40 Signed-off-by: Teresa Ho commit 6fd2b44e750cec2949c7adff737b9c89dae7270f Author: Jessica Castelino Date: Tue Jul 21 17:21:21 2020 -0400 Firmware audit As part of the dcmanager, there will be a periodic audit which scans a number of subclouds in parallel and checks whether the subcloud has all of the “applied” device images that it should have (based on the labels the images were applied against and the host labels in the subcloud), and whether all of the “applied” device images have been installed on the hosts that they should be. If either of these is not true, then the subcloud “firmware image sync status” is considered “out of sync”. This will result in the subcloud as a whole being considered “out of sync”. When a device image is “applied” in sysinv, dcmanager will be notified and will set the “firmware image sync status” to “unknown” for all subclouds since it does not know at this point which subcloud(s) the image needs to be created or applied or updated in. On the next audit, this sync status will be updated to “in sync” or “out of sync” as applicable. Change-Id: Ifbbbeaba0bcd2691332e23d288856b8417c9efa8 Story: 2007875 Task: 40420 Signed-off-by: Jessica Castelino commit 9a4ad4921a56556eb47968257f80da49cb2e586b Author: Andy Ning Date: Fri Jul 24 14:19:34 2020 -0400 Fix subcloud slow initial sync caused by keystone reauthentication Under certain conditions subcloud keystone client reauthentication will fail during initial sync. It will keep on failing in reattempts, causing endpoints to go in-sync slow after the subcloud is managed. This is caused by the keystone reauthentication reusing the existing token that is already invalid. Once that existing token expires (an hour at maximum), the reauthentication will succeed and endpoints go in-sync. This update fixed this by recreating subcloud admin session (and keystone, dcdbsync clients) by using username/password. With this the reauthentication will cover cases where subcloud admin session user changes (eg, from admin to dcmanager or vise versa). Change-Id: I8a49bf8c55e3538fc47b833ae648b667b5e9e9e5 Closes-Bug: 1887849 Signed-off-by: Andy Ning commit 0620b421fd7999ab37747114c0a0f5ec8a098dd5 Author: John Kung Date: Tue Jul 28 09:28:22 2020 -0400 Update dcmanager upgrade orchestration client access During upgrade orchestration with long running operations, it is possible that the keystone token may expire and result in unauthorized http errors. In dcmanager upgrades orchestration, the client usage is updated to retrieve the (cached) keystone client to ensure the checks for the token expiry are up to date. The unauthorized http exception handling is made explicit for the long running client operations. Change-Id: I281ecc51b1a4349ac865f0c4f6ea8244b7cb1fcc Story: 2007403 Task: 40452 Signed-off-by: John Kung commit 49f7ad12fa61430842d7da8663ec9dc8c7378f57 Author: Bin Qian Date: Wed Jul 22 12:25:10 2020 -0400 Update DC ICA expiry duration to 1 year The current 10 years expiry duration is incorrect. Change it to 1 year. Change-Id: Ia9cb8c104c1bb2b1028b29ea1cb39c8aea9d235a Closes-Bug: 1888521 Signed-off-by: Bin Qian commit 73998010b58073a15bc99bf58928376433e10937 Author: Tee Ngo Date: Mon Jul 13 21:49:57 2020 -0400 Add recovery code for failures during each upgrade state In this commit: - Pre check for online case is updated. If the subcloud deploy status is 'migrated', advance directly to activating upgrade step. Otherwise, start from the first step (installing license). - The subcloud deploy status is set to 'migrated' at the end of migrating data step. - The subcloud deploy status is set to 'complete' at the end of completing upgrade step. - Skip data migration if deploy status is either 'migrated' or 'complete'. - Reduce the number of retries for failed activation from 45 to 10. The overall wait time is still 45 minutes. - Improve error messages to user. - Add another log to give clarity to the time it takes the subcloud to respond to load import request. - Various small fixes. Tests: - Load import failure - not enough disk space - Load import failure - bad load file - Upgrade start failure - sysadmin directory too large - Controller lock failure - processing failure - Upgrade simplex failure - incorrect bmc password - Upgrade simplex failure - network glitch - Data migration failure - network glitch - Upgrade activation failure - processing failure - Upgrade completing failure - processing failure Story: 2007403 Task: 40351 Change-Id: Idf0bf8009122018e4597a7f1dede58658d0a064b Signed-off-by: Tee Ngo commit f0dd6b9101ec28ef76faa0bf4b422345cf1d7bbc Author: John Kung Date: Mon Jul 20 17:29:35 2020 -0400 Add prechecks for data_install and scratch filesystem As part of precheck state checks, check data_install values and subcloud host scratch filesystem size. The data_install values are semantically checked on subcloud update, so the precheck verifies its presence. Tests Performed: Verified data_install empty check Verified subcloud scratch host filesystem check when online Verified pre_check stage continues to next stage on success tox unit tests updated and passed Change-Id: I972309f08d648cad6719b9a340a876258f6f912a Story: 2007403 Task: 40396 Signed-off-by: John Kung commit 43492c97db804e6a88053b1abd63d4ddf35749e4 Author: Tee Ngo Date: Fri Jul 17 16:55:51 2020 -0400 Add --force option to strategy create API The new option will allow the user to retry the upgrade of the specified subcloud that previously failed during remote install or data migration step. Story: 2007403 Task: 40387 Change-Id: Ib17857340c85892ff962bfcf3f9793c4e65bbef2 Signed-off-by: Tee Ngo commit dc6d66c969c9236890013f2a029c8e65d422bac5 Author: Bin Qian Date: Wed Jul 8 17:06:16 2020 -0400 Add RestAPI to update subcloud sync-status Currently set restriction to only allow updating endpoint dc-cert Updating other endpoint status can be enabled when there is use case. Passed TCs: Test with testing client to 1. successfully update dc-cert to in-sync, out-of-sync and unknown 2. updating endpoint other than dc-cert is rejected 3. updating dc-cert status to word other than above mentioned is rejected. Change-Id: I77df581c9d1d461f72948499e9410f22819b1030 Story: 2007347 Task: 40300 Signed-off-by: Bin Qian commit 6a16d05061e23c87c53b2ca6e89a2b383db8f850 Author: Bart Wensley Date: Wed Jul 15 23:53:10 2020 -0500 Fix dc-cert sync status handling for unmanaged subcloud When a subcloud is unmanaged and then managed again, it will never go back in sync. This is because all sync_statuses are set to unknown when the unmanage is done. However, the dc-cert_sync_status is only set to in-sync when the subcloud transitions from offline to online. So in this case the dc-cert_sync_status stays as unknown. The fix is to avoid setting the dc-cert sync status to unknown when a subcloud is unmanaged. The dc-cert sync status will still being audited for unmanaged subclouds so there is no need to do this. Change-Id: I7a53c11888b8866c6077679265d69c36faf4e264 Closes-Bug: 1887733 Signed-off-by: Bart Wensley commit 6408de3d7e36c8837cb7b37c670455cb80efca61 Author: Tao Liu Date: Tue Jul 14 09:48:20 2020 -0400 Reduce the patch audit interval Currently the patch audit assumes it is waking up every 20s so it will wait through 300/20 audit cycles. Should the subcloud audit take more than 30 seconds, the patch audit only runs about every 13-14 minutes. This update stores a timestamp for the last patch audit and compare with the current time when deciding if an audit is required. In addition, it triggers the load audit when a patch audit is forced. Story: 2007267 Task: 40342 Change-Id: I29e096fd3f2c82f42263f737390b64452bb606c1 Signed-off-by: Tao Liu commit 91e79f1d608489c39025695689343458e18da5c9 Author: albailey Date: Wed Jun 24 12:23:51 2020 -0500 Add a 'type' filter for update strategy API calls and DB filtering The CLI and manager classes can pass an additional 'type' filter so that upgrade and patch strategy commands are independent of each other. This allows the patch orchestration CLI to only show and have control over patch strategies. Horizon will also only show and manage patch strategies. Note: there can still only be one strategy in the DB at one time. When the api filter attribute for the update type is not passed to the URL, any strategy type can be queried. The 'create' and 'show' commands explicitly indicate when a mismatched strategy type is present. Change-Id: I40058b8cd5d09fa2b700f0a45d10a6b59b74778c Story: 2007403 Task: 40348 Signed-off-by: albailey commit 4e4355458d0f680bf150032000e1ca9b84682e2d Author: albailey Date: Wed Jun 17 11:05:01 2020 -0500 DC upgrade state machine states will select next state Rather than having the state machine states as an ordered list, this modification allows each state to specify the next state. This enhancement allows future features like compound states or a pre-check state to enable skipping over known states or alternative transitions. This adds a pre-check state. Only the success path has been tested so far. Change-Id: Ideae9885b6026cb441bcb6ac08204784b876f3a9 Story: 2007403 Task: 40347 Signed-off-by: albailey commit b5e45f15fe697295b18c62c678c745b3d798cd77 Author: Tee Ngo Date: Mon Jul 13 21:58:50 2020 -0400 Update resources purge logic The daily dcorch database purge cron job deletes resources that are no longer used by any orch jobs but as the subcloud_resource table is set to cascade delete it also removes subcloud resources that are still in use. This is then flagged by the next dcorch audit and the subcloud resources are re-created over the next audit cycle(s). In this commit, only purge resources that are not referenced by any orch jobs and subclouds. Closes-Bug: 1887430 Change-Id: I7179a2fba21f296dd51660bb224f5d51aaf531d8 Signed-off-by: Tee Ngo commit 3941475e5f17998b6ddbc3426be46db65135ac98 Author: Jessica Castelino Date: Wed Jun 24 13:50:11 2020 -0400 Subcloud-group based orchestration Whenever an orchestration strategy is created, it will ensure that application of patches or software upgrades would maintain the below ordering for the subclouds : 1) All subclouds in default group first. 2) Then all subclouds in the “first” group, then the “second”, etc… 3) Never include subclouds from different groups in the same “step” of the strategy, to ensure they are not patched at the same time. 4) If max_parallel_subclouds and subcloud_apply_type are provided while creating strategy, it will supercede the max_parallel_subclouds and update_apply_type values set for each subcloud group 5) If max_parallel_subclouds and subcloud_apply_type are not provided, we must respect the max_parallel_subclouds and the serial vs. parallel choice set for each subcloud group. Change-Id: Ifc98e303f70716f3c7e528167658617a6455ca65 Signed-off-by: Jessica Castelino Story: 2007403 Task: 40164 commit 241ad4b31df13d0026c2c987e75a5e84b5d1d86d Author: albailey Date: Tue Jul 7 12:20:38 2020 -0500 Allow audit for additional deploy status failed states Adding the deploy status failure states to the list of auditable criteria for subclouds. This allows subclouds to be set to offline by the audit rather than skipping the audit for them. Change-Id: Ia4ce5b36c7e108afd8da98412ee4bc8c29c3a49c Closes-Bug: #1885817 Signed-off-by: albailey commit 4f5d0b4fca0d9494f3acfd32349848558a9c34eb Author: Yuxing Jiang Date: Tue Jun 30 11:50:15 2020 -0400 Return master_id from get_resource_id for deleting SNMP resource This commit modified the get_resource_id method in SysinvSyncThread. The previous code only return community/ip_address which is for the creat service operation. By this change, the get_resource_id method in this class will return resource.master_id for deleting SNMP community and trapdest. Test: 1 Add snmp trapdest and snmp community configuration from system controller, with command: system snmp-comm-add -c test1 system snmp-trapdest-add -c test1 -i 10.11.2.1 2 Wait the snmp community and trapdest configured in subcloud in subcloud controller: system snmp-comm-list system snmp-trapdest-list 3 Delete the snmp community and trapdest in system controller: system snmp-trapdest-delete 10.11.2.1 system snmp-comm-delete test1 Expected: trapdest or community should be deleted from subcloud, result can also be observed in dcorch DB and /var/log/dcorch.log. Change-Id: I8d3b40599f83263cade1799d5a1dae72d3bac2c5 Closes-Bug: 1863045 Signed-off-by: Yuxing Jiang commit 4f1c5db8094fc82b8b1defb5e574cb3e54cd4035 Author: Jessica Castelino Date: Tue Jun 30 14:08:43 2020 -0400 Sync only ‘ssl_ca’ and ‘openstack_ca’ certificates to subclouds The 'docker_registry', 'tpm_mode', and 'openstack' cert on the SystemController will have a CN and SAN with OAM IP Address, any related DNS Names for OAM IP Address and MGMT IP Address. If these certificate are synched to the subclouds, even if they are signed by a trusted CA, when any client connects to the subcloud OAM or MGMT IP, the certificate that comes back will not pass certificate validation because the certificate does not apply to the subcloud's IP Address. For this reason a fix is put in place to avoid only sync "ssl_ca" and "openstack_ca" certificates to subclouds. Change-Id: I17ac94d79beb04f559c46062dbe7826590fcdb06 Signed-off-by: Jessica Castelino Closes-Bug: 1865643 commit 0d1799d810943d9d651ef919ca23f23f56c91d9d Author: albailey Date: Tue Jun 30 13:44:30 2020 -0500 Increase the DC orch upgrade-activation timeout Testing on one subcloud took 19 minutes for the upgrade activation to complete, and so it timed out. The timeout has been increased from 15 minutes to 45 minutes. Change-Id: Ice69acbbe097ea038e2932edb44f59f024cbeefb Closes-Bug: 1885451 Signed-off-by: albailey commit cfd225b75bdf355b5df7c0125078dd15c03584b5 Author: albailey Date: Thu Jun 25 09:28:31 2020 -0500 Adding a wait to the end of migrate_data state The ansible script triggers a reboot and the state needs to wait for the reboot to complete, before the state transitions to the next state. This adds a retry for the upgrade-activate step, and increases the total max time for that task to be 15 minutes This also fixes a logging bug where a failure in one subcloud causes excessive logs while other subclouds are being processed. Closes-Bug: 1885132 Change-Id: Iabca2ecb0bacff7c5622c90722fcfb938a71a768 Story: 2007403 Task: 40172 Signed-off-by: albailey commit 8ecba753119da83e80f4933bd24dac311b5979f1 Author: Gerry Kopec Date: Thu Jun 25 14:16:45 2020 -0400 Add timeouts to DC sysinv and patching clients If a subcloud is powered off or loses communication with the system controller while the subcloud audit is waiting for a response to a sysinv or patching request, the subcloud audit will hang forever. Add some failsafe 10 minute timeouts to ensure the audit can recover. Change-Id: I369225119f89ab5a2f05137aba955dda982d51c5 Closes-Bug: 1884560 Closes-Bug: 1870413 Signed-off-by: Gerry Kopec commit dd9c780663af60666e82d2441e242dd6ca1e72f9 Author: Tao Liu Date: Tue Jun 23 17:02:19 2020 -0400 Add CLI/API to populate data_install The dcmanager subcloud update API is extended to allow populating/updating the data_install values required for remote install (upgrade to stx 4.0): In addition, the bmc-password is stored as base64encoded. Test cases: 1. Add a subcloud without install-values and update the subcloud with install-values 2. Update the subcloud install-values with new data 3. Verify the bmc_password is stored as base64encoded 4. Add a subcloud with install-values and verify the bmc_password is populated in rvmc.conf as base64encoded 5. Subcloud update with the previously supported parameters Story: 2007403 Task: 40149 Change-Id: I78db37873547fde43f085a0093765f838d2e4200 Signed-off-by: Tao Liu commit a331e544fa16b8cee20c7cc392da1b34805e6c1e Author: Jessica Castelino Date: Tue Jun 23 11:31:16 2020 -0400 Distributed Cloud re-execution susceptible to old data Added an os_O_TRUNC flag to ensure that files are truncated before writing new content to it. This avoids previous content from being appended to the new file. Change-Id: I14438d4276376089c1c46a20626f251adcac09f9 Closes-Bug: 1883113 Signed-off-by: Jessica Castelino commit e422388185102616b35946a6bd04d038e415380a Author: albailey Date: Thu Jun 11 15:22:22 2020 -0500 Add thread loop breakout handling for DC upgrade states If a state machine handler has internal loops, it can check to see if the main thread is being stopped, and can break out early. The states performing this check are: - importing load (can take up to 30 minutes) - starting (can take up to 30 minutes) - lock controller (can take up to 10 minutes) - unlock controller (can take up to an hour) - activating (can take up to 10 minutes) - completing (can take up to 10 minutes) Added a loop-check for 'starting' state to wait for upgrade to move to 'started'. Added a loop-check for 'completing' state to wait for upgrade to be deleted. Change-Id: I2d4e77d1baf917738ddc96f3cbaa76249ba17fdf Story: 2007403 Task: 40105 Signed-off-by: albailey commit 36dce2d5f072ba464e9cb1fe82e80bf14c6dc943 Author: albailey Date: Wed Jun 10 13:26:55 2020 -0500 DC upgrade state handler improvements 1) DC upgrade importing-load state should wait for 'imported' The importing_load state for software upgrade orchestration was moving to the next state after the load was 'importing' instead of 'imported'. This caused the next state to fail since it required the load to be in the imported state. 2) Use the strategy options object 'alarm-restriction-type' to determine if an 'upgrade start' should use a force to ignore alarms that are not management affecting. 3) Add polling stage after upgrade activation to wait for the activity to complete before transitioning to the next state. 4) Update the unit tests to mock timeout values so that running the unit tests is quicker. Change-Id: Ibaf77d2c33b633f915301344c7a7318a59fe3a9f Story: 2007403 Task: 40040 Signed-off-by: albailey commit 1012dd28964b7416e24b3dd60b217988d137e640 Author: John Kung Date: Tue Jun 9 10:22:56 2020 -0400 Implement the state for Simplex Upgrade The Simplex Upgrade step upgrades the simplex subcloud to the load imported in dc-vault corresponding to the target software release. Required static data is captured at subcloud add and referenced for the simplex upgrade. Dynamic data is retrieved from the online subcloud; including the bmc credentials which are obtained via a synchronized service within project allowing for access to the bmc credentials. BMC password retrieval endpoint_cache integration with https://review.opendev.org/#/c/736004/ Tests Performed: - Subcloud upgrade (install) with load corresponding to System Controller - Subcloud add persists install data - Simplex upgrade persists dynamic upgrade data from start of upgrade simplex - Obtain bmc password from online subcloud - Obtain bmc password from install data, in case the bmc is not configured Pending for subsequent commit: - tox unit tests Change-Id: Ie047139280a5780bfe47b9f9959f4c6781d6f3fa Story: 2007403 Task: 40024 Signed-off-by: John Kung commit 94946613cdb51e1a7b9c08ad8344d120da0ed59f Author: Tee Ngo Date: Fri Jun 12 13:14:57 2020 -0400 Implement simplex subcloud data migration state This commit implements the data migration state for subcloud upgrade orchestration. Story: 2007403 Task: 40071 Depends-On: https://review.opendev.org/#/c/734620/ Depends-On: https://review.opendev.org/#/c/733902/ Depends-On: https://review.opendev.org/#/c/733914/ Change-Id: I5b38f31324ddb62b6760f897a1eb3cca5159e2ef Signed-off-by: Tee Ngo commit d10fef48217d69a45084dd8b81c30d3bbe68b3af Author: Tao Liu Date: Tue Jun 16 15:08:20 2020 -0400 Configure dcmanager user for endpoint_cache The following changes are made, in order to remove the dependencies to 'admin' user and use 'dcmanager' user to authenticate with services in the subclouds: . Add endpoint_cache section to both dcorch and dcmanager . Configure dcmanager user in the endpoint_cache section . Sync the dcmanager user to subclouds . Use 'dcmanager' user for subcloud authentication by default, and it falls back to 'admin' user if the authentication fails (This is required during software upgrade). Depends-On: https://review.opendev.org/#/c/735994/ Partial-Bug: 1883758 Change-Id: Ibc78abc86a7a825f83f2cac9fd54e4183e7ccd80 Signed-off-by: Tao Liu commit e8d916756d439cf63eed52f347a2a547fc5124cb Author: Andy Ning Date: Wed Jun 17 10:40:23 2020 -0400 Set up /var/run/dcmanager as dcmanager's default temp files location dcmanager call the k8s python client to perform a number of operations. The k8s python client creates temp files under /tmp and continues use these tmp files for the life-cycle of the processes. However systemd-tmpfiles-clean.service will run every day to clean up files in /tmp dir that are older than 10 days. If the k8s client code is not triggered for more than 10 days (thus its temp files are not accessed for more than 10 days), these temp files will be removed as part of the cleanup. Certain dcmanager operations then starts to fail with an error that the tmp file is no longer there. This is a known issue of kubernetes python client: https://github.com/kubernetes-client/python/issues/765 The commit fixes this issue by setting TMPDIR to /var/run/dcmanager when sm starts dcmanager-manager. Change-Id: Ib147c2ab26e303032e18da51a506e3768bc471e0 Closes-Bug: 1883599 Signed-off-by: Andy Ning commit dee5e270a3752ac47f4282cb12a5c273e093068b Author: Bin Qian Date: Tue Jun 16 16:51:55 2020 -0400 Ensure subcloud cert data is intact In reported cases subcloud bootstrap only received tls.key. With tls.crt and root_ca_crt are both empty. Solution is wait for the data values are not empty. Change-Id: I01d71f5d000247f03d2a32eb37761f3e186ac863 Closes-Bug: 1883777 Signed-off-by: Bin Qian commit ed652edb38414343528bd78378fde06f7af5b66a Author: Bin Qian Date: Fri Jun 12 10:50:09 2020 -0400 Update dc-cert subcloud status endpoint when subcloud online Set it to in-sync and will be replaced with dc-cert validation result (coming soon). Ensure updating dc-cert status before subcloud is unmanaged. This is because the subcloud is bootstrap with in-sync dc-cert. And dc-cert is audit when subcloud is online (before set to managed). Change-Id: I2e6aa597e53da5e26f44430767eb6fd4d3e784ee Story: 2007347 Task: 40070 Signed-off-by: Bin Qian commit c1da0762d58182171c12aca783c8f00c9ef2ac2b Author: Sharath Kumar K Date: Tue May 5 06:28:22 2020 +0200 Tox and Zuul job for the bandit code scan in stx/distributedcloud Setting up the bandit tool for the scanning of HIGH severity issues in the python codes under Starlingx/distributedcloud folder. Expecting this merge will enable zuul job for CI/CD of bandit scan. Configuration files: 1. tox.ini for adding bandit environment and command. 2. test-requirements.txt for adding bandit version. 3. .zuul.yaml file for adding bandit job and configuring under check job to run code scan every time before code commit. Test: Run tox -e bandit command inside the fault folder to validate the bandit scan and result. Please note: Changes will be implemented in batches and this is Batch6 change. Story: 2007541 Task: 39648 Depends-On: https://review.opendev.org/#/c/721294/ Change-Id: I6fdfeacdb000faafc73c44eef352bf2abd38a9a2 Signed-off-by: Sharath Kumar K commit cd12fc31a1702f08f77fedcaa6386825d9ae3c6c Author: albailey Date: Thu Jun 11 08:43:21 2020 -0500 Add reboot handling to unlock state for DC upgrades When an unlock is triggered, the host is rebooted. The maximum amount of time for this state to complete is 1 hour. - Reboot time (30 minutes) - pre and post reboot time (30 minutes) The unlock state handler for upgrade orchestration will now loop and ignore failures for a period of time, while waiting for the host to become unlocked / enabled. Change-Id: Ie7ad8dbfecd6ffbe241d098c8677a8e25f400b07 Story: 2007403 Task: 40048 Signed-off-by: albailey commit 4fd65e9913d28bd097b16ad3bfb80c04ca6b5419 Author: Jessica Castelino Date: Mon May 25 13:51:18 2020 -0400 CLI command to deploy a subcloud If deployment failed, the user has no option than to delete and re-add it. If the user was re-adding the subcloud without re-installing, it would further result in a bootstrap failure. Thus, to simplify things, a new CLI command is provided to allow re-deployment. Furthermore, if the user still chooses to delete the subcloud and re-add it without a re-install, a better error message is provided asking them to re-install the host. CLI: dcmanager subcloud reconfig --deploy-config Test Cases: 1) Successfully add a subcloud with or without deployment option 2) Fail to re-add a subcloud without re-installation after a failed deployment 3) Re-deploy with new CLI command after successful and unsuccessful deployment 4) Re-deploy with new CLI command before and after the subcloud is unlocked 5) Test new CLI command by passing wrong parameters Change-Id: I9fe7e3791e3887160668281048c3c12a7f40c2af Partial-Bug: 1864756 Signed-off-by: Jessica Castelino commit 66f9341692336e86ee74712dc05e816fae5397ba Author: Jessica Castelino Date: Tue Jun 9 00:34:14 2020 -0400 Manage subcloud only when deploy_status is "complete" Subcloud can be managed only if the deploy status is "complete" i.e. either subcloud is successfully bootstrapped or subcloud is successfully deployed. This will prevent partially configured subclouds from being managed and put into service. Change-Id: Ic4562c250e651f6df64f368db641d7c58aee887c Closes-Bug: 1882786 Signed-off-by: Jessica Castelino commit 5e5c416a9b56d6c7385e6883a73c8b29ebce4709 Author: albailey Date: Tue Jun 2 15:51:32 2020 -0500 Implement import-load and starting-upgrade strategy states This commit allows the state machine to be defined through an ordered list of states. The states are mapped to by a dictionary to state handler classes. This greatly simplifies the software upgrade orch thread so that it can be refactored in the future to share functionality with other orch thread strategy types. The following states have been implemented: - 'importing load' - 'starting upgrade' - 'activating upgrade' - 'completing upgrade' The 'installing license' state now uses this design pattern. Its functionality and unit tests are now in their own files. The 'unlocking' state requires additional changes for the 'activating' and 'completing' states to pass. Change-Id: Iedc40bdea406461dd4c2ce3887a37be6ec3c999f Story: 2007403 Task: 40001 Signed-off-by: albailey commit 70f4dc8537ae89cf2eb70a78bda64073f13b02a1 Author: Jessica Castelino Date: Tue Jun 9 13:43:04 2020 -0400 Subcloud deploy fails Deploy config generated by dcmanager does not truncate the existing file after it has been overridden. This leaves some invalid content at the end of the newly generated file. Fixes have been made for this. Change-Id: I6c7e7fbacb1ee323c3675076083e046498698425 Closes-Bug: 1882826 Signed-off-by: Jessica Castelino commit efcfc7a45a170c51ea5094fd62cb7047b24fc92d Author: Andreas Jaeger Date: Thu Jun 4 14:16:23 2020 +0200 Switch to newer openstackdocstheme and reno versions Switch to openstackdocstheme 2.2.1 and reno 3.1.0 versions. Using these versions will allow especially: * Linking from HTML to PDF document * Allow parallel building of documents * Fix some rendering problems Update Sphinx version as well. Disable openstackdocs_auto_name to use 'project' variable as name. Change pygments_style to 'native' since old theme version always used 'native' and the theme now respects the setting and using 'sphinx' can lead to some strange rendering. openstackdocstheme renames some variables, so follow the renames before the next release removes them. A couple of variables are also not needed anymore, remove them. See also http://lists.openstack.org/pipermail/openstack-discuss/2020-May/014971.html Change-Id: I7da2bdaafe86b9eb930202bef3742571daa90e17 commit 125e465aff6801940f25762d7be2406d01349d18 Author: Tee Ngo Date: Mon May 25 12:16:55 2020 -0400 Extend sysinv api proxy to support load operations In this commit, the dcorch-sysinv-api-proxy is extended to support load import and delete requests. Upon receiving a successful response from sysinv api for load import/delete request, the load is saved to/removed from dc-vault accordingly. Tests: - Successful load import - Successful load delete (no subclouds have the deleted load) - Successful load delete (one subcloud has the deleted load) - Failed load import (exceeding limit) - Failed load import (bad signature) - Failed load delete (does not exist) Unit tests will be added via a separate commit under task 39903 of story 2007082. Story: 2007403 Task: 39840 Depends-On: https://review.opendev.org/#/c/730632 Change-Id: If2769b0faf093523e7e9bc97b8cdc6a5513534aa Signed-off-by: Tee Ngo commit 63c007461d782158e62aa082d74c44c452969883 Author: albailey Date: Mon May 25 12:55:20 2020 -0500 Implementing Lock and Unlock upgrade strategy states Implement the states for lock and unlock. Adding stub methods and classes for the full upgrade strategy subcloud orchestration state machine. Adding unit tests to validate all lock and unlock success and failure scenarios. Change-Id: I5b8d89bc733b867e8568043b0c7e4a79d1925890 Story: 2007403 Task: 39874 Signed-off-by: albailey commit 2f49219858ed3c3a872dcaa337ba601a68fce7e4 Author: Tao Liu Date: Tue May 26 16:20:45 2020 -0400 Combine the patch audit with the subcloud audit This commit moves the patch audit to dcmanager-audit process. Patch audit does not run at the same frequency as subcloud audit, its normal interval is approx. 5 mins, but it can be forced to run on the next audit interval by calling trigger_patch_audit RPC. Change-Id: I4d4fdd0cebe9f3f65241107c92bbc4d26c13c8f3 Story: 2007267 Task: 39864 Signed-off-by: Tao Liu commit 30f2147c398eb56fb67cb7b777560b3a18529ba5 Author: Tao Liu Date: Mon May 25 13:40:16 2020 -0400 Close any temp files after upload a patch Add a cleanup routine in the patch-api-proxy to close any temporary files that opened by third-party libraries, during a patch upload. Verified that the temp files are close and /scratch usage is not increased after a few of large patch upload. Verified that the temp files are close and /scratch usage is not increased after a failed upload. Change-Id: I68453653fe09d8b1ad80289ab3ff33e476b3acee Closes-Bug: 1880587 Signed-off-by: Tao Liu commit 566076a0a8808ee958f5edb4fb9333f426bf1490 Author: Gerry Kopec Date: Wed May 20 02:30:03 2020 -0400 Fix slow subcloud manage due to sysinv auth Synchronize keystone services project and sysinv user ids between the system controller and subcloud during subcloud bootstrap. This fixes issue where the subcloud would go offline on initial manage for 1 hour until keystone token expires. Change-Id: I2b36861df0858920eed308bd8e9a12b49b68a191 Closes-Bug: 1877419 Closes-Bug: 1877584 Signed-off-by: Gerry Kopec