Error on system restore attempt with swift enabled

[Jose Perez Carranza]

Title
-----
Failing to apply system restore with error "Failed to execute controller manifest"

Brief Description
-----------------
When trying to restore the system is giving a failure on step 8 due "Failed to execute controller manifest". More information provided on the Timestamps and logs

Severity
--------
Provide the severity of the defect.
Major

Steps to Reproduce
------------------
1. Have a functional and working deployment installed
2. Generate backups files
   $ sudo config_controller --backup <backup_name>
3. Save the file generated
4. Reinstall the system
5. Attempt to restore the system
   $ sudo config_controller --restore-system <backup_name_system.tgz>

Expected Behavior
------------------
Restore of the system should be completed

Actual Behavior
----------------
Step 8 of 25 [############## ] [32%] Failed to execute controller manifest

Reproducibility
---------------
Reproducible 100%

System Configuration
--------------------
Duplex Bare-Metal Two node system

Branch/Pull Time/Commit
-----------------------
2018.10

Timestamp/Logs
--------------
Applying the manifest is launching Warnings about missing files but files are already present.. seems to be a race condition on the system. more information below:

controller-0:~$ sudo grep -E '^(.......)?Warning|^....-..-..T..:..:..([.]...)?(.......)?.Warning|^(.......)?Error|^....-..-..T..:..:..([.]...)?(.......)?.Error' /var/log/puppet/latest/puppet.log
2018-10-21T21:34:20.225 Warning: 2018-10-21 21:34:20 +0000 Unexpected line: Ring file /etc/swift/object.ring.gz not found, probably it hasn't been written yet
2018-10-21T21:34:20.904 Warning: 2018-10-21 21:34:20 +0000 Unexpected line: Ring file /etc/swift/container.ring.gz not found, probably it hasn't been written yet
2018-10-21T21:34:23.024 Warning: 2018-10-21 21:34:23 +0000 Unexpected line: Ring file /etc/swift/account.ring.gz not found, probably it hasn't been written yet

controller-0:~$ ll /etc/swift/ |grep gz
-rw-r--r-- 1 root root 204 Oct 21 21:34 account.ring.gz
-rw-r--r-- 1 root root 206 Oct 21 21:34 container.ring.gz
-rw-r--r-- 1 root root 202 Oct 21 21:34 object.ring.gz

[Bruce Jones (brucej)]

Only fails on duplex bare metal setup. Might be setup specific. Erich to investigate this more deeply.

[Wei Zhou (wzhou007)]

I guess swift was enabled at the time system backup was done. Just wondering if swift was enabled through system parameters, like this:

system service-parameter-modify swift config service_enabled=true
system service-parameter-modify swift config fs_size_mb=50 (optional)
system service-parameter-apply swift

[Jose Perez Carranza (jgperezc)]

I was executing some tests before the backup and Yes one of those test cases included a swift enabling with those commands.

[Ghada Khalil (gkhalil)]

stx.2019.03 - confirmed to be an issue, but only with swift enabled (which is not a common configuration)

[Frank Miller (sensfan22)]

Backup & restore with Swift is only supported when Swift is configured on CEPH. Backup & restore with swift configured on the controllers is not supported [1].

[1] https://storyboard.openstack.org/#!/story/2003518