Storage nodes go for extra reboot after unlock (manifest apply failure)

[Maria Yousaf]

Brief Description
-----------------
Storage nodes go for extra reboot after unlock, after assigning OSD in newly created storage tier

Severity
--------
Major

Steps to Reproduce
------------------
1. Create a new storage tier
2. Lock a storage node
3. Assign an unused OSD on that storage node to the newly created storage tier
4. Unlock the storage node*
5. Repeat for all storage nodes

*On reboot, there is a failure displayed on the console: failed to apply puppet manifest. The storage node goes for a reboot again. Eventually, the node does come up.

Looking at the puppet manifest on storage-0, the following is seen:

2018-10-10T13:35:25.481 ^[[mNotice: 2018-10-10 13:35:25 +0000 /Stage[post]/Platform::Config::Storage::Post/File[/etc/platform/.initial_storage_config_complete]: Dependency Exec[ceph-osd-prepare-/dev/disk/by-path/pci-0000:85:00.0-nvme-1] has failures: true

Expected Behavior
------------------
A single reboot cycle should bring up the node and no manifest failures should be seen.

Actual Behavior
----------------
Manifest failures seen on unlock resulting in additional reboot.

Reproducibility
---------------
Extra reboot seen on multiple storage nodes in the system.

System Configuration
--------------------
Multi-node storage system

Branch/Pull Time/Commit
-----------------------
master as of 2018-10-09_01-52-01

Timestamp/Logs
--------------
 2018-10-10T13:35:25.481

[Ghada Khalil (gkhalil)]

stx.2019.03 - specific to storage tiers; system still comes up after extra reboot. Not required for stx.2018.10

[Ghada Khalil (gkhalil)]

Duplicate of https://bugs.launchpad.net/starlingx/+bug/1800889

[Ghada Khalil (gkhalil)]

Fix merged as of Nov 1/2018