Load import via sysinv-api proxy could fail in slow vbox

Bug #1884694 reported by Tee Ngo
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
StarlingX
Fix Released
High
Tee Ngo

Bug Description

Brief Description
-----------------
Load import via sysinv api proxy could fail in vbox if the load files are removed by sysinv-conductor from staging directory before they are copied over to dc-vault by sysinv api proxy.

Severity
--------
Major

Steps to Reproduce
------------------
Import a 20.06 load on the system controller in vbox by running the command:
system --os-region-name SystemController load-import bootimage.iso bootimage.sig

Expected Behavior
------------------
Load import via sysinv api proxy completes successfully

Actual Behavior
----------------
Load import via sysinv api proxy failed as the copy of bootimage.iso file to dc-vault took longer in vbox and did not complete before sysinv-conductor removes the load files from staging director.

This issue was only observed in a distributed cloud system in one designer's vbox running 20.01 load. The issue was not observed in a distributed cloud system in another designer's vbox running 20.06 load or on hardware.

Reproducibility
---------------
100% reproducible on the designer with slow vbox

System Configuration
--------------------
IPv4

Branch/Pull Time/Commit
-----------------------
Latest 20.01 load + load import patch

Last Pass
---------
Yes this test passed on hardware and also passed in vbox running 20.06 load.

Timestamp/Logs
--------------
N/A

Test Activity
-------------
Developer Testing

Workaround
----------
Describe workaround if available

Tee Ngo (teewrs)
Changed in starlingx:
assignee: nobody → Tee Ngo (teewrs)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to config (master)

Fix proposed to branch: master
Review: https://review.opendev.org/737430

Changed in starlingx:
status: New → In Progress
tags: added: stx.distcloud
Ghada Khalil (gkhalil)
Changed in starlingx:
importance: Undecided → High
tags: added: stx.4.0
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to config (master)

Reviewed: https://review.opendev.org/737430
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=24c88658ca94c22401f5077e2f8cd925cd179113
Submitter: Zuul
Branch: master

commit 24c88658ca94c22401f5077e2f8cd925cd179113
Author: Tee Ngo <email address hidden>
Date: Mon Jun 22 22:45:24 2020 -0400

    Implement a short wait before removing load files

    Load import could fail in slow vbox or hardware if the copying
    of the large iso file by sysinv api proxy takes longer than
    the time sysinv conductor imports the load.

    In this commit, the load import thread in sysinv conductor waits
    30 seconds before removing the load files from the staging directory.
    This gives sysinv api proxy ample amount of time to copy the load
    files to dc-vault. The wait does not apply to requests not going
    through the proxy. Sysinv api proxy will be refactored in a future
    commit to enable copying of load files directly from the import
    request.

    Closes-Bug: 1884694
    Change-Id: I7ba4af8d32707d3c0977b29c93ab89894bce70cd
    Signed-off-by: Tee Ngo <email address hidden>

Changed in starlingx:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.