Unlock host fails if an application is applied

[Yang Liu]

Brief Description
-----------------
system host-unlock controller is rejected with following error if stx-openstack is applied:

perform_app_apply() takes exactly 5 arguments (4 given)

Severity
--------
Critical

Steps to Reproduce
------------------
1. Install and configure a system
2. Upload and apply stx-openstack application
3. Attempt to lock/unlock standby controller

TC-name: test_lock_unlock_host[controller]

Expected Behavior
------------------
controller lock/unlock succeeds

Actual Behavior
----------------
# controller unlock rejected with following error:
perform_app_apply() takes exactly 5 arguments (4 given)
# controller is still unlocked, but stx-application gets stuck at applying state since then with progress=None

Reproducibility
---------------
Reproducible

System Configuration
--------------------
This issue is seen on following systems: Two node system, Multi-node system, Dedicated storage

Lab-name:
wcp113-121,sm5-6,wcp99-103

Branch/Pull Time/Commit
-----------------------
stx master as of 20190510T013000Z

Last Pass
---------
SM_5_6 [20190509T013000Z]

Timestamp/Logs
--------------
for wcp99-103:

[2019-05-10 08:39:18,361] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne host-unlock controller-1'
perform_app_apply() takes exactly 5 arguments (4 given)
[wrsroot@controller-0 ~(keystone_admin)]$

[2019-05-10 09:47:34,039] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne application-list'
+---------------+--------------------------------+-----------------+---------------+----------+----------+
| application | version | manifest name | manifest file | status | progress |
+---------------+--------------------------------+-----------------+---------------+----------+----------+
| stx-openstack | 1.0-11-centos-stable-versioned | armada-manifest | manifest.yaml | applying | None |
+---------------+--------------------------------+-----------------+---------------+----------+----------+
[wrsroot@controller-0 ~(keystone_admin)]$

Test Activity
-------------
Sanity

[Ghada Khalil (gkhalil)]

Marking as release gating; basic operations (unlock) failing. Appears to be related to recent code submission.

[Ghada Khalil (gkhalil)]

Fixed By: https://review.opendev.org/#/c/658397/

Merged on May 10 / 2019 -- marking as Fix Released manually as the bug was not linked properly in the gerrit review.

[Al Bailey (albailey1974)]

2019-05-16 17:01:25.723 95016 ERROR wsme.api [-] Server-side error: "perform_app_apply() takes exactly 5 arguments (4 given)". Detail:
Traceback (most recent call last):

  File "/usr/lib/python2.7/site-packages/wsmeext/pecan.py", line 85, in callfunction
    result = f(self, *args, **kwargs)

  File "/usr/lib64/python2.7/site-packages/sysinv/api/controllers/v1/host.py", line 1729, in patch
    return self._patch_gen(uuid, patch, profile_uuid)

  File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
    return f(*args, **kwargs)

  File "/usr/lib64/python2.7/site-packages/sysinv/api/controllers/v1/host.py", line 1737, in _patch_gen
    return self._patch(uuid, patch, profile_uuid)

  File "/usr/lib64/python2.7/site-packages/sysinv/api/controllers/v1/host.py", line 2099, in _patch
    self._reapply_system_app()

  File "/usr/lib64/python2.7/site-packages/sysinv/api/controllers/v1/host.py", line 2475, in _reapply_system_app
    pecan.request.context, db_app, False)

[Ghada Khalil (gkhalil)]

Issue reproduced on a more recent load which includes the fix: 2019-05-15_18-01-07. Bug re-opened.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/659683

[Peng Peng (ppeng)]

[2019-05-16 17:01:19,780] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne host-unlock controller-1'
[2019-05-16 17:01:25,759] 387 DEBUG MainThread ssh.expect :: Output:
perform_app_apply() takes exactly 5 arguments (4 given)

[Peng Peng (ppeng)]

Issue was reproduced on

Lab: WCP_99_103
Load: 2019-05-15_18-01-07

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/659683
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=8a11142e2fb2a666b91102ef44cd5d933cc2f382
Submitter: Zuul
Branch: master

commit 8a11142e2fb2a666b91102ef44cd5d933cc2f382
Author: Wei Zhou <email address hidden>
Date: Thu May 16 18:38:55 2019 -0400

    Unlock host fails if an application is applied

    A recent commit 483482aede588e90d2ac5886600d7a794ca9cc27
    added a new parameter "mode" to method perform_app_apply() in
    rpcapi.py. The reference to this method when the application
    is reapplied needs to be updated.

    Change-Id: I730dcead336449e9db60bfee566eee6fcb7af132
    Closes-Bug: 1828562
    Signed-off-by: Wei Zhou <email address hidden>

[Yang Liu (yliu12)]

This issue is no longer seen in WR daily sanity.

Load:
2019-05-28_17-05-57