Error handling on Kubernetes VIM

[Yi Feng]

After the following patches are merged, only the Instantiate/Terminate/Heal/Scale VNF has been verified.
https://review.opendev.org/c/openstack/tacker/+/778948

The verification item is added this time, changing the verification object from VNF to CNF or using MgmtDriver to install k8s. The verification content is the correctness of the retry operation after the LCM operation fails.

In this verification, multiple BUGs were verified, so new patches are needed to correct them.

1. The retry operation of LCM fails because the status of vnf is ERROR.
2. When the CNF scale out retry, there is no scaleGroupDict, so the retry fails.
3. When the terminate retry, the task_state of vnf_instance is ERROR, so the retry fails.
4. When the scale out retry and before_error_point is greater than 4, due to lack of resource_changes，so the retry fails.
5. When instantiate/heal/terminate retry and before_error_point is 1, the retry fails because the status of vnf is ERROR.
6. When scale in/heal retry and before_error_point is 2, because the node has been deleted, if it is deleted again, the retry will fail.
7. When instantiate retry and before_error_point is 3, the stack has been created by heat, and if it is created again, the retry will fail.
8. When instantiate retry and before_error_point is 6,
   (1) Because the master/worker node has been installed, if it is installed again, the retry will fail.
   (2) As VIM is already registered, if you register again, the retry will fail.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tacker/+/807289

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tacker/+/807289
Committed: https://opendev.org/openstack/tacker/commit/382afbabcf1273a2e83ae2e8fd06b944fcfecad4
Submitter: "Zuul (22348)"
Branch: master

commit 382afbabcf1273a2e83ae2e8fd06b944fcfecad4
Author: Yi Feng <email address hidden>
Date: Fri Sep 3 19:44:22 2021 +0900

    Fix retry operation after LCM failure

    The LCM retry operation of CNF and MgmtDriver(k8s-cluster)
    was verified.

    To fix these errors, the following things is done in this patch:
    1. When executing retry, if the status of vnf is ERROR, change ERROR to
    the status of the corresponding lifecycle.
    2. In the scale operation, the judgment condition is added. When vim is
    not kubernetes, the `scaleGroupDict` assignment will be used.
    3. When an exception occurs in terminate, the task_state in
    vnf_instance is changed to None.
    4. In the scale operation, add an else branch. If before_error_point is
    greater than 4, get resource_changes from vnf_info first. If
    resource_changes does not exist, get it by calling the
    self._scale_resource_update method.
    5. At the beginning of instantiate/heal/terminate, when modifying the
    status of vnf, add the judgment condition. When the status is not
    ERROR, it can be modified.
    6. In the heal/scale operation, add a judgment condition to judge
    whether the node exists, and only when the node exists, the delete node
    command will be executed.
    7. In the instantiate operation, before creating the stack, add to
    determine whether the stack already exists.
    8. (1) When installing the k8s node with MgmtDriver, add to determine
       whether the node already exists in the cluster. If it exists, skip
       the node and install the next node.
       (2) After installing the k8s cluster, when registering VIM, add to
       determine whether the VIM already exists, and if it exists, use it
       directly.

    Closes-Bug: #1942277
    Change-Id: If4e695120eb432155499a2e866ae651a18a320ab

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tacker 7.0.0.0rc1 release candidate.