Zuul check failed for functional test

[Yasufumi Ogawa]

Although several fixes have been applied, some failures still remained in `tacker-functional-devstack-multinode-python3`. Some of failures don’ t always occur, but occasionally. Here is one of the examples.

```
...
{1} tacker.tests.functional.nfvo.test_vim.VimTestCreate.test_duplicate_vim [2.708430s] ... ok
{1} tacker.tests.functional.vnflcm.test_vnf_instance.VnfLcmTest.test_create_show_delete_vnf_instance [0.330047s] ... ok
{0} tacker.tests.functional.common.test_cmd_executer.TestRemoteCommandExecutor.test_cmd_executor [28.750754s] ... FAILED

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):

      File "/home/zuul/src/opendev.org/openstack/tacker/tacker/tests/functional/common/test_cmd_executer.py", line 82, in test_cmd_executor
    'test_tosca_vnf_with_cirros_inline')

      File "/home/zuul/src/opendev.org/openstack/tacker/tacker/tests/functional/common/test_cmd_executer.py", line 60, in _test_cmd_executor
    host=mgmt_ip)

      File "/home/zuul/src/opendev.org/openstack/tacker/tacker/common/cmd_executer.py", line 59, in __init__
    self.__connect()

      File "/home/zuul/src/opendev.org/openstack/tacker/tacker/common/cmd_executer.py", line 66, in __connect
    password=self.__password, timeout=self.__timeout)

      File "/home/zuul/src/opendev.org/openstack/tacker/.tox/dsvm-functional/lib/python3.6/site-packages/paramiko/client.py", line 368, in connect
    raise NoValidConnectionsError(errors)

    paramiko.ssh_exception.NoValidConnectionsError: [Errno None] Unable to connect to port 22 on 192.168.120.74

{0} tacker.tests.functional.nfvo.test_nfvo.NsdTestCreate.test_create_delete_nsd [1.450342s] ... ok
...
{0} tacker.tests.functional.vnfm.test_tosca_vnf.VnfTestToscaCreate.test_create_delete_vnf_from_template [0.570376s] ... FAILED

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):

      File "/home/zuul/src/opendev.org/openstack/tacker/tacker/tests/functional/vnfm/test_tosca_vnf.py", line 116, in test_create_delete_vnf_from_template
    'inline')

      File "/home/zuul/src/opendev.org/openstack/tacker/tacker/tests/functional/vnfm/test_tosca_vnf.py", line 94, in _test_create_delete_vnf_tosca
    template_source)

      File "/home/zuul/src/opendev.org/openstack/tacker/tacker/tests/functional/vnfm/test_tosca_vnf.py", line 42, in _test_create_vnf
    vnfd_file, vnf_name)

      File "/home/zuul/src/opendev.org/openstack/tacker/tacker/tests/functional/base.py", line 385, in vnfd_and_vnf_create_inline
    vnf_instance = self.client.create_vnf(body=vnf_arg)

      File "/home/zuul/src/opendev.org/openstack/tacker/.tox/dsvm-functional/lib/python3.6/site-packages/tackerclient/v1_0/client.py", line 974, in create_vnf
    return self.legacy_client.create_vnf(body)

      File "/home/zuul/src/opendev.org/openstack/tacker/.tox/dsvm-functional/lib/python3.6/site-packages/tackerclient/v1_0/client.py", line 115, in with_params
    ret = self.function(instance, *args, **kwargs)

      File "/home/zuul/src/opendev.org/openstack/tacker/.tox/dsvm-functional/lib/python3.6/site-packages/tackerclient/v1_0/client.py", line 492, in create_vnf
    return self.post(self.vnfs_path, body=body)

      File "/home/zuul/src/opendev.org/openstack/tacker/.tox/dsvm-functional/lib/python3.6/site-packages/tackerclient/v1_0/client.py", line 338, in post
    headers=headers, params=params)

      File "/home/zuul/src/opendev.org/openstack/tacker/.tox/dsvm-functional/lib/python3.6/site-packages/tackerclient/v1_0/client.py", line 244, in do_request
    self._handle_fault_response(status_code, replybody)

      File "/home/zuul/src/opendev.org/openstack/tacker/.tox/dsvm-functional/lib/python3.6/site-packages/tackerclient/v1_0/client.py", line 187, in _handle_fault_response
    exception_handler_v10(status_code, des_error_body)

      File "/home/zuul/src/opendev.org/openstack/tacker/.tox/dsvm-functional/lib/python3.6/site-packages/tackerclient/v1_0/client.py", line 83, in exception_handler_v10
    status_code=status_code)

    tackerclient.common.exceptions.Conflict: vnf already exist with given ['tenant_id', 'name', 'deleted_at']

{0} tacker.tests.functional.vnfm.test_tosca_vnf.VnfTestToscaCreate.test_create_delete_vnf_static_ip [34.204727s] ... ok
...
```

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/754873

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/754873
Committed: https://git.openstack.org/cgit/openstack/tacker/commit/?id=b4d49961b968168b4654ca2449b16b6d7840fbd0
Submitter: Zuul
Branch: master

commit b4d49961b968168b4654ca2449b16b6d7840fbd0
Author: Itsuro Oda <email address hidden>
Date: Tue Sep 29 10:39:24 2020 +0900

    Fix functional test of RemoteCommandExecutor

    This patch fixes three problems to solve test_cmd_executor funciotnal
    test failure.

    * make the metadata server accessible from VMs
    * connect the compute1 node to the controller node via net_mgmt
    * wait while after VM start until sshd start

    Change-Id: Id5b72b7eae4ecf6b8038f06b84567324edaa98a8
    Closes-bug: #1897643
    Related-bug: #1886213

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/760274

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/760276

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/760278

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/760279

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/760280

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/760281

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/760282

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/760278
Committed: https://git.openstack.org/cgit/openstack/tacker/commit/?id=180ff8ae4ac5f4f6ca130a11ad875c9e7ae6d7f9
Submitter: Zuul
Branch: master

commit 180ff8ae4ac5f4f6ca130a11ad875c9e7ae6d7f9
Author: Koichiro Den <email address hidden>
Date: Thu Oct 29 12:47:34 2020 +0900

    Minor fixes and clean-ups

    * eliminate redundant __import__
    * fix typo
    * fix python3 issue
    * FT: set total sizes to its real full size

    Related-Bug: #1886213
    Change-Id: I940fdc289701023a3f37abb01c3f37dd312f2f7f

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/760280
Committed: https://git.openstack.org/cgit/openstack/tacker/commit/?id=efbafb9ea733ea21f2b26471f29867d0636a684b
Submitter: Zuul
Branch: master

commit efbafb9ea733ea21f2b26471f29867d0636a684b
Author: Koichiro Den <email address hidden>
Date: Thu Oct 29 12:48:08 2020 +0900

    Introduce FT keystone client initialization retry

    I noticed that keystone client initialization is likely to fail due to a
    temporary InternalServerError, leading to nondeterministic FT failures.
    Let's make it tolerable against them.

    Change-Id: I788645d7d489504b4a1c77da010fe0221bf746b7
    Related-Bug: #1886213

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/760281
Committed: https://git.openstack.org/cgit/openstack/tacker/commit/?id=3798c9f92558116a1821cc0cd4151aebb127e3cd
Submitter: Zuul
Branch: master

commit 3798c9f92558116a1821cc0cd4151aebb127e3cd
Author: Koichiro Den <email address hidden>
Date: Thu Oct 29 12:48:28 2020 +0900

    Fix and avoid an erroneous heat API call

    When the vnf maintenance plugin's implementation of
    project_instance_pre() is called, vnf instance id is likely to be empty
    because, for instance, the stack creation itself has not been done for
    some reason.

    This resolves a FT failue: vnfm.test_vnf_placement_policy.VnfTestCreate
    .test_vnf_with_placement_policy_invalid. _get_instances_with_openstack()
    is called on the cleanup path (delete_vnfd), though the vnf instance id
    is not present because create_vnf must have failed due to the invalid
    placement policy. And you must have seen the following error message in
    the tacker-server log: "heatclient.exc.HTTPNotFound: ERROR:
    The Stack (None) could not be found.", which masked the expected
    tacker.extensions.vnfm.HeatClientException: "["invalid"] is not an
    allowed value ["anti-affinity", "affinity", "soft-anti-affinity",
    "soft-affinity"].

    Related-Bug: #1886213
    Change-Id: I8bd2f7e021476470df6131a7ac8abd402e6e1bd1

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/760274
Committed: https://git.openstack.org/cgit/openstack/tacker/commit/?id=32b51d7a9665f61056a33538fa57a7d3c0fd5834
Submitter: Zuul
Branch: master

commit 32b51d7a9665f61056a33538fa57a7d3c0fd5834
Author: Koichiro Den <email address hidden>
Date: Thu Oct 29 12:49:35 2020 +0900

    Always hold VnfDf InstantiationLevels type in JSON even if it's empty

    Before Change 747678, it had been so and it was "null" rather than "{}".
    747678 did not have any intention such that as per SOL011 v3.4.1 7.1.8
    we shall make it mandatory, even though it's quite reasonable to think
    of it as optional in accordance with SOL001 v2.7.1(~v3.4.1) 6.11.2 and
    TOSCA Simple Profile in YAML v1.2(~v1.3). It was just a bug so let's
    partially revert it.

    Related-Bug: #1886213
    Change-Id: I69ffa26227b49b4ad59b71f5b54fa61a0f0b5107

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/760279
Committed: https://git.openstack.org/cgit/openstack/tacker/commit/?id=594e214b35619a5b93760902aa2d737bf0865cc3
Submitter: Zuul
Branch: master

commit 594e214b35619a5b93760902aa2d737bf0865cc3
Author: Koichiro Den <email address hidden>
Date: Thu Oct 29 12:48:14 2020 +0900

    Allow wait_until_vnf_* to withstand temporary backend failures

    I noticed that sometimes backend mysqld seems to randomly restart on FT
    environments, leading to nondeterministic FT test failures.
    Let's make it tolerable against temporary 5XX errors on show_vnf GET
    requests within the timeout period, so as to avoid nondeterministic
    and worthless FT failure noises.

    Plus, as a side improvement, let's make use of tackerclient's "retries"
    feature.

    Related-Bug: #1886213
    Change-Id: Ib96034324036524784a8b6014dfc2ed57e3508bc

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/760282
Committed: https://git.openstack.org/cgit/openstack/tacker/commit/?id=7e3f2d794db03b46d71be663d948906914d6bf56
Submitter: Zuul
Branch: master

commit 7e3f2d794db03b46d71be663d948906914d6bf56
Author: Koichiro Den <email address hidden>
Date: Thu Oct 29 12:49:10 2020 +0900

    Fix test_vnf_alarm_scale_with_instance_reservation

    * Fix empty mgmt_ip_address handling.
      ast.literal_eval('{}') does not raise an exception.
    * Fix not-to-be-cleaned-up blazar hosts. No need to test
      affinity=True case so we just do with one host.
    * Fix req_amount

    Related-Bug: #1886213
    Change-Id: Ief0c4388f80b179c098231081644d97a56bd1379

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/760276
Committed: https://git.openstack.org/cgit/openstack/tacker/commit/?id=d886ac7fec3d9cf6e0cefc5d2fa89a733a5255ae
Submitter: Zuul
Branch: master

commit d886ac7fec3d9cf6e0cefc5d2fa89a733a5255ae
Author: Koichiro Den <email address hidden>
Date: Thu Oct 29 12:49:21 2020 +0900

    Fix missing vnfc resource updating on vnf respawn via SOL003-style Heal API call

    Note: "SOL003-style" means a Heal API call without passing vnfcInstanceId
    as the comment in FT source code states.

    Related-Bug: #1886213
    Change-Id: Ia6fe6b3179703f23266b71dd6595b75d1b801191