sriov test suite fails to verify error types (false negatives)

[Thomas Gao]

Brief Description
-----------------
Several tests in sysinv/tests/api/test_interface.py fail to verify the error raised when `expect_errors=True` is set.

For example, for test_invalid_sriov_numvfs, the expected error is "Value for number of SR-IOV VFs must be > 0," while the actual error is "Unable to provision pci-sriov interface without configured mgmt interface." Since the only assertion in place is `assertEqual(http_client.BAD_REQUEST, response.status_int`, there is no mechanism to verify whether the appropriate type of error has been raised, the test passed as a false negative.

Those false-negative tests include:

- test_invalid_sriov_numvfs
- test_invalid_sriov_totalvfs_zero
- test_invalid_sriov_exceeded_totalvfs
- test_invalid_driver_for_sriov

Severity
--------
Minor: System/Feature is usable with minor issue

Steps to Reproduce
------------------
In the development repo:

> cd cgcs-root/stx/config/sysinv/sysinv/sysinv/sysinv/tests/api
> tox -e py27 test_invalid_sriov_numvfs

Expected Behavior
------------------
Test should fail when the function doesn't start with `self._create_ethernet('mgmt', constants.NETWORK_TYPE_MGMT)`

Actual Behavior
----------------
Test succeeds when the function doesn't start with `self._create_ethernet('mgmt', constants.NETWORK_TYPE_MGMT)`

Reproducibility
---------------
100% Reproducible

System Configuration
--------------------
Any machine that's set up for development.

Branch/Pull Time/Commit
-----------------------
master; 1/16/2020; 737b98c4562c6adf269be9f367d0fd21c09da65f

Last Pass
---------
new test scenario

Test Activity
-------------
Feature Testing

[Ghada Khalil (gkhalil)]

stx.4.0 / medium priority given the focus on increasing unit test coverage

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/702968
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=c22e5787c8308b61bb2d5feccd12e35110dcb558
Submitter: Zuul
Branch: master

commit c22e5787c8308b61bb2d5feccd12e35110dcb558
Author: Thomas Gao <email address hidden>
Date: Thu Jan 16 14:38:32 2020 -0500

    Verified types of errors for sriov test suites.

    Verification is done by checking whether certain messages
    are shown in the response.json['error_message']. The original
    error message "Unable to provision pci-sriov interface
    without configured mgmt interface" is promptly caught, causing
    the test to fail, as expected.

    Since the intended error message is supposed to be something
    that's not "Unable to provision pci-sriov interface without
    configured mgmt interface," I added mgmt interface
    configuration so the test won't fail.

    Closes-Bug: 1860030

    Change-Id: I6a2f11d6cf5b37166fc44b722deab47e696f9fd6
    Signed-off-by: Thomas Gao <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/705837

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/705837
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=8ac6ec70cb8a787a274fd7227eb34d2b7bcd5f5b
Submitter: Zuul
Branch: f/centos8

commit 7995dd436954b92f1c4e3f760a7609af670c84c8
Author: Jessica Castelino <email address hidden>
Date: Mon Feb 3 12:07:26 2020 -0500

    Unit test cases for helm charts

    Test cases added for API endpoints used by:
     1. helm-override-delete
     2. helm-override-show
     3. helm-override-list
     4. helm-override-update
     5. helm-chart-attribute-modify

    Story: 2007082
    Task: 38012
    Change-Id: I86763496bb41084c006f2486702c3b15bde039d2
    Signed-off-by: Jessica Castelino <email address hidden>

commit 7e2fda010299f7305b630d6db97bbe1e169a38b1
Author: Angie Wang <email address hidden>
Date: Wed Jan 29 21:18:18 2020 -0500

    Finish kubernetes networking upgrade support

    The commit completes the RPC kube_upgrade_networking
    in sysinv-conductor to run ansible playbook
    upgrade-k8s-networking.yml to upgrade networking pods
    and also updates the networking upgrade function called
    as part of sysinv-conductor startup to provide a current
    kubernetes version when running the upgrade playbook.

    The second control plane upgrade can only be performed
    after the networking upgrade is done, fix the semantic
    check in sysinv api.

    Change-Id: I8dcf5a2baedfaefb0a7ca037eb47bf7cacd686f8
    Story: 2006781
    Task: 37584
    Depends-On: https://review.opendev.org/#/c/705310/
    Signed-off-by: Angie Wang <email address hidden>

commit 52c37a35d2cd62fa1cc1933765c76c1ba8616864
Author: Jerry Sun <email address hidden>
Date: Fri Jan 31 16:10:25 2020 -0500

    Add Unit Tests for Dex Sysinv Changes

    Add unit tests for the dex helm chart changes under the same story
    and task

    Story: 2006711
    Task: 37857

    Depends-On: https://review.opendev.org/#/c/705297/

    Change-Id: I3a0e1c490e56188adfbd614fd6ebb21bfdddf49e
    Signed-off-by: Jerry Sun <email address hidden>

commit 144587a6ac9fc48b9249be99abadd35dfa49e7a7
Author: Teresa Ho <email address hidden>
Date: Fri Jan 31 15:35:04 2020 -0500

    Tox tests for OIDC client helm overrides

    Added some tox tests for OIDC client helm overrides.

    Story: 2006711
    Task: 38481

    Change-Id: If4aeaf0010c7076d1d83bacd00d6fd0122d4ffad
    Signed-off-by: Teresa Ho <email address hidden>

commit 763ddeadd4e83af6cebf752d693ee3e7d3b005b1
Author: Thomas Gao <email address hidden>
Date: Wed Jan 29 16:30:40 2020 -0500

    Fixed errors in address deletion

    Allowed address deletion despite missing associated interface or host.

    Enabled relevant unit test.

    Closes-Bug: 1860186

    Change-Id: Ie6e6358aa75091e92914a8b581b4d5203a596f56
    Signed-off-by: Thomas Gao <email address hidden>

commit 61463608169e75601b8a4f9db7c98190788d6f6a
Author: Thomas Gao <email address hidden>
Date: Tue Jan 28 15:32:58 2020 -0500

    Fixed broken sysinv address get-all api call

    Removed unexpected keyword argument that caused the error....