Deleting system network(s) and/or system address-pool(s) should be rejected after initial controller-0 unlock

[Chris Winnicki]

Title
-----
Deleting system network(s) and/or system address-pool(s) should be rejected after initial controller-0 unlock

Brief Description
-----------------
Attempting to delete system network(s) and/or system address-pool(s) should be rejected after initial controller-0 unlock

Severity
--------
Major

Steps to Reproduce
------------------
Install All-in-one Duplex or Standard system

* For example, attmpt to delete cluster-host network

[wrsroot@controller-1 ~(keystone_admin)]$ system network-list
+----+--------------------------------------+-----------------+-----------------+---------+--------------------------------------+
| id | uuid | name | type | dynamic | pool_uuid |
+----+--------------------------------------+-----------------+-----------------+---------+--------------------------------------+
| 6 | 6243710a-6c9b-4449-ad22-75165721f28e | cluster-pod | cluster-pod | False | 184022e8-a44c-4702-957e-1c6ae79876e3 |
| 1 | 8965c245-e23d-4b18-84ce-26215b24596a | mgmt | mgmt | True | 338c7c62-1af0-4d8a-a6c0-9be0d775a0f6 |
| 5 | a31ad981-033e-4a59-b242-da1665b0490d | cluster-host | cluster-host | True | 2b10b8c0-307a-4c49-9718-e7680b158667 |
| 2 | ce31239b-6917-4d10-8684-236a48923669 | pxeboot | pxeboot | True | 6e47a402-64a5-40d4-8f33-3c6f13c8b79c |
| 3 | d132e665-7d71-40de-8b6f-8dcc1cd27102 | oam | oam | False | 6701396d-5814-4b0c-bb30-a4bb62a0e986 |
| 4 | dd702da4-51b7-4259-9882-b33fe62f8c9f | multicast | multicast | False | 190c9330-1279-466e-9037-f10997888987 |
| 7 | ec720352-bc83-483e-8c85-a0ca68b7087a | cluster-service | cluster-service | False | f3f19fe4-e795-4996-a281-fa283627ad02 |
+----+--------------------------------------+-----------------+-----------------+---------+--------------------------------------+
[wrsroot@controller-1 ~(keystone_admin)]$
[wrsroot@controller-1 ~(keystone_admin)]$ system network-delete a31ad981-033e-4a59-b242-da1665b0490d
Deleted Network: a31ad981-033e-4a59-b242-da1665b0490d

[wrsroot@controller-1 ~(keystone_admin)]$ system network-list
+----+--------------------------------------+-----------------+-----------------+---------+--------------------------------------+
| id | uuid | name | type | dynamic | pool_uuid |
+----+--------------------------------------+-----------------+-----------------+---------+--------------------------------------+
| 6 | 6243710a-6c9b-4449-ad22-75165721f28e | cluster-pod | cluster-pod | False | 184022e8-a44c-4702-957e-1c6ae79876e3 |
| 1 | 8965c245-e23d-4b18-84ce-26215b24596a | mgmt | mgmt | True | 338c7c62-1af0-4d8a-a6c0-9be0d775a0f6 |
| 2 | ce31239b-6917-4d10-8684-236a48923669 | pxeboot | pxeboot | True | 6e47a402-64a5-40d4-8f33-3c6f13c8b79c |
| 3 | d132e665-7d71-40de-8b6f-8dcc1cd27102 | oam | oam | False | 6701396d-5814-4b0c-bb30-a4bb62a0e986 |
| 4 | dd702da4-51b7-4259-9882-b33fe62f8c9f | multicast | multicast | False | 190c9330-1279-466e-9037-f10997888987 |
| 7 | ec720352-bc83-483e-8c85-a0ca68b7087a | cluster-service | cluster-service | False | f3f19fe4-e795-4996-a281-fa283627ad02 |
+----+--------------------------------------+-----------------+-----------------+---------+--------------------------------------+

Expected Behavior
------------------
The above action should be rejected

Actual Behavior
----------------
Deletion of system network is successful

System Configuration
--------------------
System mode: All-in-one Duplex, Standard

Reproducibility
---------------
100%

Branch/Pull Time/Commit
-----------------------
### StarlingX
### Built from master
###

OS="centos"
SW_VERSION="19.01"
BUILD_TARGET="Host Installer"
BUILD_TYPE="Formal"
BUILD_ID="20190305T060000Z"

JOB="STX_build_master_master"
<email address hidden>"
BUILD_NUMBER="6"
BUILD_HOST="starlingx_mirror"
BUILD_DATE="2019-03-05 06:00:00 +0000"

Timestamp/Logs
--------------
Mar 14 11:39

Last time install passed
------------------------
n/a

[Ghada Khalil (gkhalil)]

Marking as release gating; related to the cluster networking feature.

[Ghada Khalil (gkhalil)]

Please include Teresa Ho on the code inspection. Any questions during investigation can also be directed to her.

[Le, Huifeng (hle2)]

Chris, for one question: is this new issue or regression issue?

[Ghada Khalil (gkhalil)]

This is related to SB: https://storyboard.openstack.org/#!/story/2004273

[Le, Huifeng (hle2)]

@Teresa, @Ghada,

By looking through the code, I think this issue is caused by missing operation check for system network-delete.

api/controllers/v1/network.py
@cutils.synchronized(LOCK_NAME)
    @wsme_pecan.wsexpose(None, types.uuid, status_code=204)
    def delete(self, network_uuid):
        """Delete a network."""
        pecan.request.dbapi.network_destroy(network_uuid)

So the questions are:
(1) Add validation in API server (before call dbapi.network_destroy) or shell ???
(2) What the logic for validation?
    e.g. the cluster network can only be deleted if all hosts (include controllers and workers) are locked? any other requirements to be checked?
(3) What the validation requirement? e.g. need check for infra_network, mgmt network and cluster-host network or only cluster network is enough?

Since this feature is related with the operation pre-condition check, Teresa should have better knowledge than us, Maybe Teresa is more suitable to fix this issue?

[Teresa Ho (teresaho)]

(1) The validation should be done in the API server, not the shell.
(2)&(3) The deletion of any of the networks should be blocked after the first controller is unlocked.

[Kailun Qin (kailun.qin)]

@Teresa, @Ghada,
Do we have a fix already available to this issue? If so, would you please kindly add the review link? Thanks.

[Ghada Khalil (gkhalil)]

@Kailun, There is no fix yet. Huifeng is assigned this bug.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/657557
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=bc39f1ea3c2f2c7e86c0b1c52c98089e3b67a403
Submitter: Zuul
Branch: master

commit bc39f1ea3c2f2c7e86c0b1c52c98089e3b67a403
Author: yizhou xu <email address hidden>
Date: Tue May 7 17:10:31 2019 +0800

    Rejected deleting network & address-pool after unlock

    This commit added a semantic check to block deleting network
    if initial controller-0 unlock.

    Change-Id: I9c46df57bb2e7ff8987bf5aa2f50706bf07b3e65
    Story: 2004273
    Closes-Bug: #1820115

[Chris Winnicki (chriswinnicki)]

Retested
Verdict: Passed

Build used:
### StarlingX
### Built from master
###

OS="centos"
SW_VERSION="19.01"
BUILD_TARGET="Host Installer"
BUILD_TYPE="Formal"
BUILD_ID="20190617T233000Z"

JOB="STX_build_master_master"
<email address hidden>"
BUILD_NUMBER="150"
BUILD_HOST="starlingx_mirror"
BUILD_DATE="2019-06-17 23:30:00 +0000"