I had a packstack rocky deployment and I cloned tempest from master branch. Tempest cleaned up also networks which it wasn't supposed to.
I had two following networks available after the deployment.
$ openstack network list
+--------------------------------------+---------+--------------------------------------+
| ID | Name | Subnets |
+--------------------------------------+---------+--------------------------------------+
| 47bd1566-e13e-42b5-ad4b-fea122f425e9 | private | 98af2db8-2624-4471-8093-e983832e5dd9 |
| b121d356-2296-4100-9880-1a4410b4fd40 | public | 860fe620-6227-4349-9d8a-5947aa0f303b |
+--------------------------------------+---------+--------------------------------------+
After init-saved-state we can see, that there are no networks mentioned in the saved_state.json file - that's quite suspicious.
$ tempest cleanup --init-saved-state
$ cat saved_state.json
{
"domains": {
"default": "Default"
},
"flavors": {
"1": "m1.tiny",
"2": "m1.small",
"3": "m1.medium",
"4": "m1.large",
"5": "m1.xlarge",
"a83007fa-4b1b-4957-a731-1d82a6a264e0": "m1.micro",
"dbc75455-e92d-4a3d-ace7-064ca8fc6324": "m1.nano"
},
"images": {},
"projects": {
"1efbbf4b524d44f694241e65dac11472": "alt_demo",
"588d28c2d68e4857b72e7bd549141d55": "demo",
"bc0183ea801d46ab869f2b40c15b2269": "admin",
"c90f86114f444c09baa8a126fdf27f8b": "services"
},
"roles": {},
"users": {
"202c643912194efab607ab7b8e1f46da": "admin",
"318fcff041f14bab9041759d54bdeecc": "cinder",
"3b69696b13604b44a47475d383d65704": "nova",
"534c5b16677044c1be2fe07d452d32dd": "aodh",
"5ac1446caa9e4b6aa34d41dc7eb2f5d4": "neutron",
"72cc57a76ea4477e8a33e88ca684d48a": "alt_demo",
"75b2eb04cebf458ea8e489d16b2bb9b5": "demo",
"8afab28d27ff466eaddb03bb423bd0a5": "swift",
"aca4fcee47054713a1e91ee1ba277912": "glance",
"b19acee3e3aa4c578848801f33c885b7": "gnocchi",
"e07fcb5e10b245649d245a66d7451a96": "placement",
"ebcd4caf6e274fb1a1eb395188233fb8": "ceilometer"
}
}
Then I tried cleanup with --dry-run option and the generated file contained nothing to clean - that was good, but it was weird that the networks were not there because they weren't present in the saved_state.json, so does it mean that the `tempest cleanup` is not able to remove networks?
$ tempest cleanup --dry-run
$ cat dry_run.json
{
"_projects_to_clean": {},
"domains": [],
"flavors": [],
"images": [],
"projects": [],
"roles": [],
"users": []
}
Then I did a little experiment and ran smoke tests:
$ tempest run --smoke
After a few seconds (let's say 30 sec) I killed the process, in order to make tempest left some networks and other resources behind.
$ openstack network list
+--------------------------------------+---------------------------------------------------------+--------------------------------------+
| ID | Name | Subnets |
+--------------------------------------+---------------------------------------------------------+--------------------------------------+
| 47bd1566-e13e-42b5-ad4b-fea122f425e9 | private | 98af2db8-2624-4471-8093-e983832e5dd9 |
| 69b82de6-2b24-4f15-b31c-64b304e4deb1 | tempest-ServersTestManualDisk-1308724128-network | e82f8798-b764-4556-9dcc-e0d08b20c02b |
| 8fe59e7d-89cb-4aed-82c4-4be235d38fdd | tempest-AttachInterfacesUnderV243Test-283077215-network | 28735895-9539-4475-b73b-70bd35ebe9c3 |
| b121d356-2296-4100-9880-1a4410b4fd40 | public | 860fe620-6227-4349-9d8a-5947aa0f303b |
| d3e43571-f9c6-48e0-99e6-f43f34c95581 | tempest-ServersTestBootFromVolume-577301809-network | 4e5db97f-b7fb-4fac-8de9-ae771f2d685a |
| edbe9f22-0dfb-44d3-ad43-497bbc8b0883 | tempest-ServerActionsTestJSON-293603464-network | 4cbf930e-dd9d-46f8-b7eb-5f15c4d09671 |
+--------------------------------------+---------------------------------------------------------+--------------------------------------+
I tried again dry cleanup:
$ tempest cleanup --dry-run
and I could see that even my private network was present in the dry_run.json file - but why now? Before it wasn't marked as a leftover resource (even it should have, because it was not in the saved_state.json), but now yes.
And after I did the real cleanup, the network was gone.
$ openstack network list
+--------------------------------------+--------+--------------------------------------+
| ID | Name | Subnets |
+--------------------------------------+--------+--------------------------------------+
| b121d356-2296-4100-9880-1a4410b4fd40 | public | 860fe620-6227-4349-9d8a-5947aa0f303b |
+--------------------------------------+--------+--------------------------------------+
I guess that the public network wasn't deleted, because its ID is also written in my tempest.conf file. But the private network shouldn't be deleted and therefore networks should be also part of the saved_state.json file.
When tempest cleanup queries the cloud for discovering initial state, it iterates only via global services [1][2] and only those services are listed for any initial state, which is then saved to saved_state.json.
Then the cleanup continues by iterating over projects [3], where every found project is cleaned by _clean_project method. Cleaning of every project means iterating over project_services [4][5] and executing their delete/dry_run methods.
Note, that these project_services [5] don't have implemented save_stae method and their list method doesn't contain code which filters saved resources out - that means that if the delete/dry_run method is executed, it will delete all resources found.
Now to the question why my private network was deleted.
When a project is cleaned in the _clean_project method and the loop over project_services [4] is on NetworkSubnetSe
The same happens when the loop [4] is on NetworkService - its list method discovers the private network and it's deleted.
Why?
Because NetworkSubnetSe
When you have a look at the definition of these project_services [8], you'll see that none of them contains saved_state method and list method which would contain code for excluding initial resources. That's very strange because those services have implemented delete method. Is it really intended? I would say that every class/service which deletes resources should have a code for their initial discovery. Otherwise we may end up in a situation in which this bug was created.
[1] https:/
[2] https:/
[3] https:/
[4] https:/
[5] https:/
[6] https:/
[7] https:/
[8] https:/