[stable/victoria] openstacksdk-functional-devstack is broken because openstacksdk is not pinned for stable/victoria

[Alexey Stupnikov]

CI is broken for stable/victoria because of openstacksdk-functional-devstack. When it comes to low-level details, all jobs fail with the same error:

2023-02-08 13:08:55.011314 | controller | + lib/neutron_plugins/services/l3:create_neutron_initial_network:177 : SUBNETPOOL_V4_ID='openstack: '\''subnet pool create shared-default-subnetpool-v4 --default-prefix-length 26 --pool-prefix 10.1.0.0/20 --share --default -f value -c id'\'' is not an openstack command. See '\''openstack --help'\''.
2023-02-08 13:08:55.011383 | controller | Did you mean one of these?
2023-02-08 13:08:55.011425 | controller | secret container create
2023-02-08 13:08:55.011495 | controller | secret container delete
2023-02-08 13:08:55.011546 | controller | secret container get
2023-02-08 13:08:55.011616 | controller | secret container list
2023-02-08 13:08:55.011667 | controller | secret delete
2023-02-08 13:08:55.011737 | controller | secret get
2023-02-08 13:08:55.011788 | controller | secret list
2023-02-08 13:08:55.011857 | controller | secret order create
2023-02-08 13:08:55.011907 | controller | secret order delete
2023-02-08 13:08:55.011976 | controller | secret order get
2023-02-08 13:08:55.012026 | controller | secret order list
2023-02-08 13:08:55.012096 | controller | secret store
2023-02-08 13:08:55.012146 | controller | secret update'
2023-02-08 13:08:55.014534 | controller | + lib/neutron_plugins/services/l3:create_neutron_initial_network:1 : exit_trap
2023-02-08 13:08:55.017274 | controller | + ./stack.sh:exit_trap:496 : local r=2
2023-02-08 13:08:55.020602 | controller | ++ ./stack.sh:exit_trap:497 : jobs -p
2023-02-08 13:08:55.023354 | controller | + ./stack.sh:exit_trap:497 : jobs=
2023-02-08 13:08:55.025998 | controller | + ./stack.sh:exit_trap:500 : [[ -n '' ]]
2023-02-08 13:08:55.028648 | controller | + ./stack.sh:exit_trap:506 : '[' -f /tmp/tmp.vrT0tUaqpZ ']'
2023-02-08 13:08:55.030992 | controller | + ./stack.sh:exit_trap:507 : rm /tmp/tmp.vrT0tUaqpZ
2023-02-08 13:08:55.035321 | controller | + ./stack.sh:exit_trap:511 : kill_spinner
2023-02-08 13:08:55.037749 | controller | + ./stack.sh:kill_spinner:406 : '[' '!' -z '' ']'
2023-02-08 13:08:55.040291 | controller | + ./stack.sh:exit_trap:513 : [[ 2 -ne 0 ]]
2023-02-08 13:08:55.042754 | controller | + ./stack.sh:exit_trap:514 : echo 'Error on exit'
2023-02-08 13:08:55.042828 | controller | Error on exit
2023-02-08 13:08:55.045491 | controller | + ./stack.sh:exit_trap:516 : type -p generate-subunit
2023-02-08 13:08:55.047939 | controller | + ./stack.sh:exit_trap:517 : generate-subunit 1675860497 1238 fail
2023-02-08 13:08:55.552728 | controller | + ./stack.sh:exit_trap:519 : [[ -z /opt/stack/logs ]]
2023-02-08 13:08:55.555477 | controller | + ./stack.sh:exit_trap:522 : /usr/bin/python3.8 /opt/stack/devstack/tools/worlddump.py -d /opt/stack/logs
2023-02-08 13:08:55.924772 | controller | ebtables v1.8.4 (nf_tables): table `broute' is incompatible, use 'nft' tool.
2023-02-08 13:08:55.924832 | controller |
2023-02-08 13:08:56.482691 | controller | + ./stack.sh:exit_trap:531 : exit 2
2023-02-08 13:08:56.488620 | controller | *** FINISHED ***
2023-02-08 13:08:57.738080 | controller | ERROR
2023-02-08 13:08:57.738349 | controller | {
2023-02-08 13:08:57.738440 | controller | "delta": "0:20:39.318179",
2023-02-08 13:08:57.738521 | controller | "end": "2023-02-08 13:08:56.508702",
2023-02-08 13:08:57.738613 | controller | "msg": "non-zero return code",
2023-02-08 13:08:57.738693 | controller | "rc": 2,
2023-02-08 13:08:57.738770 | controller | "start": "2023-02-08 12:48:17.190523"
2023-02-08 13:08:57.738849 | controller | }

When it comes to lower level details, then it looks like this problem is caused by lack of munch lib:

2023-02-08 13:08:54.308885 | controller | Could not load 'subnet_pool_create': No module named 'munch'

It looks like munch was before because it was in openstacksdk requirements (example: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_b90/866086/1/check/openstacksdk-functional-devstack/b901362/job-output.txt), but in latest master branch of openstacksdk it was dropped from requirements list.

From devstack code it looks like stable/victoria branch of openstacksdk should be used (TARGET_BRANCH is set to stable/victoria), but in reality it looks like master is used (from https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_50d/871699/3/check/openstacksdk-functional-devstack/50d5a17/job-output.txt):

2023-02-08 12:38:37.104091 | controller | HEAD is now at 3876f3f6 Update README to indicate COE resource/proxy support
2023-02-08 12:38:37.104264 | controller | Already on 'master'
2023-02-08 12:38:37.104349 | controller | opendev.org/openstack/openstacksdk checked out to:
2023-02-08 12:38:37.104429 | controller | 3876f3f6d9d45f22de7429e14744e66439d90fdf Update README to indicate COE resource/proxy support
2023-02-08 12:38:37.104525 | controller | ok: Item: Runtime: 0:00:00.219312

Sorry if I am missing something, but it looks like a problem in devstack logic.

[Elod Illes (elod-illes)]

Hi,

this problem is due to victoria is in Extended Maintenance and hence openstacksdk cannot cut a new release (we don't do releases for series in Extended Maintenance) to solve the issue. The workaround to remove the gate job was added already in Wallaby (also in Extended Maintenance), but the victoria patch hasn't merged yet, but it is on the gate:

https://review.opendev.org/c/openstack/tempest/+/872964

[Dr. Jens Harbott (j-harbott)]

I think there is a different issue here than in the tempest patch you mention. We can still disable the job instead of fixing, feel free tu submit a patch for that.

[yatin (yatinkarel)]

Patches to fix are https://review.opendev.org/q/Ic1c9cb2e04259aaf78342b4bb2161235666abb49

[Martin Kopec (mkopec)]

as these are merged, can we consider this done?
https://review.opendev.org/q/Ic1c9cb2e04259aaf78342b4bb2161235666abb49

[Martin Kopec (mkopec)]

The patches mentioned in the previous comment fixed the issue, the job has been executed successfully lately:
https://zuul.openstack.org/builds?job_name=openstacksdk-functional-devstack&branch=stable%2Fvictoria&skip=0
Closing as Fix Released.