nova-next job POST_FAILURE due to nova-manage heal _allocations testing fails

[Balazs Gibizer]

Since Friday (3rd of Sept) the nova-next job fails[1] on master with:

2021-09-05 10:20:23.082260 | controller | + /opt/stack/nova/gate/post_test_hook.sh:main:189 : openstack port unset --binding-profile allocation port-normal-qos

2021-09-05 10:20:25.387981 | controller | ++ /opt/stack/nova/gate/post_test_hook.sh:main:193 : openstack resource provider allocation show a76b03d8-98e6-4063-a5c2-8d6d9128e233 -c resources -f value

2021-09-05 10:20:27.018412 | controller | 'project_id'

2021-09-05 10:20:27.152002 | controller | + /opt/stack/nova/gate/post_test_hook.sh:main:193 : allocations=

2021-09-05 10:20:27.441731 | controller | ERROR

2021-09-05 10:20:27.442055 | controller | {

2021-09-05 10:20:27.442156 | controller | "delta": "0:01:26.706541",

2021-09-05 10:20:27.442248 | controller | "end": "2021-09-05 10:20:27.152606",

2021-09-05 10:20:27.442337 | controller | "msg": "non-zero return code",

2021-09-05 10:20:27.442424 | controller | "rc": 1,

2021-09-05 10:20:27.442510 | controller | "start": "2021-09-05 10:19:00.446065"

2021-09-05 10:20:27.442599 | controller | }

[1] https://zuul.opendev.org/t/openstack/builds?job_name=nova-next&project=openstack%2Fnova&branch=master
[2] https://zuul.opendev.org/t/openstack/build/f5b881e3601f4160a5f82a9b4cdc10ad/log/job-output.txt#62198-62200

[Balazs Gibizer (balazs-gibizer)]

I can reproduce it in a local fresh devstack. The command that fails in the script[1] is calling allocation show but the osc-placement client fails it parse the response properly resulting in a KeyError

REQ: curl -g -i -X GET http://192.168.121.158/placement/allocations/24995dd2-2966-4e3b-975c-c02368b1e086 -H "Accept: application/json" -H "OpenStack-API-Version: placement 1.29" -H "User-Agent: openstacksdk/0.59.0 keystoneauth1/4.3.1 python-requests/2.26.0 CPython/3.8.10" -H "X-Auth-Token: {SHA256}e30defd2510f68e4e8af8225168d91d40c58887815ef96d297189b33166fa3da"
Resetting dropped connection: 192.168.121.158
http://192.168.121.158:80 "GET /placement/allocations/24995dd2-2966-4e3b-975c-c02368b1e086 HTTP/1.1" 200 19
RESP: [200] Cache-Control: no-cache Connection: close Content-Length: 19 Content-Type: application/json Date: Mon, 06 Sep 2021 07:46:53 GMT Last-Modified: Mon, 06 Sep 2021 07:46:53 GMT Server: Apache/2.4.41 (Ubuntu) openstack-api-version: placement 1.29 vary: openstack-api-version x-openstack-request-id: req-1956e37b-7cd5-4cb8-b7de-e07bd7a04187
RESP BODY: {"allocations": {}}
GET call to placement for http://192.168.121.158/placement/allocations/24995dd2-2966-4e3b-975c-c02368b1e086 used request id req-1956e37b-7cd5-4cb8-b7de-e07bd7a04187
'project_id'
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/cliff/app.py", line 407, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/local/lib/python3.8/dist-packages/osc_lib/command/command.py", line 39, in run
    return super(Command, self).run(parsed_args)
  File "/usr/local/lib/python3.8/dist-packages/cliff/display.py", line 115, in run
    column_names, data = self.take_action(parsed_args)
  File "/opt/stack/osc-placement/osc_placement/resources/allocation.py", line 335, in take_action
    props['project_id'] = resp['project_id']
KeyError: 'project_id'
clean_up ShowAllocation: 'project_id'
END return value: 1

Locally reverting https://review.opendev.org/c/openstack/osc-placement/+/804458 fixes the issue

[Balazs Gibizer (balazs-gibizer)]

being fixed in https://review.opendev.org/q/topic:bug/1942740

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/osc-placement 3.1.1 release.