[Master] tripleoclient does not play well with cliff-3.4.0
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
tripleo |
Fix Released
|
High
|
yatin |
Bug Description
Jobs failing while running openstack baremetal node list with cliff-3.4.0:-
2020-08-05 05:51:21.643746 | primary | TASK [overcloud-
2020-08-05 05:51:21.643875 | primary | Wednesday 05 August 2020 05:51:21 +0000 (0:00:00.043) 0:30:18.884 ******
2020-08-05 05:51:28.122927 | primary | FAILED - RETRYING: Wait until nodes will be manageable (10 retries left).
2020-08-05 05:52:01.247231 | primary | FAILED - RETRYING: Wait until nodes will be manageable (9 retries left).
2020-08-05 05:52:34.901402 | primary | FAILED - RETRYING: Wait until nodes will be manageable (8 retries left).
2020-08-05 05:53:08.168990 | primary | FAILED - RETRYING: Wait until nodes will be manageable (7 retries left).
2020-08-05 05:53:41.332388 | primary | FAILED - RETRYING: Wait until nodes will be manageable (6 retries left).
2020-08-05 05:54:14.187384 | primary | FAILED - RETRYING: Wait until nodes will be manageable (5 retries left).
2020-08-05 05:54:47.252761 | primary | FAILED - RETRYING: Wait until nodes will be manageable (4 retries left).
2020-08-05 05:55:20.869404 | primary | FAILED - RETRYING: Wait until nodes will be manageable (3 retries left).
2020-08-05 05:55:54.463138 | primary | FAILED - RETRYING: Wait until nodes will be manageable (2 retries left).
2020-08-05 05:56:28.003778 | primary | FAILED - RETRYING: Wait until nodes will be manageable (1 retries left).
2020-08-05 05:57:01.057502 | primary | fatal: [undercloud]: FAILED! => {
2020-08-05 05:57:01.057662 | primary | "attempts": 10,
2020-08-05 05:57:01.057682 | primary | "changed": false,
2020-08-05 05:57:01.057694 | primary | "cmd": "set -o pipefail && openstack --os-cloud undercloud baremetal node list -f value -c \"Provisioning State\" | grep -v -e manageable -e available",
2020-08-05 05:57:01.057706 | primary | "delta": "0:00:02.497370",
2020-08-05 05:57:01.057717 | primary | "end": "2020-08-05 05:57:01.024299",
2020-08-05 05:57:01.057729 | primary | "failed_
2020-08-05 05:57:01.057740 | primary | "rc": 0,
2020-08-05 05:57:01.057751 | primary | "start": "2020-08-05 05:56:58.526929"
2020-08-05 05:57:01.057762 | primary | }
2020-08-05 05:57:01.057793 | primary |
2020-08-05 05:57:01.057807 | primary | STDOUT:
2020-08-05 05:57:01.057819 | primary |
2020-08-05 05:57:01.057830 | primary | 2020-08-05 05:57:00.306 119070 INFO osc_lib.shell [-] command: baremetal node list -> ironicclient.
2020-08-05 05:57:01.057859 | primary | 2020-08-05 05:57:00.308 119070 INFO osc_lib.
2020-08-05 05:57:01.057871 | primary | 2020-08-05 05:57:00.737 119070 INFO osc_lib.shell [-] END return value: 0[00m
2020-08-05 05:57:01.057882 | primary |
2020-08-05 05:57:01.057892 | primary |
2020-08-05 05:57:01.057903 | primary | STDERR:
2020-08-05 05:57:01.057914 | primary |
2020-08-05 05:57:01.057925 | primary | /usr/lib64/
2020-08-05 05:57:01.057935 | primary | return f(*args, **kwds)
Issue is happening happening as return code is 0 now and stdout is not null with new cliff as seen in above snippet.
Conditions on failing task:-
until: result.stdout == ''
failed_when: result.rc == 0
Detected while bumping cliff following uppper-constraints with
Logs:- https:/
https:/
Small reproducer(On a undercloud node):-
virtualenv clifftest
source clifftest/
pip install python-
openstack --os-cloud undercloud baremetal node list -f value -c "Provisioning State"
# Install python-
pip install python-
# Now output changes for command, some INFO messages are dumped:-
openstack --os-cloud undercloud baremetal node list -f value -c "Provisioning State"
2020-08-05 06:43:10.788 776951 INFO osc_lib.shell [-] command: baremetal node list -> ironicclient.
2020-08-05 06:43:10.790 776951 INFO osc_lib.
2020-08-05 06:43:11.264 776951 INFO osc_lib.shell [-] END return value: 0
# Issue not specific to baremetal, all commands are impacted like
openstack --os-cloud undercloud network list
2020-08-05 06:45:28.404 780181 INFO osc_lib.shell [-] command: network list -> openstackclient
2020-08-05 06:45:28.405 780181 INFO osc_lib.
/home/zuul/
len(inspect.
+------
| ID | Name | Subnets |
+------
| 5cfe57fb-
+------
2020-08-05 06:45:29.002 780181 INFO osc_lib.shell [-] END return value: 0
# If tripleoclient is removed from venv or cliff is downgraded to 3.3.0, issue no longer happens, so something is going wrong with tripleoclient. There is only 1 commit in 3.4.0 which switches to importlib-
Changed in tripleo: | |
importance: | Undecided → High |
status: | New → Triaged |
milestone: | none → victoria-3 |
tags: | added: alert |
Changed in tripleo: | |
assignee: | nobody → chandan kumar (chkumar246) |
Changed in tripleo: | |
assignee: | chandan kumar (chkumar246) → yatin (yatinkarel) |
status: | Triaged → In Progress |
Related fix proposed to branch: master /review. opendev. org/752931
Review: https:/