openstack tripleo validator run --validation service-status is broken

[Michele Baldessari]

From yesterdays's master:

(undercloud) [stack@undercloud-0 ~]$ openstack tripleo validator run --validation service-status
Running Validations without Overcloud settings.
WARNING:tripleo_common.inventory:Stack not found: None. No data found in neither neutron or heat. Only the undercloud will be added to the inventory.
Error while creating Ansible artifacts log file.Please check the access rights for /var/log/validations/artifacts//deb532d4-535e-4fc2-be15-134731dc760b_service-status.yaml_2021-04-23T07:31:24.665677Z
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/validations_libs/utils.py", line 51, in create_artifacts_dir
    os.makedirs(log_dir)
  File "/usr/lib64/python3.6/os.py", line 210, in makedirs
    makedirs(head, mode, exist_ok)
  File "/usr/lib64/python3.6/os.py", line 220, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/var/log/validations/artifacts'
'NoneType' object is not iterable

(undercloud) [stack@undercloud-0 ~]$ openstack tripleo validator run --static-inventory ~/config-download/overcloud/tripleo-ansible-inventory.yaml --validation service-status
Running Validations without Overcloud settings.
Error while creating Ansible artifacts log file.Please check the access rights for /var/log/validations/artifacts//c6ef79e0-2f4a-4c14-932b-834271c7ea7a_service-status.yaml_2021-04-23T07:32:22.852797Z
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/validations_libs/utils.py", line 51, in create_artifacts_dir
    os.makedirs(log_dir)
  File "/usr/lib64/python3.6/os.py", line 210, in makedirs
    makedirs(head, mode, exist_ok)
  File "/usr/lib64/python3.6/os.py", line 220, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/var/log/validations/artifacts'
'NoneType' object is not iterable

[Michele Baldessari (michele)]

(undercloud) [stack@undercloud-0 ~]$ sudo ls -la /var/log/validations
total 8428
drwxr-xr-x. 2 root root 4096 Apr 22 20:57 .
drwxr-xr-x. 25 root root 4096 Apr 22 20:57 ..
-rw-r--r--. 1 root root 8095563 Apr 22 20:54 5254009c-2b07-0867-13fb-000000000006_deploy_steps_playbook_2021-04-22T20:38:17.750278Z.json
-rw-r--r--. 1 root root 85434 Apr 22 20:57 5254009c-2b07-13cf-6690-000000000005_playbook_2021-04-22T20:56:28.567458Z.json
-rw-r--r--. 1 root root 78978 Apr 22 20:57 5254009c-2b07-3987-ac02-000000000006_cli-container-image-prepare_2021-04-22T20:56:16.120298Z.json
-rw-r--r--. 1 root root 85446 Apr 22 20:57 5254009c-2b07-44b4-d974-000000000005_playbook_2021-04-22T20:56:28.588026Z.json
-rw-r--r--. 1 root root 98213 Apr 22 20:56 5254009c-2b07-6024-9f3d-000000000006_cli-container-image-prepare_2021-04-22T20:54:15.146741Z.json
-rw-r--r--. 1 root root 85418 Apr 22 20:57 5254009c-2b07-9201-166f-000000000005_playbook_2021-04-22T20:56:28.600166Z.json
-rw-r--r--. 1 root root 85456 Apr 22 20:57 5254009c-2b07-9420-0a69-000000000005_playbook_2021-04-22T20:56:28.607561Z.json

So root-owned by default is the issue here. I chowned it to stack: and it sort of works? In the sense that it only finds the undercloud by default?

(undercloud) [stack@undercloud-0 ~]$ sudo chown -R stack: /var/log/validations/
(undercloud) [stack@undercloud-0 ~]$ openstack tripleo validator run --validation service-status
Running Validations without Overcloud settings.
WARNING:tripleo_common.inventory:Stack not found: None. No data found in neither neutron or heat. Only the undercloud will be added to the inventory.
+--------------------------------------+----------------+--------+------------+----------------+-------------------+-------------+
| UUID | Validations | Status | Host_Group | Status_by_Host | Unreachable_Hosts | Duration |
+--------------------------------------+----------------+--------+------------+----------------+-------------------+-------------+
| 300d821b-e386-4eac-9c5e-054cc929b776 | service-status | PASSED | all | undercloud | | 0:00:04.965 |
+--------------------------------------+----------------+--------+------------+----------------+-------------------+-------------+

Assuming that I need to forcefully pass an inventory file (which ideally would not be needed), I got this:
(undercloud) [stack@undercloud-0 ~]$ openstack tripleo validator run --static-inventory ~/config-download/overcloud/tripleo-ansible-inventory.yaml --validation service-status
Running Validations without Overcloud settings.
+--------------------------------------+----------------+--------+------------+------------------------------------------------------------------------------------------------+-------------------+-------------+
| UUID | Validations | Status | Host_Group | Status_by_Host | Unreachable_Hosts | Duration |
+--------------------------------------+----------------+--------+------------+------------------------------------------------------------------------------------...

[Jiri Podivin (jpodivin)]

Potential fix:
https://review.opendev.org/c/openstack/validations-libs/+/787696

[Michele Baldessari (michele)]

The fix above improves things considerably, thanks!

We should prolly clear up the root cause as well. In the sense that /var/log/validations owned by root comes from the stock rpm:
[stack@undercloud-0 ~]$ ls -la /var/log/validations/
total 4
drwxr-xr-x. 2 root root 6 Apr 14 18:50 .
drwxr-xr-x. 24 root root 4096 Apr 23 11:04 ..

[stack@undercloud-0 ~]$ rpm -qf /var/log/validations/
validations-common-1.1.2-0.20210414184922.3d6e5a1.el8.noarch

[stack@undercloud-0 ~]$ rpm -V validations-common
[stack@undercloud-0 ~]$

So either we loosen the permissions on /var/log/validations somehow or we just move them to the stack user by default or something?

[Jiri Podivin (jpodivin)]

The patch is still waiting for all the jobs to finish. Until then it will not be ready for review.

The original bug seems to have been caused by non-standard/legacy tripleoclient usage.

[Michele Baldessari (michele)]

So ftr now i can deploy a UC without using the --disable-validations in my BGP env. So maybe we can drop that option entirely for the UC installation?