tripleo validator "nfv-ovsdpdk-zero-packet-loss-check" fails on "Get nova libvirt namespace processes"

[Juan Pablo Martí]

Description:
tripleo validator "nfv-ovsdpdk-zero-packet-loss-check" fails on "Get nova libvirt namespace processes." This is because in the previous step "Get nova_libvirt_launcher Process" a bad process ID is returned due to printf configuration issues. The following command returns two numbers concatenated togehter:

  "cmd": "ps -Leaf | grep nova_libvirt_launcher.sh | grep -v pts | awk '{printf \"%s\", $2}'"

For example, the pgrep in the next step is:

pgrep --ns 21500518982

When the launcher process is really 21500. Suggested changes to main.yaml:

(undercloud) [stack@undercloud-162 ~]$ diff /usr/share/ansible/roles/check_nfv_ovsdpdk_zero_packet_loss/tasks/main.yml changes_to_validations/main.yml
61c61
< ps -Leaf | grep nova_libvirt_launcher.sh | grep -v pts | awk '{printf "%s", $2}'
---
> ps -Leaf | grep nova_libvirt_launcher.sh | grep -v pts -m 1 | awk '{print $2}'

Version-Release number of selected component (if applicable):
16.2

Steps to Reproduce:
openstack tripleo validator run --validation nfv-ovsdpdk-zero-packet-loss-check

Expected results:
Validation run should complete and print errors, if any

Actual results:
Validation run fails with runtime error

Environment:
RHOSP 16.1 (Train)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-validations/+/834538
Committed: https://opendev.org/openstack/tripleo-validations/commit/bf17c453ecb10c964c5f2e6139664a7f5294ac66
Submitter: "Zuul (22348)"
Branch: master

commit bf17c453ecb10c964c5f2e6139664a7f5294ac66
Author: Juan Pablo Martí <email address hidden>
Date: Mon Mar 21 16:07:55 2022 -0300

    Correct the command that gets the PID of nova_libvirt_launcher.sh.

    The problem is due to a bug that makes the command get two numbers concatenated instead of one.

    Closes-Bug: #1970755

    Signed-off-by: Juan Pablo Martí <email address hidden>
    Change-Id: I0295e31bbaba96c7e60a872a89be7e0ddd07bd36

[Juan Pablo Martí (yampilop)]

Please re-open this bug.

I'm getting failure again in RHOSP 16.2.2 since there's no nova_libvirt_launcher.sh process anymore.

[root@overcloud-computeovsdpdk-0 ~]# cat /etc/rhosp-release
Red Hat OpenStack Platform release 16.2.2 (Train)
[root@overcloud-computeovsdpdk-0 ~]# ps aux | grep nova_libvirt_launcher
root 9556 0.0 0.0 12136 1156 pts/0 S+ 17:40 0:00 grep --color=auto nova_libvirt_launcher

In roles/check_nfv_ovsdpdk_zero_packet_loss/tasks/main.yml from the validations:

- name: Get nova_libvirt_launcher Process
  become: true
  shell: |-
    ps -Leaf | grep nova_libvirt_launcher.sh | grep -v pts -m 1 | awk '{print $2}'
  register: nova_libvirt_launcher

- name: Get nova libvirt namespace processes
  become: true
  shell: |-
    pgrep --ns {{ nova_libvirt_launcher.stdout }}
  register: nova_libvirt_proceses

It's trying to get the PID of the process belonging to the same namespace as nova_libvirt_launcher.sh.

As there is no more running process related to nova_libvirt_launcher.sh, I searched the script to see the contents:

[root@overcloud-computeovsdpdk-0 ~]# cat /var/lib/container-config-scripts/nova_libvirt_launcher.sh
#!/bin/bash
set -xe
if [[ -f /usr/lib/systemd/kvm-setup ]]; then
  /usr/lib/systemd/kvm-setup
fi
exec systemd-run --scope --slice=system /usr/sbin/libvirtd

So probably we need to change the task to:

- name: Get nova_libvirt_launcher Process
  become: true
  shell: |-
    ps -Leaf | grep libvirtd | grep -v pts -m 1 | awk '{print $2}'
  register: nova_libvirt_launcher

Is that right? Did I get the logic of how it works?

[Bogdan Dobrelya (bogdando)]

It looks about right, yes. Maybe adjust to grep /usr/sbin/libvirtd.
Also, in Wallaby, there is no more libvirtd in the favor of modular daemons, like virtqemud

[Bogdan Dobrelya (bogdando)]

Also, you could squash two greps into a single grep 'libvirt[d]' -m1

[Bogdan Dobrelya (bogdando)]

rather: ps -Leaf | awk '/\/usr\/sbin\/libvirtg[d]/ {print $2; exit}'

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-validations/+/861770

[Juan Pablo Martí (yampilop)]

Created a review for solving it using your suggestions.

https://review.opendev.org/c/openstack/tripleo-validations/+/861770

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-validations/+/861770
Committed: https://opendev.org/openstack/tripleo-validations/commit/d6cbb646d6bdd84ae3d18c3439e2813dfc51c6d8
Submitter: "Zuul (22348)"
Branch: master

commit d6cbb646d6bdd84ae3d18c3439e2813dfc51c6d8
Author: Juan Pablo Marti <email address hidden>
Date: Tue Oct 18 12:25:53 2022 -0300

    Replace the task "Get nova_libvirt_launcher Process" to "Get libvirtd Process"

    The process nova_libvirt_launcher.sh does not exist anymore. Replaced the search for libvirtd process. Changed the registered variable name for clarity.

    Closes-Bug: #1970755

    Signed-off-by: Juan Pablo Marti <email address hidden>
    Change-Id: I59e7255f5c6d9b11ee0d14f171c15843f4871225

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-validations 16.0.0 release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/tripleo-validations/+/865515

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-validations/+/865515
Committed: https://opendev.org/openstack/tripleo-validations/commit/7092063227adcf55f8dbc24061a43f9d6163460d
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 7092063227adcf55f8dbc24061a43f9d6163460d
Author: Juan Pablo Marti <email address hidden>
Date: Tue Oct 18 12:25:53 2022 -0300

    Replace the task "Get nova_libvirt_launcher Process" to "Get libvirtd Process"

    The process nova_libvirt_launcher.sh does not exist anymore. Replaced the search for libvirtd process. Changed the registered variable name for clarity.

    Closes-Bug: #1970755

    Signed-off-by: Juan Pablo Marti <email address hidden>
    Change-Id: I59e7255f5c6d9b11ee0d14f171c15843f4871225
    (cherry picked from commit d6cbb646d6bdd84ae3d18c3439e2813dfc51c6d8)