Kubevirt: cdi pods are missing toleration labels

[Gleb Aronsky]

Brief Description
-----------------
CDI pods are stuck in pending state on duplex and standard environments. This is due to the tolerations labels being missing for cdi pods apiserver, uploadproxy, and deployment.

Severity
--------
Minor

Steps to Reproduce
------------------
Install on a DX environment. Pods remain in pending state.

cdi cdi-apiserver-7d47cd8d8d-b4j5f 0/1 Pending 0 97m
cdi cdi-deployment-56f769f8cd-glzfx 0/1 Pending 0 97m
cdi cdi-operator-66dc4988b8-9f9kh 1/1 Running 0 97m
cdi cdi-operator-66dc4988b8-fq6lf 1/1 Running 0 97m
cdi cdi-uploadproxy-57d6f66c88-f7cbp 0/1 Pending 0 97m

Expected Behavior
------------------
All CDI pods should start.

Actual Behavior
----------------
CDI pods remain in pending state.

Reproducibility
---------------
100% on DX.

System Configuration
--------------------
Multi-node, AIO-DX, and standard.

Branch/Pull Time/Commit
-----------------------

Last Pass
---------
Did this test scenario pass previously? If so, please indicate the load/pull time info of the last pass.
Use this section to also indicate if this is a new test scenario.

Timestamp/Logs
--------------
Attach the logs for debugging (use attachments in Launchpad or for large collect files use: https://files.starlingx.kube.cengn.ca/)
Provide a snippet of logs here and the timestamp when issue was seen.
Please indicate the unique identifier in the logs to highlight the problem

Test Activity
-------------
Testing

Workaround
----------

Add the following tolerations to pending cdi pods
" - effect: NoSchedule
    key: node-role.kubernetes.io/control-plane
    operator: Exists
"

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/app-kubevirt/+/913596

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/app-kubevirt/+/913596
Committed: https://opendev.org/starlingx/app-kubevirt/commit/9e259393815e35031eeafd1203436d6c1c139c56
Submitter: "Zuul (22348)"
Branch: master

commit 9e259393815e35031eeafd1203436d6c1c139c56
Author: Aronsky, Gleb <email address hidden>
Date: Mon Mar 18 13:34:17 2024 -0700

    Add tolerations to CDI pods

    Add missing tolerations to CDI pods.
    CDI pods missing these tolerations may
    remain in a 'pending' state.

    PASS:
      Install kubevirt-app and verify toleration
      labels on all cdi pods.

    Closes-Bug: 2058283

    Change-Id: Ie4a1fb855e7d35b9dcef1eeedfae36cd623f2756
    Signed-off-by: Aronsky, Gleb <email address hidden>