CENGN build failure near Registering signing server access key

[Scott Little]

Brief Description
-----------------
CENGN build failed in the start-containers step
It was attempting to 'Registering signing server access key'
and issued a command containing two container names with a 'CR' between them.

stx-tools pods list:
02:25:32 NAME READY STATUS RESTARTS AGE
02:25:32 debian-master-stx-builder-574f465fc7-9fgnh 1/1 Terminating 1 (10m ago) 5d
02:25:32 debian-master-stx-builder-574f465fc7-pkfqd 1/1 Running 0 15s

We need to filter out non-Running containers when attempting to discover the container name.

Perhaps also wait for terminating containers to fully exit.

Severity
--------
Major

Steps to Reproduce
------------------
N/A

Expected Behavior
------------------
Container start is a success

Actual Behavior
----------------
Container start fails

Reproducibility
---------------
N/A

System Configuration
--------------------
N/A

Branch/Pull Time/Commit
-----------------------
Apr 11

Last Pass
---------
Apr 3

Timestamp/Logs
--------------
+ stx control status
02:25:31 STX-k8s - INFO: helm list:
02:25:31
02:25:31 NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
02:25:31 debian-master default 1 2023-04-11 02:25:15.896802569 -0400 EDT deployed stx-builder-0.1.0 1.16.0
02:25:31 STX-k8s - INFO: stx-tools deployments list:
02:25:31 NAME READY UP-TO-DATE AVAILABLE AGE
02:25:31 debian-master-stx-builder 1/1 1 1 15s
02:25:31 debian-master-stx-builder-files-http 1/1 1 1 15s
02:25:31 debian-master-stx-docker 1/1 1 1 15s
02:25:31 debian-master-stx-lat-tool 1/1 1 1 15s
02:25:31 debian-master-stx-pkgbuilder 1/1 1 1 15s
02:25:31 debian-master-stx-repomgr 1/1 1 1 15s
02:25:31 STX-k8s - INFO: stx-tools pods list:
02:25:32 NAME READY STATUS RESTARTS AGE
02:25:32 debian-master-stx-builder-574f465fc7-9fgnh 1/1 Terminating 1 (10m ago) 5d
02:25:32 debian-master-stx-builder-574f465fc7-pkfqd 1/1 Running 0 15s
02:25:32 debian-master-stx-builder-files-http-74f57b64d5-fdllh 1/1 Running 0 15s
02:25:32 debian-master-stx-docker-7dcd784fd7-vjsgr 1/1 Running 0 15s
02:25:32 debian-master-stx-lat-tool-6b96479fb5-nqq5w 1/1 Running 0 15s
02:25:32 debian-master-stx-pkgbuilder-6d5c497745-t8psj 1/1 Running 0 15s
02:25:32 debian-master-stx-repomgr-76dffbff77-7cmtq 1/1 Running 0 15s
02:25:32 + '[' -f /var/lib/jenkins/.ssh/id_rsa ']'
02:25:32 + notice 'Registering signing server access key'
02:25:32 + set +x

02:25:32 ###
02:25:32 ### Registering signing server access key
02:25:32 ###

02:25:32 + stx control keys-add --key-type=signing-server '--key=~/.ssh/id_rsa'
02:25:32 STX-Control - INFO: Prepare keys for accessing signing server!
02:25:32 STX-k8s - INFO: Running: minikube -p minikube-jenkins-stx-debian-master kubectl -- get pods --selector 'app.kubernetes.io/instance=debian-master,app.kubernetes.io/name=stx-builder' | tail -n +2 | awk '{print $1}'
02:25:32 STX-Control - INFO: run pod cmd: minikube -p minikube-jenkins-stx-debian-master kubectl -- exec -ti debian-master-stx-builder-574f465fc7-9fgnh
02:25:32 debian-master-stx-builder-574f465fc7-pkfqd -- mkdir /home/jenkins/.ssh
02:25:32 error: you must specify at least one command for the container
02:25:32 /bin/sh: line 1: debian-master-stx-builder-574f465fc7-pkfqd: command not found
02:25:32 STX-Control - INFO: run pod cmd: minikube -p minikube-jenkins-stx-debian-master kubectl -- exec -ti debian-master-stx-builder-574f465fc7-9fgnh
02:25:32 debian-master-stx-builder-574f465fc7-pkfqd -- mkdir /root/.ssh
02:25:32 error: you must specify at least one command for the container
02:25:32 /bin/sh: line 1: debian-master-stx-builder-574f465fc7-pkfqd: command not found
02:25:32 STX-Control - INFO: run pod cmd: minikube -p minikube-jenkins-stx-debian-master kubectl -- cp ~/.ssh/id_rsa debian-master-stx-builder-574f465fc7-9fgnh
02:25:32 debian-master-stx-builder-574f465fc7-pkfqd:/home/jenkins/.ssh
02:25:32 error: one of src or dest must be a remote file specification
02:25:32 /bin/sh: line 1: debian-master-stx-builder-574f465fc7-pkfqd:/home/jenkins/.ssh: No such file or directory
02:25:32 STX-Control - INFO: run pod cmd: minikube -p minikube-jenkins-stx-debian-master kubectl -- cp ~/.ssh/id_rsa debian-master-stx-builder-574f465fc7-9fgnh
02:25:32 debian-master-stx-builder-574f465fc7-pkfqd:/root/.ssh
02:25:33 error: one of src or dest must be a remote file specification
02:25:33 /bin/sh: line 1: debian-master-stx-builder-574f465fc7-pkfqd:/root/.ssh: No such file or directory
02:25:33 STX-k8s - INFO: Running: minikube -p minikube-jenkins-stx-debian-master kubectl -- get pods --selector 'app.kubernetes.io/instance=debian-master,app.kubernetes.io/name=stx-lat-tool' | tail -n +2 | awk '{print $1}'
02:25:33 STX-Control - INFO: run pod cmd: minikube -p minikube-jenkins-stx-debian-master kubectl -- exec -ti debian-master-stx-lat-tool-6b96479fb5-nqq5w -- mkdir /root/.ssh
02:25:33 Unable to use a TTY - input is not a terminal or the right kind of file
02:25:33 STX-Control - INFO: run pod cmd: minikube -p minikube-jenkins-stx-debian-master kubectl -- cp ~/.ssh/id_rsa debian-master-stx-lat-tool-6b96479fb5-nqq5w:/root/.ssh
02:25:34 + stx build prepare
02:25:34 STX-k8s - INFO: Running: minikube -p minikube-jenkins-stx-debian-master kubectl -- get pods --selector 'app.kubernetes.io/instance=debian-master,app.kubernetes.io/name=stx-builder' | tail -n +2 | awk '{print $1}'
02:25:34 error: you must specify at least one command for the container
02:25:34 /bin/sh: line 1: debian-master-stx-builder-574f465fc7-pkfqd: command not found
02:25:34 Traceback (most recent call last):
02:25:34 File "/home/localdisk/designer/jenkins/debian-master/repo/stx-tools/stx/lib/stx/stx_build.py", line 200, in handleBuild
02:25:34 subprocess.check_call(cmd, shell=True)
02:25:34 File "/usr/lib64/python3.6/subprocess.py", line 311, in check_call
02:25:34 raise CalledProcessError(retcode, cmd)
02:25:34 subprocess.CalledProcessError: Command 'minikube -p minikube-jenkins-stx-debian-master kubectl -- exec -ti debian-master-stx-builder-574f465fc7-9fgnh

Test Activity
-------------
Build

Workaround
----------
N/A

[Scott Little (slittle1)]

get_pod_name() in stx/lib/stx/k8s.py needs to filter for only 'Running' pods.
It should also fail if more than one match.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/tools/+/880221

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/tools/+/880221
Committed: https://opendev.org/starlingx/tools/commit/ae18863a347587bb4928a4c606ccf4a7d99ccf96
Submitter: "Zuul (22348)"
Branch: master

commit ae18863a347587bb4928a4c606ccf4a7d99ccf96
Author: Scott Little <email address hidden>
Date: Wed Apr 12 14:37:45 2023 -0400

    get_pod_name: Only return running pods

    get_pod_name was observed to return a list of all
    pods satisfying the query without consideration of
    the state of the pod. In one observed scenario the name of
    a runing pod and an terminating pod were returned.

    All subsequent commands used the podname with out checking
    for multiple entries. They clearly expect only a single
    name, not multiple.

    Modify get_pod_name to only return a single rinning pod name.

    Closes-bug: 2015885
    Signed-off-by: Scott Little <email address hidden>
    Change-Id: I0b7bf22c4481fc4a81012fc14ac9d388afd6b065