kubectl cmd failed by container not found "calicoctl"

[Peng Peng]

Brief Description
-----------------
create copy the calicoctl IP pool config file to pod calicoctl failed by error: unable to upgrade connection: container not found ("calicoctl")

Severity
--------
Major

Steps to Reproduce
------------------
Create calicoctl service account calicoctl;
create calicoctl cluster role calicoctl;
create calicoctl cluster role binding calicoctl;
create calicoctl pod calicoctl;
create calicoctl IP pool config file;
create copy the calicoctl IP pool config file to pod calicoctl failed

TC-name: WCP_112

Expected Behavior
------------------
kubectl copy successfully

Actual Behavior
----------------
kubectl copy failed

Reproducibility
---------------
Intermittent first time saw this issue, rerun pass.

System Configuration
--------------------
One node system

Lab-name: WCP_112

Branch/Pull Time/Commit
-----------------------
2020-08-13_20-00-00

Last Pass
---------
2020-08-11_20-00-00

Timestamp/Logs
--------------
[2020-08-14 06:06:17,613] 314 DEBUG MainThread ssh.send :: Send 'kubectl cp /tmp/calicoctl-ippool-sriov-pool-group0-data0-vf1.yaml -n kube-system calicoctl:/tmp/calicoctl-ippool-sriov-pool-group0-data0-vf1.yaml'
[2020-08-14 06:06:17,838] 436 DEBUG MainThread ssh.expect :: Output:
error: unable to upgrade connection: container not found ("calicoctl")
controller-0:~$

Test Activity
-------------
Sanity

[Peng Peng (ppeng)]

log added at
https://files.starlingx.kube.cengn.ca/launchpad/1891676

[Ghada Khalil (gkhalil)]

@Peng, please monitor for a re-occurrence. We need a better understanding of how reproducible the issue is.

[Steven Webster (swebster-wr)]

This is coming from an automated test which spawns a container with the calicoctl application installed. We can probably change the test case to set this up via .yaml files.