ci.centos jobs are failing cico node provisioning with " The requested operation requires an API key"

[Ronelle Landy]

The infra team did some work to address some Jenkins vulnerabilities - see email entitled "Happening now: secret, key and password rotation". (which may or may not be related to this current failure).

As of February 15, we see jobs running on ci.centos failing provisioning consistently with the following error:

23:05:12 ~/workspace/tripleo-quickstart-gate-ocata-delorean-quick-basic
23:05:12 + /home/centos/workspace/tripleo-quickstart-gate-ocata-delorean-quick-basic/bin/cico node get --arch x86_64 --release 7 --count 1 --retry-count 6 --retry-interval 60 -f csv
23:05:13 The requested operation requires an API key.
23:05:13 Build step 'Execute shell' marked build as failure

This causes all jobs running under RedHat RDO CI to fail.

See full log at:

https://ci.centos.org/view/rdo/view/tripleo-gate/job/tripleo-quickstart-gate-ocata-delorean-quick-basic/2983/consoleFull

[Ronelle Landy (rlandy)]

We likely require access to the slaves to test this out.

 David Simard was able to get this 'literal' command to work:
$ /home/centos/workspace/tripleo-quickstart-gate-master-delorean-quick-basic/bin/cico node get --arch x86_64 --release 7 --count 1 --retry-count 6 --retry-interval 60 -f csv
"host_id","hostname","ip_address","chassis","used_count","current_state","comment","distro","rel","centos_version","architecture","node_pool"
293,"n38.gusty","172.19.2.166","gusty",924,"Deployed","ac2e8ab8","","","7","x86_64",1

[Attila Darazs (adarazs)]

I did a bit of investigation on this, still not knowing the cause though. What is known:

* The cico client expects the API key as "CICO_API_KEY" in the environment.
* according to dmsimard the variable is defined in /etc/profile, appears correctly in the `env |grep CICO` output and he can request a node with the exact command that fails in our jenkins jobs
* Jenkins just got updated to a new version which has something to do with increasing security, so it might be a reason why this env variable is not propagated

https://lists.rdoproject.org/pipermail/dev/2018-February/008573.html

[Attila Darazs (adarazs)]

The problem might be that Jenkins now doesn't source /etc/profile, only bashrc. David just added the variable in bashrc as well and we'll test if that helps.

[Ronelle Landy (rlandy)]

adarazs - thank you for investigating

[Attila Darazs (adarazs)]

Looks to me that new jobs are not failing anymore on the node request. This was solved by putting the variable in bashrc.