periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-security-* on different branches is failing with error "Quota exceeded, too many key pairs."

[Sandeep Yadav]

Description:-

periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-security-* on different branches is failing with error "Quota exceeded, too many key pairs."

Logs:-
https://logserver.rdoproject.org/openstack-component-security/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-security-master/2404fd5/job-output.txt
~~~
2020-11-04 19:21:58.392563 | TASK [ovb-manage : Create a key in case of extranode]
2020-11-04 19:21:58.862959 | primary | Generating public/private rsa key pair.
2020-11-04 19:21:58.863145 | primary | Your identification has been saved in /home/zuul/extranode-id_rsa.
2020-11-04 19:21:58.863168 | primary | Your public key has been saved in /home/zuul/extranode-id_rsa.pub.
2020-11-04 19:21:58.863183 | primary | The key fingerprint is:
2020-11-04 19:21:58.863200 | primary | SHA256:kQS2FqV5pTaInXpk9dzDNMK3tUwcLd9U1YTdr0qvlog zuul@node-0000559845
2020-11-04 19:21:58.863215 | primary | The key's randomart image is:
2020-11-04 19:21:58.863264 | primary | +---[RSA 3072]----+
2020-11-04 19:21:58.863279 | primary | | +o+.o o..*O|
2020-11-04 19:21:58.863299 | primary | | + X *.=..* *|
2020-11-04 19:21:58.863314 | primary | | . @ B o.+= =o|
2020-11-04 19:21:58.863329 | primary | | = o o ..o +|
2020-11-04 19:21:58.863343 | primary | | . . S . |
2020-11-04 19:21:58.863358 | primary | | . . . |
2020-11-04 19:21:58.863372 | primary | | . o + |
2020-11-04 19:21:58.863387 | primary | | E . + . |
2020-11-04 19:21:58.863401 | primary | | ... |
2020-11-04 19:21:58.863422 | primary | +----[SHA256]-----+
2020-11-04 19:22:01.009177 | primary | Quota exceeded, too many key pairs. (HTTP 403) (Request-ID: req-b02abb35-40a3-4459-aae6-3a5a0a880009)
2020-11-04 19:22:01.444545 | primary | ERROR
2020-11-04 19:22:01.444824 | primary | {
2020-11-04 19:22:01.444926 | primary | "delta": "0:00:02.416794",
2020-11-04 19:22:01.444992 | primary | "end": "2020-11-04 19:22:01.113447",
2020-11-04 19:22:01.445053 | primary | "msg": "non-zero return code",
2020-11-04 19:22:01.445151 | primary | "rc": 1,
2020-11-04 19:22:01.445213 | primary | "start": "2020-11-04 19:21:58.696653"
2020-11-04 19:22:01.445271 | primary | }
2020-11-04 19:22:01.507307 |
~~~

Another example:

https://logserver.rdoproject.org/openstack-component-security/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-security-train/ea938c1/job-output.txt

[Sandeep Yadav (sandeepyadav93)]

Hello,

Looks like this was caused because keypair for tls jobs were not cleaned up.

cleanup-keys.sh which was added in https://review.rdoproject.org/r/#/c/29167/ was not found in cron job.

Ideally, those keypairs should be deleted in job itself, cleanup script mainly needed for cases where it's missed in job. We will try to investigate on why job cleanup not working.

We executed that script(cleanup-keys.sh) manually for now. Will check internally with team if we can add cleanup-keys.sh in cron jobs.