RDO jobs failing with POST_FAILURE

[Marios Andreou]

RDO jobs are failing with POST_FAILURE and there are no logs generated. There have been many examples today like at [1][2] and just now the train container-push job (periodic promotion) failed with the same at [3][4] rhel/centos so this also blocks promotions.

[1] https://review.rdoproject.org/r/23882
[2] https://review.rdoproject.org/r/#/c/23873
[3] https://review.rdoproject.org/zuul/build/f220f10c7bce4051ade3bdc6d4914fe5
[4] https://review.rdoproject.org/zuul/build/1af2d63ebd3e468db53fc995d54d319d

[Marios Andreou (marios-b)]

internal sf-ops folks are looking into that (via chandan earlier on irc) fbo/mhu are at least aware of the issue

[Marios Andreou (marios-b)]

17:57 < jpena> raukadah: the ceph cluster it's running on is currently having I/O issues (some disks failed and it is rebuilding)
17:57 < raukadah> jpena: and reagrding rdo log server perf issue?
17:57 < raukadah> we have got hitten by post failures
17:57 < jpena> raukadah: same thing, it's running on the same ceph

[chandan kumar (chkumar246)]

https://review.rdoproject.org/zuul/builds?result=POST_FAILURE from this the post failure is gone, it was not on weekend, we can wait for today and close it.

[Marios Andreou (marios-b)]

ack looks like 29th was the last time we saw that so marking invalid (closest thing i could find to 'closed' :/) we can revisit if we see it again