If Jenkins is down at submission, the ticket gets stuck in queued forever

[Evan]

I ran a deployment and Jenkins failed to start due to a race (as far as we can tell). I hadn't noticed this and tried to feed a ticket through, which is still sitting in the queued state an hour later, long after restarting Jenkins.

Nothing in the rabbit queues.
Nothing queued up in Jenkins.

[Andy Doan (doanac)]

i wish we had the lander's run_worker log for this, but i think this might be easy to re-create. I suspect this could be our issue:

http://bazaar.launchpad.net/~canonical-ci-engineering/ubuntu-ci-services-itself/trunk/view/head:/lander/lander/run_worker.py#L125

Looks like we try a fixed amount of times to contact jenkins before we give up. I'm thinking we probably want these loops to be "while True" type things.

It would be nice to have this contribute to our "engine health" page in the webui. This service doesn't have a REST api running though, so it might not be possible for phase0

[Chris Johnston (cjohnston)]

I think this *should* be fixed for now based on bug #1288947... It still needs to be figured out, but I don't think we ever ran into this issue before we started using the IS charm.

[Andy Doan (doanac)]

So the bug won't happen anymore. But I think we probably want to make the lander code more resilient still. I'm going to knock the importance down to "high" since we aren't dead in the water now.

[Andy Doan (doanac)]

I'd like to add a health status for this, but that might take a bit longer. this is the minimal for phase-0, and i'll see what i can do to get more done.