System Will occasionally hang when submitting results

[Seth Beinhart]

System will hang forever when submitting results. The dashboard reports the following log.

11:54:23 PM INFO: Submitting the test result with parameters = {u'token': '<HIDDEN>', u'stream': u'/anonymous/catalina-p...
11:54:23 PM DEBUG: server RPC endpoint URL: <omitting for privacy >/
11:54:24 PM INFO: Dashboard : http://<omitting for privacy >

Hangs forever

Canceling the job does not release the system.

[Michael Hudson-Doyle (mwhudson)]

I think I've seen this a couple of times (even since the last round of fixes designed to fix this). I think it has actually submitted results though and is for some reason not exiting cleanly.

[Seth Beinhart (seth-beinhart)]

Antonio, I have a seen this happen under 2 cases which may help you narrow down your search.

The first is if fastboot becomes unresponsive during deploy_linaro_image. The process falls into a state of uninterrupted sleep. PDU flip will recover the over all process. Without this flip it will hang forever.

The second is when i am running a 24 hour test which generates a large bundle..

Hopefully these help you.

[Antonio Terceiro (terceiro)]

to reproduce: try to launch some zombie process from the dispatcher. look at how the scheduler monitor deals with this.

[Tyler Baker (tyler-baker)]

I'm see this issue on community ATM. Looking at the postgres log there are a huge amount of transaction deadlocks.