Icebox generates calculations marked "success" but empty

[Daniele Viganò]

In a specific condition Icebox will report a calculation as "successful" (green) but actually the calculation has not run and the maps are empty.

It looks like an issue with concurrency. How to reproduce (with a dev environment):

1) Upload three calculations
2) 1 is starting and 2, 3 are in queue
3) 1 ends, 2 and 3 starts at the same time (first issue)
4) 2 and 3 become successful (green) at the very same time
5) 2 and 3 don't contain any result

On the engine server's log they report only:

[2014-11-17 16:33:27,535 hazard job #2 - PROGRESS MainProcess/1309] ** pre_executing (hazard)
[2014-11-17 16:33:28,560 hazard job #3 - PROGRESS MainProcess/1309] ** pre_executing (hazard)

The issue probably is more related to the engine server rather than the platform itself.

[Daniele Viganò (daniele-vigano)]

Using the command line upload tool provided by Michele to send jobs directly to the API, everything works as expected. So problem caused by the platform.

[Daniele Viganò (daniele-vigano)]

After further investigation, a guilty looks to be the platform and the fact that in debugging we are running it in a single process.

The workflow is:

1) calculation #1 ends and "creating layers" phase is starting
2) calculation #2 sends a callback to the platform but it's busy with the creation layers, #2 fails
3) same with #3 just after #2

Is still unclear why the platform reports the calculations as successful.

This will impact scalability and looks related to https://bugs.launchpad.net/oq-platform/+bug/1321772

[Daniele Viganò (daniele-vigano)]

The issue persist using gunicorn with 8 workers.

pip install gunicorn
cd ~/oq-platform/openquakeplatform/openquakeplatform
gunicorn -w 8 wsgi:application -b 0.0.0.0:8000