openquake error on gemsuns cluster - long calculation

[Laurentiu D.]

2011-10-18 22:24:39,475 #29 gemsun03.ethz.ch INFO PoolWorker-14/26409 hazard] Computing MEAN curves for 2312 sites (job_id 29)
[2011-10-18 22:24:39,475 #29 gemsun03.ethz.ch INFO PoolWorker-14/26409 hazard] Computing MEAN curves for 2312 sites (job_id 29)
[2011-10-18 22:24:39,475 #29 gemsun03.ethz.ch INFO PoolWorker-14/26409 hazard] Computing MEAN curves for 2312 sites (job_id 29)
[2011-10-18 22:24:39,903 #29 gemsun03.ethz.ch INFO PoolWorker-5/26400 hazard] Computing MEAN curves for 2355 sites (job_id 29)
[2011-10-18 22:24:39,903 #29 gemsun03.ethz.ch INFO PoolWorker-5/26400 hazard] Computing MEAN curves for 2355 sites (job_id 29)
[2011-10-18 22:24:39,903 #29 gemsun03.ethz.ch INFO PoolWorker-5/26400 hazard] Computing MEAN curves for 2355 sites (job_id 29)
[2011-10-18 22:22:20,825 #29 gemsun04.ethz.ch INFO PoolWorker-28/17554 hazard] Computing MEAN curves for 2312 sites (job_id 29)
[2011-10-18 22:20:36,602 #29 gemsun02.ethz.ch CRITICAL MainProcess/12238 root] Job failed with exception: 'unsupported operand type(s) for +: 'NoneType' and 'str''
Traceback (most recent call last):
  File "/usr/bin/openquake", line 124, in <module>
    job.run_job(FLAGS.config_file, FLAGS.output_type)
  File "/usr/lib/pymodules/python2.7/openquake/job/__init__.py", line 80, in run_job
    a_job.launch()
  File "/usr/lib/pymodules/python2.7/openquake/job/__init__.py", line 469, in launch
    self.execute()
  File "/usr/lib/pymodules/python2.7/openquake/java.py", line 283, in unwrap_exception
    return func(*targs, **tkwargs)
  File "/usr/lib/pymodules/python2.7/openquake/hazard/opensha.py", line 71, in preloader
    return fn(self, *args, **kwargs)
  File "/usr/lib/pymodules/python2.7/openquake/hazard/opensha.py", line 424, in execute
    map_serializer=self.serialize_mean_hazard_map)
  File "/usr/lib/pymodules/python2.7/openquake/hazard/opensha.py", line 323, in do_means
    flatten_results=True)
  File "/usr/lib/pymodules/python2.7/openquake/utils/tasks.py", line 107, in distribute
    the_results = _handle_subtasks(subtasks, flatten_results)
  File "/usr/lib/pymodules/python2.7/openquake/utils/tasks.py", line 167, in _handle_subtasks
    raise WrongTaskParameters(exc.args[0])
openquake.utils.tasks.WrongTaskParameters: unsupported operand type(s) for +: 'NoneType' and 'str'
[2011-10-18 22:20:39,612 #29 - INFO MainProcess/12239 supervisor] Process 12238 not running
[2011-10-18 22:20:39,615 #29 - INFO MainProcess/12239 supervisor] job finished with status u'failed'
[2011-10-18 22:20:39,615 #29 - INFO MainProcess/12239 root] Recording stop time for job 29 to job_stats
[2011-10-18 22:20:39,661 #29 - INFO MainProcess/12239 root] Cleaning up after job 29
[2011-10-18 22:20:47,355 #29 - INFO MainProcess/12239 root] KVS garbage collection removed 247475 keys for job 29
[2011-10-18 22:20:47,429 #29 - INFO MainProcess/12239 supervisor] Job 29 finished in 1 day, 5:27:39.941746
[2011-10-18 22:20:47,429 #29 - INFO MainProcess/12239 supervisor] Exiting supervisor for job 29
Aborted

real 1767m42.127s
user 36m43.510s
sys 7m40.750s

[Muharem Hrnjadovic (al-maisan)]

A first and partial fix: https://github.com/gem/openquake/pull/545

[Laurentiu D. (laurentiu.danciu)]

Received unregistered task of type 'openquake.hazard.tasks.compute_hazard_curve'.
The message has been ignored and discarded.

Did you remember to import the module containing this task?
Or maybe you are using relative imports?
Please see http://bit.ly/gLye1c for more information.

The full contents of the message body was:
{'retries': 0, 'task': 'openquake.hazard.tasks.compute_hazard_curve', 'args': [], 'expires': None, 'eta': None, 'taskset': '532260fa-711e-4a77-9df6-2bb456fba10c', 'kwargs': {'site_list': [Site(73.0, 16.0), Site(73.1, 16.0), Site(73.2, 16.0), Site(73.3, 16.0), Site(73.4, 16.0), Site(73.5, 16.0), Site(73.6, 16.0), Site(73.7, 16.0), Site(73.8, 16.0), Site(73.9, 16.0), Site(74.0, 16.0), Site(74.1, 16.0), Site(74.2, 16.0), Site(74.3, 16.0), Site(74.4, 16.0), Site(74.5, 16.0), Site(74.6, 16.0), Site(74.7, 16.0), Site(74.8, 16.0), Site(74.9, 16.0), Site(75.0, 16.0), Site(73.0, 16.1), Site(73.1, 16.1), Site(73.2, 16.1), Site(73.3, 16.1), Site(73.4, 16.1), Site(73.5, 16.1), Site(73.6, 16.1), Site(73.7, 16.1), Site(73.8, 16.1), Site(73.9, 16.1), Site(74.0, 16.1), Site(74.1, 16.1), Site(74.2, 16.1), Site(74.3, 16.1), Site(74.4, 16.1), Site(74.5, 16.1), Site(74.6, 16.1), Site(74.7, 16.1), Site(74.8, 16.1), Site(74.9, 16.1), Site(75.0, 16.1), Site(73.0, 16.2), Site(73.1, 16.2), Site(73.2, 16.2), Site(73.3, 16.2), Site(73.4, 16.2), Site(73.5, 16.2), Site(73.6, 16.2), Site(73.7, 16.2), Site(73.8, 16.2), Site(73.9, 16.2), Site(74.0, 16.2), Site(74.1, 16.2), Site(74.2, 16.2), Site(74.3, 16.2), Site(74.4, 16.2), Site(74.5, 16.2), Site(74.6, 16.2), Site(74.7, 16.2), Site(74.8, 16.2), Site(74.9, 16.2), Site(75.0, 16.2), Site(73.0, 16.3), Site(73.1, 16.3), Site(73.2, 16.3), Site(73.3, 16.3), Site(73.4, 16.3), Site(73.5, 16.3), Site(73.6, 16.3), Site(73.7, 16.3), Site(73.8, 16.3), Site(73.9, 16.3), Site(74.0, 16.3), Site(74.1, 16.3), Site(74.2, 16.3), Site(74.3, 16.3), Site(74.4, 16.3), Site(74.5, 16.3), Site(74.6, 16.3), Site(74.7, 16.3), Site(74.8, 16.3), Site(74.9, 16.3), Site(75.0, 16.3), Site(73.0, 16.4), Site(73.1, 16.4), Site(73.2, 16.4), Site(73.3, 16.4), Site(73.4, 16.4), Site(73.5, 1...

[Muharem Hrnjadovic (al-maisan)]

A similar error occurred during a test run in the gemsun cluster. See attached log files.

[Muharem Hrnjadovic (al-maisan)]

The exception above has been fixed -- the current front line is here: https://bugs.launchpad.net/openquake/+bug/894024 (rabbitmq is crashing despite a relatively low number of queues).