UHS demo GEM Cluster failure

[Laurentiu D.]

laurentiu@gemcontrol:~$ openquake --config-file=/usr/openquake/demos/uhs/config.gem --output-type=xml --log-level=debug
[2012-04-18 16:54:11,534 #592 - INFO MainProcess/8731 supervisor] Entering supervisor for job 592
[2012-04-18 16:54:11,549 #592 - DEBUG MainProcess/8731 amqplib] Start from server, version: 8.0, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2011 VMware, Inc.', u'capabilities': {}, u'platform': u'Erlang/OTP', u'version': u'2.7.1'}, mechanisms: [u'PLAIN', u'AMQPLAIN'], locales: [u'en_US']
[2012-04-18 16:54:11,551 #592 - DEBUG MainProcess/8731 amqplib] Open OK! known_hosts []
[2012-04-18 16:54:11,552 #592 - DEBUG MainProcess/8731 amqplib] using channel_id: 1
[2012-04-18 16:54:11,553 #592 - DEBUG MainProcess/8731 amqplib] Channel open
[2012-04-18 16:54:11,797 #592 gemcontrol.ethz.ch INFO MainProcess/8730 root] Storing source model from job config
[2012-04-18 16:54:12,203 #592 gemcontrol.ethz.ch INFO MainProcess/8730 root] Storing GMPE map from job config
[2012-04-18 16:54:12,205 #592 gemcontrol.ethz.ch DEBUG MainProcess/8730 hazard] -data_length: 1
[2012-04-18 16:54:12,205 #592 gemcontrol.ethz.ch DEBUG MainProcess/8730 hazard] -#subtasks: 1
[2012-04-18 16:55:10,859 #592 gemmicro02.(null) INFO PoolWorker-26/55813 root] Computing UHS for job_id=592, site=Site(0.0, 0.0), realization=0. UHS results will be serialized to the database.
[2012-04-18 16:54:19,479 #592 gemcontrol.ethz.ch CRITICAL MainProcess/8730 root] Calculation failed with exception: 'invalid byte sequence for encoding "UTF8": 0x00
'
Traceback (most recent call last):
  File "/usr/bin/openquake", line 166, in <module>
    log_level=args.log_level)
  File "/usr/lib/pymodules/python2.7/openquake/engine.py", line 727, in run_calculation
    _launch_calculation(calc_proxy, sections)
  File "/usr/lib/pymodules/python2.7/openquake/engine.py", line 800, in _launch_calculation
    calculator.execute()
  File "/usr/lib/pymodules/python2.7/openquake/calculators/hazard/uhs/core.py", line 264, in execute
    ath=uhs_task_handler, ath_args=ath_args)
  File "/usr/lib/pymodules/python2.7/openquake/utils/tasks.py", line 82, in distribute
    _check_exception(results)
  File "/usr/lib/pymodules/python2.7/openquake/utils/tasks.py", line 94, in _check_exception
    raise result
django.db.utils.DatabaseError: invalid byte sequence for encoding "UTF8": 0x00

[2012-04-18 16:54:20,543 #592 - INFO MainProcess/8731 root] Recording stop time for job 592 to calc_stats
[2012-04-18 16:54:20,575 #592 - INFO MainProcess/8731 root] Cleaning up after job 592
[2012-04-18 16:54:20,576 #592 - INFO MainProcess/8731 root] KVS garbage collection removed 5 keys for job 592
[2012-04-18 16:54:20,581 #592 - DEBUG MainProcess/8731 amqplib] Closed channel #1
[2012-04-18 16:54:20,582 #592 - INFO MainProcess/8731 supervisor] Job 592 finished in 0:00:09.023892
[2012-04-18 16:54:20,582 #592 - INFO MainProcess/8731 supervisor] Exiting supervisor for job 592

[Lars Butler (lars-butler)]

First attempt at a solution:

on gemcontrol
in /etc/postgresql/9.1/main/postgresql.conf

set
`standard_conforming_strings = off`

and restarted postgres.

Dr. Danciu is trying to run openquake again to see if that resolves the issue.

Related bug: bug # 911714, which is related to a Django bug: https://code.djangoproject.com/ticket/16778

[Lars Butler (lars-butler)]

There are still other issues on the cluster, but it appears this one is solved.