Cannot ingest Yohkoh data in db

[MEDOC IAS]

Hello,
When I try to retrieve Yohkoh data I have a problem during the ingestion in db.

the command :
python downloader.py -s "1996-05-07 00:00:00" -e "1996-05-08 00:00:00" -d gsfc_backup_sxt -l /var/www/helioviewer/log/gsfc-old-backup-1996-05-07.log

The error :
Traceback (most recent call last):
  File "downloader.py", line 177, in <module>
    sys.exit(main())
  File "downloader.py", line 65, in main
    daemon.start(args.start, args.end)
  File "/var/www/helioviewer/install/helioviewer/hvpull/net/daemon.py", line 101, in start
    self.query(starttime, endtime)
  File "/var/www/helioviewer/install/helioviewer/hvpull/net/daemon.py", line 193, in query
    self.acquire(new_urls)
  File "/var/www/helioviewer/install/helioviewer/hvpull/net/daemon.py", line 277, in acquire
    self.ingest(finished)
  File "/var/www/helioviewer/install/helioviewer/hvpull/net/daemon.py", line 379, in ingest
    process_jp2_images(images, self.image_archive, self._db)
  File "/var/www/helioviewer/install/helioviewer/jp2.py", line 40, in process_jp2_images
    insert_images(subset, sources, root_dir, cursor, mysql, step_fxn)
  File "/var/www/helioviewer/install/helioviewer/jp2.py", line 73, in insert_images
    source = sources[img["observatory"]][img["instrument"]][img["detector"]][img["measurement"]]
KeyError: 'AlMg'

Another error :
Traceback (most recent call last):
  File "downloader.py", line 177, in <module>
    sys.exit(main())
  File "downloader.py", line 65, in main
    daemon.start(args.start, args.end)
  File "/var/www/helioviewer/install/helioviewer/hvpull/net/daemon.py", line 101, in start
    self.query(starttime, endtime)
  File "/var/www/helioviewer/install/helioviewer/hvpull/net/daemon.py", line 193, in query
    self.acquire(new_urls)
  File "/var/www/helioviewer/install/helioviewer/hvpull/net/daemon.py", line 277, in acquire
    self.ingest(finished)
  File "/var/www/helioviewer/install/helioviewer/hvpull/net/daemon.py", line 379, in ingest
    process_jp2_images(images, self.image_archive, self._db)
  File "/var/www/helioviewer/install/helioviewer/jp2.py", line 40, in process_jp2_images
    insert_images(subset, sources, root_dir, cursor, mysql, step_fxn)
  File "/var/www/helioviewer/install/helioviewer/jp2.py", line 75, in insert_images
    source = sources[img["observatory"]][img["instrument"]][img["detector"]][img["measurement"]]
KeyError: 'Al.1'

[MEDOC IAS (pablo-alingery)]

NB :
 if I add the following in /var/www/helioviewer/install/helioviewer/jp2.py line 73 I can ingest data in db ( directory created Al.1 instead of Al01)
       if img["measurement"]=="AlMg":
                img["measurement"] ="AlMgMn"
        elif img["measurement"]=="Al.1":
                img["measurement"] ="thin-Al"

Regards,
Pablo
IAS

[Jeff Stys (jstys-z)]

Hello Pablo-

I would be happy to look into this.

Could you please let me know what revision of the helioviewer code you are running on your server?

   bzr revno

thank you,
Jeff

[MEDOC IAS (pablo-alingery)]

Hi,

The version is 796 on prod server but I have tested and reproduced the same on our test machine as well and the version is 799 on it.

see :
root@helioviewer:/var/www/helioviewer# bzr revno
796
 root@inf-helio-test:/usr/local/helioviewer/PabloIas# bzr revno
799

Regards,
Pablo

[Jeff Stys (jstys-z)]

Also please supply your data server specification file you are using for Yohkoh SXT from the /var/www/helioviewer/install/helioviewer/hvpull/servers/ directory.

[Jack Ireland (jack-ireland)]

For what it's worth, if we have a copy of the data server specification file we used, and if we used the same kind of naming convention, then it would be called msu.py, since the data came from Montana State University.

[MEDOC IAS (pablo-alingery)]

Hello,
The file we used was named gsfc_backup_sxt.py (attached ) we can obviously rename it into msu.py to fit the convention .

Regards,
Pablo

[Jack Ireland (jack-ireland)]

Pablo,

No need to rename it. I'm not sure that the original repository that we got the data from exists anymore. The file name you have makes more sense anyway, given that you are downloading the data from GSFC. Jeff is looking in to the issue.

PROBA2 data should be simpler. The server is defined by the file rob.py, which should be in your version of the Helioviewer software.

Jack

[Jeff Stys (jstys-z)]

Pablo-

The value of the FITS keyword <WAVELNTH> in the JP2000 files must match the value of the `name` field in the `measurements` table in the database.

This is awkward in the case where you would like to display a different measurement name in the user interface than what is encoded in <WAVELNTH>.

We have previously addressed this issue in a development branch of the helioviewer.org source code that we are targeting for the version 3.0 milestone. In that code branch, there is both a FITS name and a display name for each measurement in the database.

 In the meantime, the workaround you described of editing the install/helioviewer/jp2.py file to override the value found in <WAVELNTH> with the value specified in the database is reasonable. Another option would be to set the name of the measurements in the database to match the value in the FITS keyword prior to the important, and then change it back after the import.

My apologies for the inconvenience.