Data Load Failure - Bad Image Header

[Karthik]

I tried to load all the data, but only part of the data was loaded.

Moreover, I did not get the message "Found %d JPEG2000 images", instead the message still was
"Searching for JPEG 2000 Images...".

However, I started the data load and ended up loading only part of the data set.

Please see the attached screen shot of the error message on the console for the data load process.

[Keith Hughitt (keith-hughitt)]

Hmm. from the screenshot it looks like the database info entered was not accepted -- I wonder if the BadImage message is in fact misleading and should not have been raised.

Can you wipe the db (e.g. drop database helioviewer) and double check the login information to see if it works then?

Also- can you tell me a little bit about your system? e.g. os/version, and output from :

python -c 'import sunpy; sunpy.util.system_info()'

[Karthik (karthikgp)]

I changed the permission of the folders, such that all the directories had read and execute access, and after that I do not see the message "(1045, "Access denied for user 'helioviewer'@'localhost' (using password: YES)")" when the program fails.

However, even after that program raised an exception with BadImage message.

Then, when tried loading just the folder contents of one wavelength and and one year (131/2010), with root directory /var/www/jp2, the program loaded that data. The data was available for only six months in this year and wavelength.

Next I tried loading the contents of folder (131/2011), the program took long time (and memory) to process the entire year data, and the program just hangs and throws an exception saying that database already exists.

I believe, initially program throwed the exception of "BadImage" because of large amount of memory consumption when I try to load 6 tera bytes of data. And, when I try to load the data separately in chunks, the program assumes that it has to create data base schema and other data base related stuff from scratch.

Hope this makes sense. See the attached screen shots related to OS, Sunpy, and messages from program for your reference .

[Keith Hughitt (keith-hughitt)]

Okay, here is what I would try:

1) run the installer as a normal user (no sudo)
2) use a different database name to avoid collisions with the existing partially-filled one, or delete the old one first.
3) update SunPy (latest version, 0.2, includes some bug-fixes that may help with the BadImage issue).

Do you know how many images you currently have and are attempting to add with the installer? I don't think the installer has been tested for more than ~1 million images, so it's possible that it needs to be optimized to deal with this kind of situation (e.g. when reading the images in, process them 1000 at a time).

Try the above and then let me know if you run into either memory or BadImage issues and we will try and deal with them separately.

[Karthik (karthikgp)]

I will try these and let you know. Thanks.

[Karthik (karthikgp)]

Here is what I did.

1) Updated SunPy to version 0.2
2) Deleted the old database that I had.
3) Ran the installer as a normal user.
4) Also, the total count of images that we have here little more than 9 million.

This time, I did not data load GUI, but it ran with console.
In the console I entered the jp2 data directory where the images are stored. Moreover, it the script did not ask for database details this time.

After sometime, I get BadImage exception.

See the attached screenshot for details.

[Keith Hughitt (keith-hughitt)]

Karthik,

It looks like for some reason some corrupt images were downloaded into your archive, but the installer is not providing very helpful output regarding which images are problematic.

I just checked and I did add logic in the installer to gracefully handle corrupt images (they should be automatically moved to a "quarantine" folder).

Have you updated the Helioviewer.org code recently? You can use "bzr revno" to determine the current code revision. The latest revision is currently 788.

[Karthik (karthikgp)]

Keith,
The version that I have is 786.

I will use "bzr pull" to update to the latest version. Will let you know after I try with new version of the code.

[Karthik (karthikgp)]

Keith, Even with latest code I get an error, similar to the previous one.

The current version number of the code base that I have is 788.

See the attached screen shot for details.

[Keith Hughitt (keith-hughitt)]

Okay, I think I tracked down the issue -- I had started making some changes a while back to combine code for the Helioviewer.org installer and download daemon, but it appear that I accidentally pushed some unfinished changes to the trunk.

I made a few changes which I believe should fix the problem. When you have the chance, try pulling the latest revision and running it again and let me know how it goes.

[Karthik (karthikgp)]

Keith, Sorry to get back late. I tried running the program two times so far and because of that it took some while.

The program outputs the bad image header files in output, however, after it reaches system maximum RAM, the process runs very slow. How much GB of RAM do you have on your system. I can increase the RAM here with few more GBs. At present we have around 32GB of RAM in our system.

See the screen shot of the application run and memory storage so far. Is there a way that we can update the code to print message on how much work is still pending (something like progress bar).

[Keith Hughitt (keith-hughitt)]

Karthik,

I have not run into this issue before, but I don't think the installer has been tested with such large image archives. We probably need to make some changes to the installer so that it processes a number of files at a time and then frees up memory before moving onto the rest.

Since this issue is separate from your original problem, however, would you mind filing a separate bug request? That way I can close this one before moving onto the memory issue.

[Karthik (karthikgp)]

Thanks Keith. I will create a separate bug for the memory issue.