[feature request] Automated scan to look for missing data

[Jack Ireland]

Sometimes the download process on hv.org goes down for a couple of hours, and then gets restarted manually. Sometimes the manual restart misses can miss the start and end times corresponding to when the automated process went down. This feature request is to write a script which regularly performs a scan of the data available online and compares that to the data in Helioviewer.org. The scan would examine different time-scales, and each scan would happen at different time-scales.

(1) once a week, scan the previous week.
(2) once a week, scan the previous 4 weeks (an approximate month).
(3) once a week? 4 weeks?, scan the previous year (365 days).

Each of these scans would happen for each of the data providers registered. The new hvpull / downloader code provides the functionality that can enable a scanning to look for missing data. It would also be interesting, but not necessary, to maintain a record of how much data was downloaded by these scans.

[Keith Hughitt (keith-hughitt)]

One thing to be careful of is that you don't accidentally re-download corrupted images. There is a separate database table to track corrupt images, so it should be pretty straight-forward; just make sure you use the database instead of the filesystem and check both "images" and "corrupt".