[feature request] Automated scan to look for missing data

Bug #1169227 reported by Jack Ireland on 2013-04-15
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Helioviewer.org
Medium
Jack Ireland

Bug Description

Sometimes the download process on hv.org goes down for a couple of hours, and then gets restarted manually. Sometimes the manual restart misses can miss the start and end times corresponding to when the automated process went down. This feature request is to write a script which regularly performs a scan of the data available online and compares that to the data in Helioviewer.org. The scan would examine different time-scales, and each scan would happen at different time-scales.

(1) once a week, scan the previous week.
(2) once a week, scan the previous 4 weeks (an approximate month).
(3) once a week? 4 weeks?, scan the previous year (365 days).

Each of these scans would happen for each of the data providers registered. The new hvpull / downloader code provides the functionality that can enable a scanning to look for missing data. It would also be interesting, but not necessary, to maintain a record of how much data was downloaded by these scans.

Changed in helioviewer.org:
importance: Undecided → Medium
assignee: nobody → Jack Ireland (jack-ireland)
milestone: none → 3.0.0
Keith Hughitt (keith-hughitt) wrote :

One thing to be careful of is that you don't accidentally re-download corrupted images. There is a separate database table to track corrupt images, so it should be pretty straight-forward; just make sure you use the database instead of the filesystem and check both "images" and "corrupt".

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers