Remove videos/screenshots which have not been accessed recently and regenerate on-demand

Bug #887609 reported by Keith Hughitt on 2011-11-08
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Helioviewer.org
High
Unassigned

Bug Description

A few changes are required to support this:

1. Write a python script to automatically scan movies and screenshots older than ~1 month and remove those which have not been accessed recently. Status should be updated in the database to reflect the change. Movie thumbnails should be preserved for use with video gallery, etc.

2. When a movie or screenshot that has been removed in this manner is later requested, it should then be regenerated, using a dedicated queue (which should be used infrequently) so that is ready quickly. A message should be presented to the user that the video is being re-created and will be available shortly, and to try back in 3-5 minutes (for movies).

Other considerations:
1. Would it be more or less efficient to store last access time and number of accesses in the database, instead of simply using OS access times?
2. If so, then all movie requests should go through the API, instead of directly referring to video (e.g. in HTML5 player).
2.

Changed in helioviewer.org:
assignee: nobody → Keith Hughitt (keith-hughitt)
milestone: none → 2.3.1
Changed in helioviewer.org:
milestone: 2.3.1 → 2.4.0
Keith Hughitt (keith-hughitt) wrote :

Last access time could be checked using either the file information (e.g. stat), or by storing the access time in the database. The later would require more queries, but would be significantly faster for finding movies and screenshots to be removed.

Basic approach:

1. Add lastAccess field to movies and screenshots tables
2. Add a new movie status (e.g. "4-deleted")
3. Add new column to screenshot db: "cached" (tinyint(1) unsigned)
4. Modify downloadMovie, playMovie, downloadScreenshot, getMovieStatus, etc. (getUserVideos?) to deal with deleted items.
5. Create a separate queue for movie_recreation which takes priority over on_demand_movie and alternate_movie
6. Whenever API call is made which uses an item (downloadMovie, etc) update the lastAccess field in db

Changed in helioviewer.org:
assignee: Keith Hughitt (keith-hughitt) → nobody
Jack Ireland (jack-ireland) wrote :

Movie recreation from the stored request will need to be implemented. In addition, some kind of "aging out" of material in the cache needs to be implemented.

Changed in helioviewer.org:
milestone: 2.4.0 → 3.0.0
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers