[feature request] Tracking image acquisition performance

[Jack Ireland]

We assume that having images made available as close to real time is a desirable feature, but we don't know how well we are doing. We could collect more data on how close to real time images appear in our database. This would be a measure of performance of each data pipeline. We have a webpage that shows if each of our data streams are reasonably up to date or not, but we do not track this performance over time. It would be good to know the historical performance of each pipeline, so we can get a better idea of the performance of our image acquisition systems, both currently and over time.

[Jeff Stys (jstys-z)]

A fairly simple way of accomplishing the data collection needed for this would be to add a timestamp field to the images table. The date field stores the date of the image itself, but the timestamp field will automatically indicate the date at which the record was created (or modified). Since we don't modify existing rows in the images table (except in very rare circumstances), the delta between the timestamp field and the date field will tell us how fresh the data was when it was imported.

This adds a fair amount of overhead to the database (in terms of storage size of the table and possibly the index) since it contains over 40 million records, but may be worth doing, at least temporarily to get a snapshot. Of course it's not going to magically give us historical data and it's unlikely that we can get that historical data from file system timestamps.