Add Prometheus scrapable with metrics to daisy

[David Lawson]

We'd like to add a /metrics URL to daisy with some Prometheus scrapable metrics. Definitely we'd like to get a counter for cores retraced and cores uploaded, broken up by suite/arch if that makes sense. Maybe a counter for total retrace time, again broken up by suite/arch? That could give us some interesting insight into average retrace time. Any other metrics you think would be useful would be good as well. Once we've got the raw data we can mangle it into averages and stuff like that in prometheus/grafana.

[Brian Murray (brian-murray)]

While there are counters for cores retraced and cores uploaded these are from rabbitmq and not from the retracers directly, additionally looking at rabbitmq doesn't provide us information about whether or not the retrace was successful or not so there is definitely some room for improvement.

[Brian Murray (brian-murray)]

Is json "prometheus scrapable"? I ask as the Error Tracker already produces json for its own use e.g. https://errors.ubuntu.com/api/1.0/retracers-average-processing-time/?limit=32767&format=json.

[Junien F (axino)]

As far as I know, no. prometheus has its own format, see https://github.com/prometheus/client_python

[Brian Murray (brian-murray)]

I've gone ahead and added the following code to the server team metrics which is now populating the foundationsro data source with retrace results in the foundations_production_retracer_results table.

https://github.com/CanonicalLtd/metrics/blob/master/metrics/foundations_retracers_results.py

I hope that same data source can be used on the Error Tracker dashboard.

[Brian Murray (brian-murray)]

There is also this script which records the average time to retrace on a release and architecture basis.

https://github.com/CanonicalLtd/metrics/blob/master/metrics/foundations_retracers_avg_time.py