Add median latency to grafana dashboard

[Samuel Allan]

We have a few latency figures available on the grafana dashboard: average, 90%tile, 99%tile, 99.9%tile. However, one more would be useful to see a clearer picture of the distribution: the median.

Often the distribution, especially if the device is stressed, can be probably like a hockey stick - the top 90% or so can be orders of magniture higher than the rest of the numbers. For example, i'm looking at one now where the 90%tile is 23.5ms, the 99%tile is 1.13s, and 99.9%tile is 10.8s. This can skew the average high, resulting in output that almost looks like a bug (prompting https://bugs.launchpad.net/charm-woodpecker/+bug/2043350 for example).

It would be useful to include the median (50%tile metric), to help visualise the distribution better. Perhaps also the stddev, but definitely at least the median.

[Samuel Allan (samuelallan)]

Example of added information with the 50%tile - here seeing the median being a lot lower than the average demonstrates that the distribution is top heavy.