Daily builds are poorly interleaved

[Paul Sokolovsky]

In a-b frontend, we have support for spreading/interleaving daily builds based on job name hash, see lp:891628. However, lately this feature works poorly - most of the builds are hoarded together by the end of 00:00-07:00 UTC window. Need to fix this.

[Paul Sokolovsky (pfalcon)]

Corresponding changes were deployed. I went thru bunch of builds and flipped their "daily" setting to propagate changes. They seem to have dispersed previous clusterizations we had. Need to monitor daily builds for few days though.

[Paul Sokolovsky (pfalcon)]

md5 must be very bad hash function, because there's still not the picture I personally would expect from my changes. Ee still have clusterization patterns in our build times. Specifically:

00:00-01:30 - Very few builds started
01:30-04:00 - Large share of builds are started
04:00-06:00 - Less packed than previous, but still much more crowded than 00:00-01:30, and in this window, 3 build manage to get clashed again to start at the same time.

But still, this provides enough improvement over previous situation - even though there's clusterization, but at least within a cluster, builds are interleaved by at least 15min, which allows to avoid great deal of git servers thrashing (well, except that sticky 3 builds-at-the-same-time anomaly).

[Paul Sokolovsky (pfalcon)]

Closing per above.