Add random offset to juju(?) jobs

[Matt Verran]

While investigating the ongoing issue with mysql instances health checks going offline as per https://bugs.launchpad.net/snap-openstack/+bug/2016541 I noticed there is a regular job that runs all mysql instances at exactly the same time. While on a multi-node MaaS install this is probably less of an issue it may cause scalability on smaller installs.

Even on larger installs avoiding synchronised timed tasks is probably best avoided.

[Matt Verran (mv-2112)]

Using nmon, have identified the 'juju jobs' as the pebble process. Suggest this may be partial cause of https://bugs.launchpad.net/snap-openstack/+bug/2049857.

When this occurs each pebble process is pushing 3-5MB/s of disk writes. On larger systems this can probably be absorbed better than an all-in-one, although still doesn't feel like the right thing to do to have them triggered in sync.

Next step, identify what it is writing...

[Matt Verran (mv-2112)]

1708420563.025950 newfstatat(AT_FDCWD, "/var/log/mysql/archive_error/error.log-20240220_08:24", {st_mode=S_IFREG|0640, st_size=9488, ...}, 0) = 0
1708420563.026052 newfstatat(AT_FDCWD, "/var/log/mysql/archive_error/error.log-20240220_08:25", {st_mode=S_IFREG|0640, st_size=5152, ...}, 0) = 0
1708420563.026132 newfstatat(AT_FDCWD, "/var/log/mysql/archive_error/error.log-20240220_08:26", {st_mode=S_IFREG|0640, st_size=4928, ...}, 0) = 0
1708420563.026210 newfstatat(AT_FDCWD, "/var/log/mysql/archive_error/error.log-20240220_08:27", {st_mode=S_IFREG|0640, st_size=5376, ...}, 0) = 0
1708420563.026288 newfstatat(AT_FDCWD, "/var/log/mysql/archive_error/error.log-20240220_08:28", {st_mode=S_IFREG|0640, st_size=5376, ...}, 0) = 0
1708420563.026365 newfstatat(AT_FDCWD, "/var/log/mysql/archive_error/error.log-20240220_08:29", {st_mode=S_IFREG|0640, st_size=5376, ...}, 0) = 0
1708420563.026448 newfstatat(AT_FDCWD, "/var/log/mysql/archive_error/error.log-20240220_08:30", {st_mode=S_IFREG|0640, st_size=5376, ...}, 0) = 0
1708420563.026525 newfstatat(AT_FDCWD, "/var/log/mysql/archive_error/error.log-20240220_08:31", {st_mode=S_IFREG|0640, st_size=5376, ...}, 0) = 0
1708420563.026603 newfstatat(AT_FDCWD, "/var/log/mysql/archive_error/error.log-20240220_08:32", {st_mode=S_IFREG|0640, st_size=5376, ...}, 0) = 0
1708420563.026679 newfstatat(AT_FDCWD, "/var/log/mysql/archive_error/error.log-20240220_08:33", {st_mode=S_IFREG|0640, st_size=5376, ...}, 0) = 0
1708420563.026756 newfstatat(AT_FDCWD, "/var/log/mysql/archive_error/error.log-20240220_08:34", {st_mode=S_IFREG|0640, st_size=5376, ...}, 0) = 0
1708420563.026834 newfstatat(AT_FDCWD, "/var/log/mysql/archive_error/error.log-20240220_08:35", {st_mode=S_IFREG|0640, st_size=5376, ...}, 0) = 0
1708420563.026911 newfstatat(AT_FDCWD, "/var/log/mysql/archive_error/error.log-20240220_08:36", {st_mode=S_IFREG|0640, st_size=5152, ...}, 0) = 0
1708420563.026989 newfstatat(AT_FDCWD, "/var/log/mysql/archive_error/error.log-20240220_08:37", {st_mode=S_IFREG|0640, st_size=5376, ...}, 0) = 0
1708420563.027068 newfstatat(AT_FDCWD, "/var/log/mysql/archive_error/error.log-20240220_08:38", {st_mode=S_IFREG|0640, st_size=5152, ...}, 0) = 0
1708420563.027163 newfstatat(AT_FDCWD, "/var/log/mysql/archive_error/error.log-20240220_08:39", {st_mode=S_IFREG|0640, st_size=5376, ...}, 0) = 0
1708420563.027242 newfstatat(AT_FDCWD, "/var/log/mysql/archive_error/error.log-20240220_08:40", {st_mode=S_IFREG|0640, st_size=5376, ...}, 0) = 0
1708420563.027319 newfstatat(AT_FDCWD, "/var/log/mysql/archive_error/error.log-20240220_08:41", {st_mode=S_IFREG|0640, st_size=5376, ...}, 0) = 0
1708420563.027411 newfstatat(AT_FDCWD, "/var/log/mysql/archive_error/error.log-20240220_08:42", {st_mode=S_IFREG|0640, st_size=5376, ...}, 0) = 0
1708420563.027503 newfstatat(AT_FDCWD, "/var/log/mysql/archive_error/error.log-20240220_08:43", {st_mode=S_IFREG|0640, st_size=5376, ...}, 0) = 0
1708420563.027582 newfstatat(AT_FDCWD, "/var/log/mysql/archive_error/error.log-20240220_08:44", {st_mode=S_IFREG|0640, st_size=5376, ...}, 0) = 0
1708420563.027660 newfstatat(AT_FDCWD, "/var/log/mysql/archive_error/error.log-20240220_08:45", {st_mode=S_IFREG|0640, st_size=5382, ....

[Paulo Machado (paulomachado)]

Tracking under gh issue: https://github.com/canonical/mysql-k8s-operator/issues/387 as an enhancement