ubuntu-repository-cache doesn't time out on upstream mirror connection issues

[Paul Gear]

rsyncing metadata with peer ubuntu-repository-cache units uses an rsync command which includes --timeout=120, but rsync from the upstream servers doesn't include this, and so can hang indefinitely, causing the metadata to become out of date. Here's an example from our Azure mirrors: https://pastebin.canonical.com/153029/, which hung for 15 hours before I killed it and started another.

Given that /var/lib/juju/agents/unit-ubuntu-repository-cache-0/charm/hooks/ubuntu-repository-cache-sync is run hourly, it should call rsync with a timeout of 1800 seconds or less, I would think.

[Paul Gear (paulgear)]

Actually, here's a better pastebin showing the difference between the two rsync invocations: https://pastebin.canonical.com/153028/

[Paul Gear (paulgear)]

It also may be worth creating a Nagios check to see how many rsyncs are running and alerting if it gets too high.

[Haw Loeung (hloeung)]

Seeing this today as well:

root 1210 0.0 0.0 23660 2292 ? Ss Mar13 0:11 cron
root 13580 0.0 0.0 59644 3032 ? S Apr09 0:00 \_ CRON
www-sync 13582 0.0 0.0 4448 688 ? Ss Apr09 0:00 | \_ /bin/sh -c juju-run ubuntu-repository-cache/0 /var/lib/juju/agents/unit-ubuntu-repository-cache-0/charm/hooks/ubuntu-repository-cache-sync
www-sync 13584 0.0 0.1 236868 31260 ? Sl Apr09 0:04 | \_ juju-run ubuntu-repository-cache/0 /var/lib/juju/agents/unit-ubuntu-repository-cache-0/charm/hooks/ubuntu-repository-cache-sync
root 58472 0.0 0.0 59644 3032 ? S Apr10 0:00 \_ CRON
www-sync 58474 0.0 0.0 4448 800 ? Ss Apr10 0:00 | \_ /bin/sh -c juju-run ubuntu-repository-cache/0 /var/lib/juju/agents/unit-ubuntu-repository-cache-0/charm/hooks/ubuntu-repository-cache-sync
www-sync 58476 0.0 0.0 236868 26572 ? Sl Apr10 0:02 | \_ juju-run ubuntu-repository-cache/0 /var/lib/juju/agents/unit-ubuntu-repository-cache-0/charm/hooks/ubuntu-repository-cache-sync
root 61144 0.0 0.0 59644 3032 ? S Apr10 0:00 \_ CRON
www-sync 61146 0.0 0.0 4448 804 ? Ss Apr10 0:00 | \_ /bin/sh -c juju-run ubuntu-repository-cache/0 /var/lib/juju/agents/unit-ubuntu-repository-cache-0/charm/hooks/ubuntu-repository-cache-sync
www-sync 61147 0.0 0.0 236868 26552 ? Sl Apr10 0:02 | \_ juju-run ubuntu-repository-cache/0 /var/lib/juju/agents/unit-ubuntu-repository-cache-0/charm/hooks/ubuntu-repository-cache-sync
root 6831 0.0 0.0 59644 3032 ? S Apr10 0:00 \_ CRON
www-sync 6832 0.0 0.0 4448 784 ? Ss Apr10 0:00 | \_ /bin/sh -c juju-run ubuntu-repository-cache/0 /var/lib/juju/agents/unit-ubuntu-repository-cache-0/charm/hooks/ubuntu-repository-cache-sync
www-sync 6833 0.0 0.1 236868 29108 ? Sl Apr10 0:02 | \_ juju-run ubuntu-repository-cache/0 /var/lib/juju/agents/unit-ubuntu-repository-cache-0/charm/hooks/ubuntu-repository-cache-sync
root 10313 0.0 0.0 59644 3032 ? S Apr10 0:00 \_ CRON
www-sync 10315 0.0 0.0 4448 764 ? Ss Apr10 0:00 | \_ /bin/sh -c juju-run ubuntu-repository-cache/0 /var/lib/juju/agents/unit-ubuntu-repository-cache-0/charm/hooks/ubuntu-repository-cache-sync
www-sync 10317 0.0 0.1 236868 30872 ? Sl Apr10 0:02 | \_ juju-run ubuntu-repository-cache/0 /var/lib/juju/agents/unit-ubuntu-repository-cache-0/charm/hooks/ubuntu-repository-cache-sync
root 15906 0.0 0.0 59644 3032 ? S Apr10 0:00 \_ CRON
www-sync 15907 0.0 0.0 4448 852 ? Ss Apr10 0:00 | \_ /bin/sh -c juju-run ubuntu-repository-cache/0 /var/lib/juju/agents/unit-ubuntu-repository-cache-0/charm/hooks/ubuntu-repository-cache-sync
www-sync 15908 0.0 0.0 236868 28620 ? Sl Apr10 0:02 | \_ juju-run ubuntu-repository-cache/0 /var/lib/juju/agents/unit-ubuntu-repository-cache-0/charm/hooks/ubuntu-repository-cache-sync
root 24153 0.0 0.0 59644 3032 ? S Apr10 0:00 \_ CRON
www-sync 24155 0.0 0.0 4448 688 ? Ss Apr10 0:00 | \_ /bin/sh -...

[David Lawson (deej)]

A timeout is probably an excellent idea but I think we should combine this with adding locking (I've filed a separate bug for that) and set the timeout to be quite long. Particularly with distant regions, it's entirely possible to hit a slow mirror and take an hour or so to sync. We either need to allow that case or make sure that hitting the timeout doesn't break the mirror, which currently it does.