sync with --link-dest=ubuntu_active/ doesn't always work

[Haw Loeung]

Hi,

As seen on more than one occasion, metadata sync'ed can fill up the disk. It looks like entire copies are sync'ed / stored per below:

| root@machine-1:/srv/ubuntu-repository-cache/apache/data# du -h --max-depth 1
| 9.0G ./ubuntu_2020-09-15_19:46:32_u1
| 23G ./ubuntu_2020-09-14_18:57:02_u1
| 23G ./ubuntu_2020-09-14_17:57:02_u1
| 54G .

| root@machine-1:/srv/ubuntu-repository-cache/apache/data# ls -l
| total 20
| drwxrwxr-x 7 www-sync www-data 4096 Sep 14 17:30 ubuntu_2020-09-14_17:57:02_u1
| -rw-r--r-- 1 root root 0 Sep 14 17:59 ubuntu_2020-09-14_17:57:02_u1_good
| drwxrwxr-x 7 www-sync www-data 4096 Sep 14 18:26 ubuntu_2020-09-14_18:57:02_u1
| -rw-r--r-- 1 root root 0 Sep 14 19:06 ubuntu_2020-09-14_18:57:02_u1_good
| drwxrwxr-x 7 www-sync www-data 4096 Sep 15 18:25 ubuntu_2020-09-15_19:46:32_u1
| lrwxrwxrwx 1 root root 75 Sep 14 18:56 ubuntu_active -> /srv/ubuntu-repository-cache/apache/data/ubuntu_2020-09-14_17:57:02_u1_good
| lrwxrwxrwx 1 root root 75 Sep 14 18:56 ubuntu_next -> /srv/ubuntu-repository-cache/apache/data/ubuntu_2020-09-14_17:57:02_u1_good

We should figure out what's causing this and fix it.

[Junien F (axino)]

Was there a recent leadership change ? machine-lock.log will tell you.

When this happens, please also "du" each directory independently so that we get a better picture of what's going on (since hardlinks are at play).