Disk usage calculation is very slow with NFS

[Yaju Mahida]

https://mahara.org/interaction/forum/topic.php?id=8585&offset=0&limit=10#post34233

We experienced a very strange issue while upgrading from Mahara 18.10.3 to 19.10.1. It was stuck at cron_site_data_daily() in the htdocs\lib\db\upgrade.php, taking forever to calculate disk usage and failing the upgrade process after 10 minutes. It was opening the database transaction and waiting forever for dirsize function to finish.

    if ($oldversion < 2019011500) {

        log_debug('run cron_site_data_daily function to update data with new chartjs structure');

        cron_site_data_daily();

    }

The cron job call the same function and it is stuck for a very long time due to the same reasons.

We believe this is an NFS file locking issue. The data root is mounted on an NFS volume, and it is around 500 GB. Even the Linux du utility takes around 20 minutes to calculate the size of this NFS mount point.

We fixed this issue by swapping the dirsize function (htdocs\lib\function.dirsize.php) with a custom function which utilizes disk_total_space and disk_free_space.

function usqdirsize($path) {

    $disktotalspace = disk_total_space($path);

    $diskfreespace = disk_free_space($path);

    $diskusage = $disktotalspace - $diskfreespace;

    return $diskusage;

}

Question: do we need to calculate the size of a directory by iterating its contents? If not happy to commit this patch with a bug report.

[Mahara Bot (dev-mahara)]

Patch for "master" branch: https://reviews.mahara.org/10780

[Yaju Mahida (yvm)]

Hi Roberts,

Yesterday, replied to your comments and still, it shows in the draft mode!

We have a dedicated NFS mount and we are calculating disk size instead of going through all directories as this takes forever.

For some, it might be possible that they are using shared directories and this will not give correct disk usage.

Just tested using Robert's GetDirectorySize logic and same behaviour
[INF] 64 (lib/cron.php:549) Skipping long-running cron job core_cron_site_data_daily (Wed, 26 Feb 2020 11:57:01 +1000).

[Yaju Mahida (yvm)]

For some reasons, my reply is in draft mode since yesterday.

https://reviews.mahara.org/#/c/10780/4/htdocs/lib/function.dirsize.php

[Mahara Bot (dev-mahara)]

Patch for "main" branch: https://reviews.mahara.org/12078

[Mahara Bot (dev-mahara)]

Reviewed: https://reviews.mahara.org/12078
Committed: https://git.mahara.org/mahara/mahara/commit/9f7756f3a3b9ce7502e0a0bde025e798616c8d73
Submitter: Robert Lyon (<email address hidden>)
Branch: main

commit 9f7756f3a3b9ce7502e0a0bde025e798616c8d73
Author: Yaju Mahida <email address hidden>
Date: Wed Feb 26 09:51:59 2020 +1000

Bug 1864760 Disk usage calculation is very slow with NFS

Change-Id: Ic6ad6fe9cf3e9183182c28426bdc3b3a3c37a571