I took a day and looked through the code/tests for GlusterFS and couldn't figure out how to reliably trigger the crash. For my system, I experienced brick crashes about once per hour, but only with specific volumes. For instance, the volumes which I serve with NFS Ganesha never triggered this crash (they triggered a different **crash with NFS Ganesha, but not within GlusterFS). Only a locally mounted volume which backs a Minio instance (an S3 API compatible server) used by Restic clients (an incremental backup system) triggered this crash. I attempted to replicate the workload by running various file system benchmarking tools within their own user namespace (i.e. lots of small file creations/deletions) but that never triggered the crash either. I've been running the above patch since 2024-05-06 and haven't experienced a single crash. Unfortunately I'm at a loss for what else to try.
FWIW I filed this because my experience upgrading my NAS machines from 22.04 to 24.04 was extremely poor due to issues with GlusterFS and NFS Ganesha.
Hi Bryce!
I really appreciate you getting this submitted.
I took a day and looked through the code/tests for GlusterFS and couldn't figure out how to reliably trigger the crash. For my system, I experienced brick crashes about once per hour, but only with specific volumes. For instance, the volumes which I serve with NFS Ganesha never triggered this crash (they triggered a different **crash with NFS Ganesha, but not within GlusterFS). Only a locally mounted volume which backs a Minio instance (an S3 API compatible server) used by Restic clients (an incremental backup system) triggered this crash. I attempted to replicate the workload by running various file system benchmarking tools within their own user namespace (i.e. lots of small file creations/ deletions) but that never triggered the crash either. I've been running the above patch since 2024-05-06 and haven't experienced a single crash. Unfortunately I'm at a loss for what else to try.
FWIW I filed this because my experience upgrading my NAS machines from 22.04 to 24.04 was extremely poor due to issues with GlusterFS and NFS Ganesha.
**https:/ /bugs.launchpad .net/ubuntu/ +source/ nfs-ganesha/ +bug/2065856