issue reading netcdf4 files with nfs v3 on ubuntu 20.04

[Balagopal Pillai]

I was upgrading a small 16.04 hpc cluster to 20.04 and noticed the following. An nfs v3 mount causes netcdf4 file reads to fail. Nfs v4 mounts from the nfs server running 20.04 works fine. Netcdf3 files work fine with both nfs v3 and v4 mounts. I checked a 16.04 setup and 18.04 setup and they have no issues reading netcdf4 files with nfs v3 mounts.

     To reproduce this issue, please try any netcdf4 file and try to read the headers like this -
ncdump -s -h <filename.nc>. I needed v3 mounts, as I am using posix acls with zfs on the nfs server. So this is an issue particular to nfs server on 20.04 and I would have to move to 18.04 and stay there for a lots of other 16.04 servers and clusters till a solution is found to this issue. Thanks.

[Ubuntu Foundations Team Bug Bot (crichton)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1903974/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

[Rafael David Tinoco (rafaeldtinoco)]

Hello,

Isn't that happening because of differences in locking ?

https://rdrr.io/bioc/rhdf5/man/h5fileLocking.html

In NFSv4 file locking is part of the NFS protocol (different than NFSv3
that needed a network lock manager (NLM).

Perhaps you could follow instructions from RHDF5 and document here your
discoveries ?

Thanks

[Balagopal Pillai (balagopalp)]

The test is done with a netcdf4 file from here - http://rain.ucis.dal.ca/ctm/GEOS_0.5x0.625_NA.d/MERRA2/1980/01/MERRA2.19800101.A1.05x0625.NA.nc4

From 20.04 nfs server

v3 mount test - mount -t nfs -o vers=3 192.168.1.225:/nfs /tmp/test/

ncdump -h -s MERRA2.19800101.A1.05x0625.NA.nc4|tail -2
ncdump: MERRA2.19800101.A1.05x0625.NA.nc4: MERRA2.19800101.A1.05x0625.NA.nc4: NetCDF: HDF error

v4 mount test - mount -t nfs -o vers=4 192.168.1.225:/nfs /tmp/test/

ncdump -h -s MERRA2.19800101.A1.05x0625.NA.nc4|tail -2
  :_Format = "netCDF-4" ;
}

It works fine.

From 18.04 nfs server -

v3 mount test -

ncdump -h -s MERRA2.19800101.A1.05x0625.NA.nc4 |tail -2
  :_Format = "netCDF-4" ;
}

      Works fine as expected. So something has changed between 18.04 and 20.04 that breaks v3 mounts it seems. Thanks.

[Balagopal Pillai (balagopalp)]

Thanks Rafael! Disabling hdf5 file locking solved it.

 export HDF5_USE_FILE_LOCKING=FALSE

 I didn't have to do this for v3 mounts before. The only time I remember of users having to use this is when using xarray and dask to open the netcdf file in parallel. Anyhow I am happy that it works now. Thanks a lot for your help. Please see the results below -

export HDF5_USE_FILE_LOCKING=TRUE
ncdump -h -s MERRA2.19800101.A1.05x0625.NA.nc4 |tail -2
ncdump: MERRA2.19800101.A1.05x0625.NA.nc4: MERRA2.19800101.A1.05x0625.NA.nc4: NetCDF: HDF error

export HDF5_USE_FILE_LOCKING=FALSE
ncdump -h -s MERRA2.19800101.A1.05x0625.NA.nc4 |tail -2
  :_Format = "netCDF-4"

[Rafael David Tinoco (rafaeldtinoco)]

Nice to read that! Thanks for the feedback.