The more I dig into this....the more I suspect kernel related. Basically an open() is done to an NFS mounted file with O_SYNC, followed by a write(). On the RHEL machine, the NFS write is done correctly...with a FILE_SYNC sent to the server, on the Ubuntu client, the NFS write is sent UNSTABLE, i.e. it doesn't wait to receive notification from the server that the data was sync'd.
The more I dig into this....the more I suspect kernel related. Basically an open() is done to an NFS mounted file with O_SYNC, followed by a write(). On the RHEL machine, the NFS write is done correctly...with a FILE_SYNC sent to the server, on the Ubuntu client, the NFS write is sent UNSTABLE, i.e. it doesn't wait to receive notification from the server that the data was sync'd.