Comment 1 for bug 1853174

######### On the NFS client side ###########

# link(): SVr4, 4.3BSD, POSIX.1-2001 (but see NOTES), POSIX.1-2008.

int link(const char *oldpath, const char *newpath);

ERRORS:

EACCES Write access to the directory containing newpath is denied, or search permission is denied for one of the directories in the path prefix of oldpath or newpath.

Notes:

- Hard links, as created by link(), cannot span filesystems. Use symlink(2) if this is required.

Bugs:

On NFS filesystems, the return code may be wrong in case the NFS server performs the link creation and dies before it can say so. Use stat(2) to find out if the link got created.

----

So, unless ganesha deviates from POSIX.1-2008, OR the ganesha nfsd RPC thread crashes, EACCESS should be a result of either NEWPATH having bad permissions for NFS user OR there is an issue with directory (+x) permissions in ONE directory of EITHER oldpath or newpath (/aaa/bbb/cccc <- the full path I mean).

----

######### On the NFS SERVER side ###########

Considering the NFS server is giving us EACCESS, then...

With https://github.com/nfs-ganesha/nfs-ganesha/wiki/NFS-Ganesha-Architecture:

the EACCESS comes from the def_handle_ops abstraction (FSAL/default_methods.c): fsal_test_access() or fsal_obj_ops->test_access(). We also have the MDCACHE in between final fsal_test_access() and the caller.

The stacktrace:

fsal_print_access_by_acl()
  fsacl_check_access_acl()
    fsal_test_access()
      mdcache_test_access() -> comes from mdcache(ops->test_access()) call
        mdcache_handle_ops_init() -> initializes fsal_obj_ops for:
          mdcache_fsal_init() ops->test_access = mdcache_test_access;
            start_fsals()

would tell us the exact error if:

 if (!isFullDebug(COMPONENT_NFS_V4_ACL))
  return;

debug was enabled.

--------

For the fsal_test_access():

/* test_access
 * common (default) access check method for fsal_obj_handle objects.
 * NOTE: A fsal can replace this method with their own custom access
 * checker. If so and they wish to have an option to switch
 * between their custom and this one, it their test_access
 * method's responsibility to do that test and select this one.
 */

looks like a filesystem abstraction can replace the check method. fsal_test_access() is the DEFAULT FSAL check access but not the only one.

NOTE: I'm not sure what is the underlying FSAL you're using (ceph ?)

--------

From the FSAL abstraction layer:

/**
 * @brief Check access for a given user against a given object
 *
 * This function checks whether a given user is allowed to perform the
 * specified operations against the supplied file. The goal is to
 * allow filesystem specific semantics to be applied to cached
 * metadata.
 *
 * This method must read attributes and/or get them from a cache.
 *
 * @param[in] obj_hdl Handle to check
 * @param[in] access_type Access requested
 * @param[out] allowed Returned access that could be granted
 * @param[out] denied Returned access that would be granted
 * @param[in] owner_skip Skip test if op_ctx->creds is owner
 *
 * @return FSAL status.
 */
  fsal_status_t (*test_access)(struct fsal_obj_handle *obj_hdl,
          fsal_accessflags_t access_type,
          fsal_accessflags_t *allowed,
          fsal_accessflags_t *denied,
          bool owner_skip);

Considering MDCACHE is calling the FSAL (abstraction layer) test_access method, now we have to understand why the underlying FSAL is giving us EACCESS.

--------

In vanilla upstream git tree I could only find 2 places implementing that function:

The (*test_access) API is ONLY implemented by the MDCACHE:

 ops->test_access = mdcache_test_access;

or the "default" FSAL function:

 .test_access = fsal_test_access, /* default is use common test */

So independently of a FSAL being used, the error must come either from MDCACHE (*test_access) layer or the FSAL (*test_access) layer.

--------

Since we already checked fsal_test_access, checking the mdcache_test_access:

/**
 * @brief Check access for a given user against a given object
 *
 * Currently, all FSALs use the default method. We call the default method
 * directly, so that the test uses cached attributes, rather than having the
 * lower level need to query attributes each call. This works as long as all
 * FSALs call the default method. This should be revisited if a FSAL wants to
 * override test_access().
 *
 * @note If @a owner_skip is provided, we test against the cached owner. This
 * is because doing a getattrs() potentially on each read and write (writes
 * invalidate cached attributes) is a huge performance hit. Eventually, finer
 * grained attribute validity would be a better solution
 *
 * @param[in] obj_hdl Handle to check
 * @param[in] access_type Access requested
 * @param[out] allowed Returned access that could be granted
 * @param[out] denied Returned access that would be granted
 * @param[in] owner_skip Skip test if op_ctx->creds is owner
 *
 * @return FSAL status.
 */
static fsal_status_t mdcache_test_access(struct fsal_obj_handle *obj_hdl,
      fsal_accessflags_t access_type,
      fsal_accessflags_t *allowed,
      fsal_accessflags_t *denied,
      bool owner_skip)
{
 mdcache_entry_t *entry =
  container_of(obj_hdl, mdcache_entry_t, obj_handle);

 if (owner_skip && entry->attrs.owner == op_ctx->creds->caller_uid)
  return fsalstat(ERR_FSAL_NO_ERROR, 0);

 return fsal_test_access(obj_hdl, access_type, allowed, denied,
    owner_skip);
}

So it is very likely that we're getting the access error in the default fsal_test_access() function, not in the mdcache layer.

--------

We're likely getting an error from here (probability based on code):

 if (IS_FSAL_ACE4_REQ(access_type) ||
     (attrs.acl != NULL && IS_FSAL_ACE4_MASK_VALID(access_type))) {
  status = fsal_check_access_acl(op_ctx->creds,
            FSAL_ACE4_MASK(access_type),
            allowed, denied, &attrs);

 } else { /* fall back to use mode to check access. */
  status = fsal_check_access_no_acl(op_ctx->creds,
        FSAL_MODE_MASK(access_type),
        allowed, denied, &attrs);
 }

Depending whether we're using NFSv4, using v4 ACL list, or other, using mode bits only. AS you already checked regular directory permissions (hopefully from full paths of oldpath and newpath of the link() call with EACCESS error)..

I'll continue with v4 case only:

--------

/**
 * @brief Check access using v4 ACL list
 *
 * @param[in] creds
 * @param[in] v4mask
 * @param[in] allowed
 * @param[in] denied
 * @param[in] p_object_attributes
 *
 * @return ERR_FSAL_NO_ERROR, ERR_FSAL_ACCESS, or ERR_FSAL_NO_ACE
 */

static fsal_status_t fsal_check_access_acl(struct user_cred *creds,
        fsal_aceperm_t v4mask,
        fsal_accessflags_t *allowed,
        fsal_accessflags_t *denied,
        struct attrlist *p_object_attributes)

There are some cases where ERR_FSAL_ACCESS will be returned here AND posix2fsal_error() function tells us that the ERR_FSAL_ACCESS error code is:

 case EACCES:
  return ERR_FSAL_ACCESS;

EACCESS.

 LogFullDebug(COMPONENT_NFS_V4_ACL,
       "file acl=%p, file uid=%u, file gid=%u, ", pacl, uid, gid);

TODO: Enabling Debug will give you the exact acl/uid/gid for the broken access. From the acl you will get a the ACES which is the FSAL ACE giving access type, erm, uid, gid, so you will have the exact EACCESS issue.

--------

- test_access()
 - fsal_access():
  - check_open_permission()
  - file_To_Fattr()
  - fsal_link() - OUR CASE
  - fsal_lookup()
  - fsal_readdir()
  - nfs3_read()
  - nfs4_read()
  - open2_by_name()

(*test_access) is called by:

- allocate_deallocate()

- check_open_permission() - check permissions when opening a file

- fsal_check_setattr_perms()
- fsal_remove_access()
- fsal_rename_access()

- nfs_access_op() - perform version independent ACCESS: checks access_mask to use
- nfs3_read() - NFSPROC3_READ function
- nfs3_write() - NFSPROC3_WRITE function

- nfs4_op_write() - NFS4_OP_WRITE function
- nfs4_read() - NFS4_OP_READ function
- nfs4_readdir_callback()

--------

/**
 *
 * @brief Links a new name to a file
 *
 * This function hard links a new name to an existing file.
 *
 * @param[in] obj The file to which to add the new name. Must
 * not be a directory.
 * @param[in] dest_dir The directory in which to create the new name
 * @param[in] name The new name to add to the file
 *
 * @return FSAL status
 * in destination.
 */
fsal_status_t fsal_link(struct fsal_obj_handle *obj,
   struct fsal_obj_handle *dest_dir,
   const char *name)
{

It does 3 initial checks (can't be dir and must be in same FS).

Then it does:

 if (!op_ctx->fsal_export->exp_ops.fs_supports(
   op_ctx->fsal_export,
   fso_link_supports_permission_checks)) {
  status = fsal_access(dest_dir,
   FSAL_MODE_MASK_SET(FSAL_W_OK) |
   FSAL_MODE_MASK_SET(FSAL_X_OK) |
   FSAL_ACE4_MASK_SET(FSAL_ACE_PERM_EXECUTE) |
   FSAL_ACE4_MASK_SET(FSAL_ACE_PERM_ADD_FILE));

  if (FSAL_IS_ERROR(status))
   return status;
 }

which is calling fsal_access() for the dest dir.

--------

Our execution path is likely:

nfs3_link() -> fsal_link()

  LogFullDebug(COMPONENT_NFSPROTO, "failed link: fsal_status=%s",fsal_err_txt(fsal_status));

or

nfs4_op_link() -> fsal_link()

will call nfs4_Errno_verbose() with:

  case ERR_FSAL_ACCESS:
    nfserror = NFS4ERR_ACCESS;

--------