"Operation not permitted" from chmod when trying to create branch on ntfs-3g or fat filesystem

[David Jaša]

Bazaar can't init repository on ntfs-3g (fuse) filesystem:

david@cihla:/mnt/data/skola$ bzr init
bzr: ERROR: Transport error: [Errno 1] Operation not permitted: '/mnt/data/skola/.bzr' [Errno 1] Operation not permitted: '/mnt/data/skola/.bzr'
david@cihla:/mnt/data/skola$ bzr init
bzr: ERROR: File exists: u'/mnt/data/skola/.bzr': [Errno 17] File exists: '/mnt/data/skola/.bzr'
david@cihla:/mnt/data/skola$ bzr version
Bazaar (bzr) 1.0.0
  Python interpreter: /usr/bin/python 2.5.2.alpha.0
  Python standard library: /usr/lib/python2.5
  bzrlib: /usr/lib/python2.5/site-packages/bzrlib
  Bazaar configuration: /home/david/.bazaar
  Bazaar log file: /home/david/.bzr.log

[Aaron Bentley (abentley)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

David Jaša wrote:
> Public bug reported:
>
> Bazaar can't init repository on ntfs-3g (fuse) filesystem:
>
> david@cihla:/mnt/data/skola$ bzr init
> bzr: ERROR: Transport error: [Errno 1] Operation not permitted: '/mnt/data/skola/.bzr' [Errno 1] Operation not permitted: '/mnt/data/skola/.bzr'
> david@cihla:/mnt/data/skola$ bzr init

The root cause is that ntfs-3g gives "Operation not permitted" for chmod.

Aaron
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHrzdL0F+nu1YWqI0RAhLEAJ9HW3EIQkib+JgpHZtspdwqCJDAHQCfd1FC
PC3JfJvnoPLtpeF2tB/i30U=
=7T15
-----END PGP SIGNATURE-----

[David Jaša (dejv)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Aaron Bentley napsal(a):
|
| The root cause is that ntfs-3g gives "Operation not permitted" for
| chmod.
|
| Aaron

Does bzr's design allow to recover from not being able to chmod
files/directories?

David

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHr1sI5ckLLQrmzAYRAsUYAJ49xC2ZmvBTsnB2wMcgTYil016h+wCfVnIB
YRBvfUx4eRxt3QyGZ/CFg0Y=
=XDFA
-----END PGP SIGNATURE-----

[Robert Collins (lifeless)]

On Sun, 2008-02-10 at 20:14 +0000, David Jaša wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Aaron Bentley napsal(a):
> |
> | The root cause is that ntfs-3g gives "Operation not permitted" for
> | chmod.
> |
> | Aaron
>
> Does bzr's design allow to recover from not being able to chmod
> files/directories?

Yes, of course. But generally a chmod failures means 'no permissions
here' so we have been treating it as a fatal error.

-Rob
--
GPG key available at: <http://www.robertcollins.net/keys.txt>.

[David Jaša (dejv)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Robert Collins napsal(a):
| Yes, of course. But generally a chmod failures means 'no permissions
| here' so we have been treating it as a fatal error.
|
| -Rob

It would be nice if bzr had an option to ignore chmod failure on some
branch then. Managing documents with bzr on shared filesystem on
dual-boot machine is sensible scenario, isn't it?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHr5Lg5ckLLQrmzAYRAvU1AJ4tSkEAeLhjTuMFzLW8w9e29090KwCePC51
fdgfEzsyWOiyXF2bILJMN1k=
=ARKp
-----END PGP SIGNATURE-----

[Mander (arwintcher)]

I just wanted to chime in and say that's exactly what my setup is. For complicated reasons I used the Wubi installer on my Vista machine, so at the moment everything is ntfs. I'm new to both Linux and version control but I want to use it to manage my PhD work. Is there a workaround for this?

[Misha Dorman (mishad-work)]

I too would like to be able use bzr to manage repos and branches and access them from both "sides" of my dual-boot setup.

[Hg/Mercurial has the same problem, btw.]

What is the cleanest way to fix this? A simple flag (ideally remembered in the branch) to ignore such errors and continue?

A variant of transports/local.py that handles the exceptions differently, which can be specified via a URI scheme?

Why is failure-to-chmod treated as a fatal error? Could it be made non-fatal (warn-and-continue) in all cases, or would that introduce data loss/security problem risks?

Thinking more laterally:

Is there an alternative way of accessing NTFS shares from Linux (rather than ntfs-3g) which allows writing and supports chmod?

Do any of the ext2/3 IFS solutions for Windows provide read-write support such that they could be used (mounting an "alien" ext2 fs via an IFS in Windows, and using bzr.exe against the IFS)?

[John A Meinel (jameinel)]

So... I'm a bit curious if you couldn't mount with the right options and current umask so that the chmod isn't necessary, or is, at least, a no-op.
That would mean that the file mask should be exactly the directory mask, only without the executable bit (0777, 0666, umask 0000 for example, or 0775 0664 and a umask of 0002).

I certainly see this code in "local.py":
st = os.fstat(fd)
if mode is not None and mode != S_IMODE(st.st_mode):
    # Because of umask, we may still need to chmod the file.
    # But in the general case, we won't have to
    os.chmod(abspath, mode)

Alternatively, bzrdir._dir_mode and _file_mode set the modes that we want to achieve, and they can be set to "None" to indicate we don't want to do any chmodding. At the moment, this is triggered when "transport.stat('directory')" raises TransportNotPossible.

So this simple plugin could do the trick:

from bzrlib import transport, errors
from bzrlib.transport import local

class AltLocal(local.LocalTransport):

  def stat(self, relpath):
    raise errors.TransportNotPossible()

transport.register_transport('file:///', AltLocal)

Put the above code into a file like ~/.bazaar/plugins/no_local_stat.py

And see if ntfs-3g starts working for you. There are certainly other things you could do, like test "self.base" for being inside the mount point, etc.

[Misha Dorman (mishad-work)]

John,

Thanks for looking at this.

I tried the plugin idea, but it didn't work. (Very nice idea though -- its for this sort of thing that I like bzr!) The plugin was found -- if I insert a syntax error I get a corresponding error -- but it didn't seem to have any effect.

Based on suggestions elsewhere, I tried mounting with the ntfs-3g -o silent option. This works (it causes chmod/chown to suppress their own errors) BUT only if _none_ of the uid, gid or umask options are specified. [Apparently the problem is that ntfs-3g is OK but FUSE raises the error; see http://forum.ntfs-3g.org/viewtopic.php?p=896&sid=c20dc858ef417d7c223e95603e216a0a]. Unfortunately, I'm using gid to allow non-root users to have access to the mounted drive.

I also tried the "make the fmask/dmask/umask match" idea: setting dmask=770, umask=660, umask=007 (I want no access except to root and members of the disk-access group). Unfortunately that also didn't work, though I'm not sure exactly why.

I feel so close, but not quite there .... :-(

[Savvas Radevic (medigeek)]

Note: this bug might be connected to bug #240429
https://bugs.launchpad.net/bzr/+bug/240429

(As Misha commented: "[Hg/Mercurial has the same problem, btw.]")

[Savvas Radevic (medigeek)]

oops, I meant: https://bugs.launchpad.net/bugs/240429

[Savvas Radevic (medigeek)]

Misha have you tried the last comment?
http://forum.ntfs-3g.org/viewtopic.php?p=3341#p3341

Quote:
A little late, but nonetheless...
I was able to overcome problems with preserving times:
Code:
cp: preserving times for 'foo': Operation not permitted

by adding 'uid=' option in /etc/fstab, like this:

Code:
# /dev/sda3
UUID=E486D32886D2FA2A /media/files ntfs-3g defaults,umask=007,gid=46,uid=1000 0 1

[Vernon Cole (vernondcole)]

This is also related to bug 248333 where the X bit gets fouled up on an NTSF volume.
Is there a way that bzr can detect that it is running on an NTSF volume and take special steps?

[Martin Pool (mbp)]

On 22 April 2010 02:42, Vernon Cole <email address hidden> wrote:
> This is also related to bug 248333 where the X bit gets fouled up on an NTSF volume.
> Is there a way that bzr can detect that it is running on an NTSF volume and take special steps?

Yes, it should be possible to detect this, by any of

1- calling fstatfs
2- a per-location user configuration option
3- testing whether the x-bit can be set and read back

--
Martin <http://launchpad.net/~mbp/>

[nairbv (nairbv)]

This might be related to #199654.

[Martin Pool (mbp)]

I've just been testing this on a linux vfat mount, and these operations work:
 - checkout --lightweight
 - branch
 - init
 - st on the resulting branch (though it does demonstrate bug 248333)

[Martin Pool (mbp)]

I can't reproduce this on ntfs either, but misha's comments in #7 are suggestive - perhaps it's something about gvfs, perhaps something about having the device not owned by the user in question.

[Martin Pool (mbp)]

I also can't reproduce this on a gvfs-mounted ntfs filesystem, on Ubuntu Oneiric: branching onto there from bzr just works. So, perhaps something has changed in the filesystem implementation so that chmod no longer fails?

If anyone still can reproduce this, please let us know:
 * the traceback from ~/.bzr.log
 * how this filesystem is mounted
 * the output of 'mount'
 * what OS and version you're using

[Sebastian Korczyk (sebastian-korczyk)]

I think I can reproduce similiar thing, I've tried to import project from launchpad, using eclipse plugin, hovewer it was failing with unexpected error, I've checked and same happens when using bzr from command line. When running command below on ntfs system it is failing, when run it on ext4 filesystem, everything was fine:

When running: bzr branch lp:civ4stratoverlay civ.white

after entering passwd for RSA key I'm receiving such error:
bzr: ERROR: [Errno 1] Operacja niedozwolona: '/media/win/workspace/civ3/civ.white/.bzr/checkout/limbo/new-2/Config/init.xml'

Where "Operacja niedozwolona" means - illegal operation.

Below bzr.log (first when command was succesfull on Linux filesystem, and later when it has failed on)

System Debian Wheezy: 3.0.0-1-686-pae

fstab: /dev/sda4 /media/win ntfs defaults,umask=000,gid=1000,noatime 0 0

Mount output for this partition:
/dev/sda4 on /media/win type fuseblk (rw,nosuid,nodev,noatime,user_id=0,group_id=0,default_permissions,allow_other,blksize=4096)

nie 2012-08-19 00:13:19 +0200
0.048 bazaar version: 2.6.0dev2
0.048 bzr arguments: [u'branch', u'lp:civ4stratoverlay', u'civ.white']
0.052 looking for plugins in /home/logan/.bazaar/plugins
0.052 looking for plugins in /usr/lib/python2.7/dist-packages/bzrlib/plugins
0.065 encoding stdout as sys.stdout encoding 'UTF-8'
0.124 ssh implementation is OpenSSH
10.176 creating repository in file:///home/logan/test/civ2/civ.white/.bzr/.
10.346 Using fetch logic to copy between RemoteRepository(bzr+ssh://bazaar.launchpad.net/%2Bbranch/civ4stratoverlay/.bzr/)(RemoteRepositoryFormat(_network_name='Bazaar pack repository format 1 (needs bzr 0.92)\n')) and KnitPackRepository('file:///home/logan/test/civ2/civ.white/.bzr/repository/')(RepositoryFormatKnitPack1())
10.354 fetching: <PendingAncestryResult heads:frozenset(['<email address hidden>']) repo:RemoteRepository(bzr+ssh://bazaar.launchpad.net/%2Bbranch/civ4stratoverlay/.bzr/)>
16.536 creating branch <bzrlib.branch.BzrBranchFormat6 object at 0x89857ec> in file:///home/logan/test/civ2/civ.white/
16.836 created new branch BzrBranch6(file:///home/logan/test/civ2/civ.white/)
16.842 trying to create missing lock '/home/logan/test/civ2/civ.white/.bzr/checkout/dirstate'
16.877 opening working tree '/home/logan/test/civ2/civ.white'
[26197] 2012-08-19 00:13:36.417 INFO: Branched 36 revisions.
17.002 Transferred: 353kB (20.9kB/s r:351kB w:1kB)
17.002 return code 0

nie 2012-08-19 00:15:36 +0200
0.048 bazaar version: 2.6.0dev2
0.048 bzr arguments: [u'branch', u'lp:civ4stratoverlay', u'civ.white']
0.052 looking for plugins in /home/logan/.bazaar/plugins
0.052 looking for plugins in /usr/lib/python2.7/dist-packages/bzrlib/plugins
0.065 encoding stdout as sys.stdout encoding 'UTF-8'
0.124 ssh implementation is OpenSSH
9.957 ignore error on chmod of u'/media/win/workspace/civ3/civ.white/.bzr': OSError(1, 'Operacja niedozwolona')
9.957 ignore error on chmod of u'/media/win/workspace/civ3/civ.white/.bzr/branch-lock': OSError(1, 'Operacja niedozwolona')
9.962 ignore error on chmod of u'/media/win/workspace/civ3/civ.white/.bzr/README.26213.azorek.sb...

[Martin Packman (gz)]

Sebastian, so your eventual error there is from utime rather than chmod. While setting the correct mtime on files isn't strictly required, bazaar does expect to be able to do that when creating the working tree.

So, I expect the following would not fail:

    bzr branch --no-tree lp:civ4stratoverlay civ.white

See bug 124795 on ntfs-3g for some discussion about this and pointers on how to fix your mount options.

[Sebastian Korczyk (sebastian-korczyk)]

Martin, I can confirm that when running command you had provided it has passed on ntfs filesystem. So I will go to discusion on bug 124795 that maybe message in such case should be more clear.

[Peter (fpeter)]

bzr init-repo sftp://xxxxxx/bzr/foo
bzr: ERROR: [Errno 13] Permission denied

the target was fat32 formatted.
a better error message needed.