Branches with symlinks can't be checked out on Windows

[Nicholas Allen]

Windows does not support symlinks but there is a plugin for bzr that emulates this behavior. If the plugin is installed then this works. I think this should be a core feature of bzr and not a plugin. Also displaying a stack trace is not a useful error message for the user so even if it is decided to keep it as a plugin this is not the correct behavior. When checking out a branch that uses symlinks on Windows the following exception is raised:

bzr branch http://panoramicfeedback.com/opensource/bzr/be BugsEverywhere
.................................................................................................................................................................................................................................................................................................................................................................................................
bzr: ERROR: exceptions.AttributeError: 'module' object has no attribute 'symlink'

Traceback (most recent call last):
 File "bzrlib\commands.pyc", line 626, in run_bzr_catch_errors
 File "bzrlib\commands.pyc", line 588, in run_bzr
 File "bzrlib\commands.pyc", line 292, in run_argv_aliases
 File "bzrlib\builtins.pyc", line 756, in run
 File "bzrlib\bzrdir.pyc", line 688, in sprout
 File "bzrlib\bzrdir.pyc", line 914, in create_workingtree
 File "bzrlib\workingtree.pyc", line 2113, in initialize
 File "bzrlib\transform.pyc", line 1008, in build_tree
 File "bzrlib\transform.pyc", line 1093, in new_by_entry
 File "bzrlib\transform.pyc", line 904, in new_symlink
 File "bzrlib\transform.pyc", line 318, in create_symlink
AttributeError: 'module' object has no attribute 'symlink'

bzr 0.13.0candidate1 on python 2.4.4.final.0 (win32)
arguments: ['c:\\Programme\\Bazaar\\bzr.exe', 'branch', 'http://panoramicfeedback.com/opensource/bzr/be', 'BugsEverywhere']

** please send this report to <email address hidden>

nia@client0055 /d
$ bzr version
Bazaar (bzr) 0.13.0candidate1
Using python interpreter: c:\Programme\Bazaar\bzr.exe
Using python standard library: c:\Programme\Bazaar\lib\library.zip
Using bzrlib: c:\Programme\Bazaar\lib\library.zip\bzrlib

[Jelmer Vernooij (jelmer)]

  status confirmed

I agree, the least we should do is give a proper error.

--
Jelmer Vernooij <email address hidden> - http://samba.org/~jelmer/

[Alexander Belchenko (bialix)]

Workaround for this bug (report clear error instead of traceback) was merged to bzr.dev as revno.3020.

About core support of fake symlink on Windows: pure python implementation is ready but walkdirs almost 80% slower on FAT32 with it. I have idea how improve performance with C-extension. I expecting to finish this work after 1.0 release.

[Aaron Bentley (abentley)]

Is high priority still appropriate for this bug, now that there's a clear diagnostic?

[Martin Pool (mbp)]

Aaron, I think medium is ok.

[anatoly techtonik (techtonik)]

> bzr checkout --lightweight --hardlink lp:update-manager
Connected (version 2.0, client Twisted)
Authentication (publickey) successful!
Secsh channel 1 opened.
bzr: ERROR: Unable to create symlink 'AutoUpgradeTester/DistUpgrade' on this platform

D:\.src\python>bzr version
Bazaar (bzr) 1.14.1
  ...
  Bazaar log file: C:\Documents and Settings\Quest\Мои документы\.bzr.log

...
17.985 Traceback (most recent call last):
  File "bzrlib\commands.pyo", line 727, in exception_to_return_code
  File "bzrlib\commands.pyo", line 922, in run_bzr
  File "bzrlib\commands.pyo", line 559, in run_argv_aliases
  File "bzrlib\builtins.pyo", line 1209, in run
  File "bzrlib\branch.pyo", line 1156, in create_checkout
  File "bzrlib\bzrdir.pyo", line 1565, in create_workingtree
  File "bzrlib\workingtree_4.pyo", line 1441, in initialize
  File "bzrlib\transform.pyo", line 2035, in build_tree
  File "bzrlib\transform.pyo", line 2121, in _build_tree
  File "bzrlib\transform.pyo", line 2269, in new_by_entry
  File "bzrlib\transform.pyo", line 980, in new_symlink
  File "bzrlib\transform.pyo", line 428, in create_symlink
UnableCreateSymlink: Unable to create symlink 'AutoUpgradeTester/DistUpgrade' on this platform

17.985 return code 3

Windows XP, NTFS

[Stephan Ruegamer (sadig)]

Hi Guys,

just tried to branch launchpadlib on windows...and bzr 1.5 tells me:

bzr branch lp:launchpadlib
You have not informed bzr of your Launchpad ID, and you must do this to
write to Launchpad or access private data. See "bzr help launchpad-login".
bzr: ERROR: Unable to create symlink 'bootstrap.py' on this platform

BZR is using py2.5 because some of the needed deps are not available for py2.6

Regards,

\sh

[Robert Collins (lifeless)]

On Thu, 2009-06-11 at 09:21 +0000, Stephan Hermann wrote:
> bzr: ERROR: Unable to create symlink 'bootstrap.py' on this platform

This is the key: contact the launchpadlib developers and ask them to not
use symlinks.

-Rob

[Johannes Sasongko (sjohannes)]

While waiting for someone to come up with the proper fix, could there be a switch to simply skip symlinks?

Something like an --ignore-symlinks option to bzr branch/checkout would be very useful, I think. I'm not really worried about other operations, as long as I can at least get the data. Or is there any other way to do this that I'm missing?

[THirsch (thirsch)]

I would love to have an option like --ignore-symlinks as well to allow at least for a checkout/branch that I can work on (with knowledge about the missing symlinks)

Second: On Vista and Windows7 there IS support for real symbolic links.
There is a great tool (with some documentation) out there, that is freely available (ln.exe and LinkShellExtension at http://www.schinagl.priv.at/) and the author (Hermann Schinagl - Hermann.Schinagl (at) gmx.net) may be able to direct you to some MSDN or other source of information (or even his source code....)
Sorry, I am not a programmer and could not lend a helping hand.
All your current work is greatly appreciated (I love bazaar).
Please keep on providing and enhancing this excellent work.
Thomas

[John A Meinel (jameinel)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

THirsch wrote:
> I would love to have an option like --ignore-symlinks as well to allow
> at least for a checkout/branch that I can work on (with knowledge about
> the missing symlinks)
>
> Second: On Vista and Windows7 there IS support for real symbolic links.
> There is a great tool (with some documentation) out there, that is freely available (ln.exe and LinkShellExtension at http://www.schinagl.priv.at/) and the author (Hermann Schinagl - Hermann.Schinagl (at) gmx.net) may be able to direct you to some MSDN or other source of information (or even his source code....)
> Sorry, I am not a programmer and could not lend a helping hand.
> All your current work is greatly appreciated (I love bazaar).
> Please keep on providing and enhancing this excellent work.
> Thomas
>
I'm pretty sure the Windows symlinks only support directories as
targets, but maybe that changed?

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAktNDRQACgkQJdeBCYSNAAMz0gCcDDCm4EGBQZCHaUIPsl3SuhoG
O5IAmQHQOk5vKL6YyQGBipiE7m1t1XV1
=OTHz
-----END PGP SIGNATURE-----

[THirsch (thirsch)]

> I'm pretty sure the Windows symlinks only support directories as
> targets, but maybe that changed?

No, it works with files as well (just tested with LinkShell Extension from http://www.schinagl.priv.at/)
That's, what seems to be new with Vista/Win7.
So bazaar could check for this versions and allow mimic/handling of linux symlinks on those platforms.
Regards
Thomas

[Markus Mayer (sunside)]

The windows internal command line tool for this job is mklink.exe, btw.

Markus

Am 13.01.2010 01:00, schrieb John A Meinel:
> THirsch wrote:
>> I would love to have an option like --ignore-symlinks as well to allow
>> at least for a checkout/branch that I can work on (with knowledge about
>> the missing symlinks)
>
>> Second: On Vista and Windows7 there IS support for real symbolic links.
>> There is a great tool (with some documentation) out there, that is freely available (ln.exe and LinkShellExtension at http://www.schinagl.priv.at/) and the author (Hermann Schinagl - Hermann.Schinagl (at) gmx.net) may be able to direct you to some MSDN or other source of information (or even his source code....)
>> Sorry, I am not a programmer and could not lend a helping hand.
>> All your current work is greatly appreciated (I love bazaar).
>> Please keep on providing and enhancing this excellent work.
>> Thomas
>
> I'm pretty sure the Windows symlinks only support directories as
> targets, but maybe that changed?
>
> John
> =:->
>

[THirsch (thirsch)]

Hi,
any advance with this issue?
Anyone working on it. or is it considered to be a 'nice to have'?
Best regards
Thomas

[Jason R. Coombs (jaraco)]

My project, jaraco.windows, adds symlink support to Python under Vista and later. Just easy_install jaraco.windows, then "__import__('jaraco.windows.filesystem').windows.filesystem.patch_os_module()". It adds 'symlink' to the os module, which may be sufficient for bazaar to support symlinks. One caveat is if you're creating a symlink to a directory that doesn't exist, you must supply a third parameter, target_is_directory=True, to the symlink call (otherwise Windows will assume the link is to a file).

[Alexander Belchenko (bialix)]

Jason R. Coombs пишет:
> My project, jaraco.windows, adds symlink support to Python under Vista
> and later.

And how it helps for Windows XP users?

[Martin Pool (mbp)]

On 11 April 2010 06:06, Jason R. Coombs <email address hidden> wrote:
> My project, jaraco.windows, adds symlink support to Python under Vista
> and later.  Just easy_install jaraco.windows, then
> "__import__('jaraco.windows.filesystem').windows.filesystem.patch_os_module()".
> It adds 'symlink' to the os module, which may be sufficient for bazaar
> to support symlinks. One caveat is if you're creating a symlink to a
> directory that doesn't exist, you must supply a third parameter,
> target_is_directory=True, to the symlink call (otherwise Windows will
> assume the link is to a file).

Thanks for that. What does it actually create on disk?

--
Martin <http://launchpad.net/~mbp/>

[GEM (nimp3)]

@jason
using on vista easy_install jaraco.windows,
__import__('jaraco.windows.filesystem').windows.filesystem.patch_os_module()

I have this error with python 2.5 :
SyntaxError: future feature print_function is not defined

to do for python >=2.6
import __future__

if hasattr(__future__, 'print_function'):
    from __future__ import print_function
else:
    raise ImportError('Python >= 2.6 is required')

but for python 2.5 ?

[GEM (nimp3)]

or a script for all version for "print" :

see: http://www.daniweb.com/code/snippet217214.html

# module cbprint
# (C) 2008 Gribouillis at www.daniweb.com
"""This module defines a Print function to use with python 2.x or 3.x.
Usage:
from cbprint import Print
Print("hello", "world")
It's interface is that of python 3.0's print. See
http://docs.python.org/3.0/library/functions.html?highlight=print#print
"""
__all__ = ["Print"]
import sys
try:
    Print = eval("print") # python 3.0 case
except SyntaxError:
    try:
        D = dict()
        exec("from __future__ import print_function\np=print", D)
        Print = D["p"] # 2.6 case
        del D
    except SyntaxError:
        del D
        def Print(*args, **kwd): # 2.4, 2.5, define our own Print function
            fout = kwd.get("file", sys.stdout)
            w = fout.write
            if args:
                w(str(args[0]))
                sep = kwd.get("sep", " ")
                for a in args[1:]:
                    w(sep)
                    w(str(a))
            w(kwd.get("end", "\n"))

[GEM (nimp3)]

for python 2.5

edit c:\python2.5\lib\site-packages\cbprint.py with the previous code.

extract jaraco from jaraco.windows-1.8-py2.5.egg in site-packages

delete jaraco.windows-1.8-py2.5.egg

edit site-packages\easy-install.pth, delete line :
./jaraco.windows-1.8-py2.5.egg

in site-packages\jaraco\windows\filesystem.py and site-packages\jaraco\windows\
change :
from __future__ import print_function
by
#from __future__ import print_function
from cbprint import Print

delete files filesystem.pyc and privilege.pyc

in site-packages\jaraco\windows\api\filesystem.py in the head of file add lines :
import ctypes
line 6, change :
BOOLEAN, LPWSTR, DWORD, LPVOID, HANDLE, FILETIME,
by
BOOLEAN, LPWSTR, DWORD, HANDLE, FILETIME, # I remove LPOID not define in py2.5
after line 10 from .....
add this code :
try:
    LPVOID = ctypes.wintypes.LPVOID
except AttributeError:
    # LPVOID wasn't defined in Py2.5, guess it was introduced in Py2.6
    LPVOID = ctypes.c_void_p

delete file site-packages\jaraco\windows\api\filesystem.pyc

in site-packages\jaraco\windows\reparse.py :
line 11 change :
LPOVERLAPPED = wintypes.LPVOID
by :
try:
    LPVOID = ctypes.wintypes.LPVOID
except AttributeError:
    # LPVOID wasn't defined in Py2.5, guess it was introduced in Py2.6
    LPVOID = ctypes.c_void_p
LPOVERLAPPED = LPVOID
lines 22 and 24, change :
wintypes.LPVOID,
by
LPVOID,

delete file :site-packages\jaraco\windows\reparse.pyc

open a console windowsand tape :
python (enter)
__import__('jaraco.windows.filesystem').windows.filesystem.patch_os_module() (enter, should work now without error, even for python 2.5)

[GEM (nimp3)]

this is for vista and 7.
sorry for xp.

[Parth Malwankar (parthm)]

Some wiki notes on this: http://wiki.bazaar.canonical.com/WindowsSymlinkSupport

[anatoly techtonik (techtonik)]

Please add --ignore-symlinks key.

[Chris Hecker (checker-lp)]

ditto

[xaav (xaav)]

Possible solution: http://en.wikipedia.org/wiki/NTFS_symbolic_link

[xaav (xaav)]

In fact, since http://bugs.python.org/issue1578269 was fixed, this shouldn't be too hard to implement.

[Jelmer Vernooij (jelmer)]

On Sat, 2011-06-04 at 16:46 +0000, Geoff wrote:
> Possible solution: http://en.wikipedia.org/wiki/NTFS_symbolic_link
The problem with NTFS symbolic links is that they require administrator
privileges, which makes them unusable in a lot of situations.

Cheers,

Jelmer

[Vincent Ladeuil (vila)]

> The problem with NTFS symbolic links is that they require administrator privileges

Quoting http://bugs.python.org/issue1578269 :

  Maybe it's useful to know that Administrator privileges are not required in order be able to create symlinks. You just need a specific permission that by default is assigned only to Administrators.

  How to set SeCreateSymbolicLinkPrivilege on Windows Vista or newer:
   http://stackoverflow.com/questions/815472/how-do-i-grant-secreatesymboliclink-on-windows-vista-home-edition/2085468#2085468

But anyway, if I read the bug report correctly (what an epic, that's kind of scary), this won't be available before python-3.2, so we won't even start to look at it before migrating to python 3...

[Sasha (sasha-2)]

FAR manager is managing all kinds of NTFS links without need for admin rights, C sources at http://www.farmanager.com/opensource.php?l=en

[Parth Malwankar (parthm)]

Hello,

I was wondering if the solution to this issue should be considered to consist of two parts.

  (a) Enable checkout of branches containing symlinks on windows. Show warnings about symlinks with commands like branch, status etc. This should be simpler and can be done in the short term.
  (b) In the longer run other options like those discussed in this bug can be explored to give a more complete solution.

IMHO, (a) should unblock a most users like me who need this feature. In my particular case, we have a situation where the repository resides on unix while some users would prefer to use windows along with various IDEs for editing the files. The users are OK if the symlinks don't exist on Windows. So, just having the ability to checkout the branch should help a lot.

I have created an initial patch for (a) in the branch lp:~parthm/bzr/81689-win-symlink-warning with the "branch" command working on Windows. If (a) seems like a good approach, I could work more on the patch and update other commands and tests to complete the patch and put it up for further review.

Following is the log of whats working ("branch"). As can be seen, commands like status and diff still need work:

D:\ext-src>brun branch lp:~parthm/+junk/project-with-symlink

D:\ext-src>c:/python27/python.exe d:/ext-src/bzr.dev/81689-win-symlink-warning/b
zr branch lp:~parthm/+junk/project-with-symlink
Unable to creat symlink "softlink" on this platform.
Branched 2 revisions.

D:\ext-src>cd project-with-symlink

D:\ext-src\project-with-symlink>ls
hardlink hello

D:\ext-src\project-with-symlink>bzr st
removed:
  softlink@

D:\ext-src\project-with-symlink>..\brun diff

D:\ext-src\project-with-symlink>c:/python27/python.exe d:/ext-src/bzr.dev/81689-
win-symlink-warning/bzr diff
=== removed symlink 'softlink'
=== target was u'hello'

[Martin Pool (mbp)]

Hi Parth!

I think the main drawback with (a) is just that people may commit
changes that delete the symlinks, which may annoy their collaborators.
 However, perhaps that's less total annoyance than not being able to
check out the branch on Windows at all. I would probably take the
patch if the implementation is ok, and perhaps we can come back for b
later.

[Parth Malwankar (parthm)]

Hi Martin,

As a part of ignoring symlinks on Windows, I plan to update affected commands to show the consistent behavior of showing the "Ignoring 'xxx' as symlinks are not supported on this platform" warning. So, this should ensure that symlinks are not deleted during Windows commits. I have modified to patch for "status", "diff" and "commit" as well. Will work some more on this over the next few days and submit a merge proposal once there patch is cleanup and has tests. I still need to verify if "check" works correctly. Are there any other commands I should looks at.

Below is the interaction with "status", "diff" and "commit" with the new changes:

D:\ext-src\project-with-symlink>ls
hardlink hello

D:\ext-src\project-with-symlink>..\brun.bat st
Ignoring "softlink" as symlinks are not supported on this platform.

D:\ext-src\project-with-symlink>..\brun.bat diff
Ignoring "softlink" as symlinks are not supported on this platform.

D:\ext-src\project-with-symlink>..\brun.bat ci
Committing to: D:/ext-src/project-with-symlink/
missing softlink
Ignoring "softlink" as symlinks are not supported on this platform.
bzr: ERROR: No changes to commit. Please 'bzr add' the files you want to commit,
 or use --unchanged to force an empty commit.

[Martin Pool (mbp)]

On 20 February 2012 18:52, Parth Malwankar <email address hidden> wrote:
>
> As a part of ignoring symlinks on Windows, I plan to update affected
> commands to show the consistent behavior of showing the "Ignoring 'xxx'
> as symlinks are not supported on this platform" warning.

This sounds a bit like you're planning to do it command-by-command.
It doesn't belong in the command layer; it should be somewhere near
the tree.

--
Martin

[Parth Malwankar (parthm)]

> This sounds a bit like you're planning to do it command-by-command.
> It doesn't belong in the command layer; it should be somewhere near
> the tree.
>

The main changes at the moment are in transform.py and delta.py,
though some changes are present in commit.py and diff.py.
I will go ahead and make the merge proposal so that I can improve the
patch based on review comments.

[Sasha (sasha-2)]

The http://wiki.bazaar.canonical.com/WindowsSymlinkSupport proposals are still rely on "we could not support" concept. What is wrong to call external executables to cover the case?
"mklink" does the creation and "dir" recognition whether file/folder is symlink and where it points to.

While external executables call is slower than API, it is better than not support windows as platform or cut-off the symlink feature as proposed early.

[Basil Peace (grv87)]

It's sad to hear that there is no progress in this issue. I want to check out source of pytz package for own building, but it has symbolic links.

I want make some notes for this discussion and also for mentioned wiki page.

1. NTFS is the main file system on modern Windows. There is no reason to implement symbolic links for FAT.
Simply it should be noted in documentation that users should have NTFS if they need symbolic links. Modern developers have it.
2. Problems with privileges can be avoided by UAC manifest, however, I don't sure if it could be suitable without code signing.
First link from google about checking privileges from Python: http://stackoverflow.com/questions/2094663/determine-if-windows-process-has-privilege-to-create-symbolic-link
3. Using of external executables is the last idea, only if everything else doesn't work. There are a lot of reasons to avoid it, including security considerations.

[Ian Brandt (ian-ianbrandt)]

I had a symlink in Bazaar that prevented Windows users from checking out the project. I've removed the symlink, added a copy of the file in its place, and committed a new revision. On Windows I still get the same error trying to branch the project. Does this bug apply if there is a symlink anywhere in the history of the project? I would have thought a checkout would work if the latest revision contains no symlinks.

More details from IRC:

[12:14pm] ianbrandt: abentley: Hello. I noticed you started the wiki page on Windows symlink support (http://wiki.bazaar.canonical.com/WindowsSymlinkSupport). Do you know if there is any workaround for getting "bzr: ERROR: Unable to create symlink 'foo' on this platform" on checkout even after the 'foo' symlink has been removed and replaced with a file in the latest commit?
[12:15pm] abentley: ianbrandt: No, if the symlink was removed in the latest commit, I'm surprised that you can't do a checkout.
[12:17pm] ianbrandt: abentley: Okay. Me too. This is with 2.5.1. On my mac when I do revno I get 4102. Same on Windows where the repository was checkout out, but the tree failed to populate.
[12:21pm] ianbrandt: bzr ls -R --kind=symlink at the root of the project returns nothing on my mac.
[12:21pm] ianbrandt: bzr revert -r 4102 on Windows results in "bzr: ERROR: No final name for trans_id 'new-1120' file-id: None root trans-id: 'new-0'"

[Ian Brandt (ian-ianbrandt)]

I've removed the file in question from the latest revision, and now the project will check out on Windows. So apparently Bazaar can't checkout a project when it contains a file with the same name of a former symlink.

[Alexander Belchenko (bialix)]

Ian Brandt пишет:
> I've removed the file in question from the latest revision, and now the
> project will check out on Windows. So apparently Bazaar can't checkout
> a project when it contains a file with the same name of a former
> symlink.

My best guess is: you can restore in the next revision your file. I.e.
you need delete symlink, commit, then add new file with the name of
symlink. That's because bzr is able to track change of symlink to
plain file, and that might confuse bzr on windows.