%run doesn't work with paths returned by tempfile.NamedTemporaryFile()

[Brian Granger]

In test_magic we use tempfile.NamedTemporaryFile() to create files that are used with %run during the test suite. The paths created by tempfile don't play well with %run:

To reproduce, do:

import tempfile
f = tempfile.NamedTemporaryFile()
f.write('pass\n')
%run $f.name

[Brian Granger (ellisonbg)]

Because of this bug, many tests in test_magic are disabled in win32 because they use %run with tempfiles. When this bug is fixed, these tests need to be un-skipped.

[Fernando Perez (fdo.perez)]

Brian, I don't see this bug:

In [5]: import tempfile

In [6]: f = tempfile.NamedTemporaryFile()

In [7]: f.write('print 999\n')

In [8]: f.flush()

In [9]: %run $f.name
999

This works fine for me on linux. Is it an issue with spaces in the paths you're seeing instead? It's possible that the problem is that %run doesn't handle paths with spaces in them...

[Brian Granger (ellisonbg)]

I see paths that have the ~ in them (used to shorten the path). I
don't know how to fix this and I need to keep moving to get this out.

Brian

On Fri, Apr 24, 2009 at 3:42 PM, Fernando Perez <email address hidden> wrote:
> Brian, I don't see this bug:
>
> In [5]: import tempfile
>
> In [6]: f = tempfile.NamedTemporaryFile()
>
> In [7]: f.write('print 999\n')
>
> In [8]: f.flush()
>
> In [9]: %run $f.name
> 999
>
>
> This works fine for me on linux.  Is it an issue with spaces in the paths you're seeing instead?  It's possible that the problem is that %run doesn't handle paths with spaces in them...
>
> --
> %run doesn't work with paths returned by tempfile.NamedTemporaryFile()
> https://bugs.launchpad.net/bugs/366353
> You received this bug notification because you are a member of IPython
> Developers, which is subscribed to IPython.
>
> Status in IPython - Enhanced Interactive Python: New
>
> Bug description:
> In test_magic we use tempfile.NamedTemporaryFile() to create files that are used with %run during the test suite.  The paths created by tempfile don't play well with %run:
>
> To reproduce, do:
>
> import tempfile
> f = tempfile.NamedTemporaryFile()
> f.write('pass\n')
> %run $f.name
>

--
Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo
<email address hidden>
<email address hidden>

[Robert Kern (robert-kern)]

On Fri, Apr 24, 2009 at 17:57, Brian Granger <email address hidden> wrote:
> I see paths that have the ~ in them (used to shorten the path).  I
> don't know how to fix this and I need to keep moving to get this out.

%run "$f.name"

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco

[Fernando Perez (fdo.perez)]

On Fri, Apr 24, 2009 at 4:13 PM, Robert Kern
<email address hidden> wrote:
> On Fri, Apr 24, 2009 at 17:57, Brian Granger <email address hidden> wrote:
>> I see paths that have the ~ in them (used to shorten the path).  I
>> don't know how to fix this and I need to keep moving to get this out.
>
> %run "$f.name"

Thanks, Robert. Brian, did that work?

f

[Brian Granger (ellisonbg)]

> %run "$f.name"

This doesn't work - I get the same error as before, that it can't find
the file. I think this is an issue of how the path is written, but I
haven't had a chance to really investigate it. If I could copy and
paste from the retarded Windows Command Prompt, I would show you the
error.

Brian

> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
>  -- Umberto Eco
>
> --
> %run doesn't work with paths returned by tempfile.NamedTemporaryFile()
> https://bugs.launchpad.net/bugs/366353
> You received this bug notification because you are a member of IPython
> Developers, which is subscribed to IPython.
>
> Status in IPython - Enhanced Interactive Python: New
>
> Bug description:
> In test_magic we use tempfile.NamedTemporaryFile() to create files that are used with %run during the test suite.  The paths created by tempfile don't play well with %run:
>
> To reproduce, do:
>
> import tempfile
> f = tempfile.NamedTemporaryFile()
> f.write('pass\n')
> %run $f.name
>

--
Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo
<email address hidden>
<email address hidden>

[Robert Kern (robert-kern)]

On Fri, Apr 24, 2009 at 22:43, Brian Granger <email address hidden> wrote:
>> %run "$f.name"
>
> This doesn't work - I get the same error as before, that it can't find
> the file.  I think this is an issue of how the path is written, but I
> haven't had a chance to really investigate it.  If I could copy and
> paste from the retarded Windows Command Prompt, I would show you the
> error.

You can via the menu:

  http://www.ss64.com/nt/cmd.html

But this is much, much better than the Windows Command Prompt:

  http://sourceforge.net/projects/console/

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco

[Brian Granger (ellisonbg)]

> You can via the menu:
>
>  http://www.ss64.com/nt/cmd.html
>
> But this is much, much better than the Windows Command Prompt:
>
>  http://sourceforge.net/projects/console/

Thanks, that will help a lot!

Brian

[Brian Granger (ellisonbg)]

Here is the actual error and the path names that seem to be problematic:

In [1]: import tempfile

In [2]: f = tempfile.NamedTemporaryFile()

In [3]: f.name
Out[3]: 'c:\\docume~1\\admini~1\\locals~1\\temp\\tmpfydqw_'

In [4]: f.write('pass\n')

In [5]: %run $f.name
ERROR: File `c:docume~1admini~1locals~1temptmpfydqw_.py` not found.

On Fri, Apr 24, 2009 at 9:18 PM, Brian Granger <email address hidden> wrote:
>> You can via the menu:
>>
>>  http://www.ss64.com/nt/cmd.html
>>
>> But this is much, much better than the Windows Command Prompt:
>>
>>  http://sourceforge.net/projects/console/
>
> Thanks, that will help a lot!
>
> Brian
>

--
Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo
<email address hidden>
<email address hidden>

[Jörgen Stenarson (jorgen-stenarson)]

Brian Granger skrev:
> Here is the actual error and the path names that seem to be problematic:
>
> In [1]: import tempfile
>
> In [2]: f = tempfile.NamedTemporaryFile()
>
> In [3]: f.name
> Out[3]: 'c:\\docume~1\\admini~1\\locals~1\\temp\\tmpfydqw_'
>
> In [4]: f.write('pass\n')
>
> In [5]: %run $f.name
> ERROR: File `c:docume~1admini~1locals~1temptmpfydqw_.py` not found.
>

Perhaps you can use something like this to get the full pathname first.

The code below was based on a function from
<https://svn.participatoryculture.org/svn/dtv/tags/Democracy-Player-0.9.1.1/tv/platform/windows-xul/platformutils.py>

import ctypes
_GetLongPathName = ctypes.windll.kernel32.GetLongPathNameW
_GetLongPathName.argtypes = [ctypes.c_wchar_p, ctypes.c_wchar_p,
ctypes.c_uint ]

def get_long_path_name(path):
     buf = ctypes.create_unicode_buffer(260)
     rv = _GetLongPathName(path, buf, 260)
     if rv == 0 or rv > 260:
         return path
     else:
         return buf.value

or using win32api (which is not a standard library but comes with pywin32)

import win32api
win32api.GetShortPathName (path_name)

I would probably choose the first approach because ctypes is a
dependency of pyreadline which most users of ipython would have anyway.
But pywin32 is not.

using either of these solutions you can do

In [7]: import win32api, tempfile

In [8]: f = tempfile.NamedTemporaryFile()

In [9]: f.name
Out[9]: 'c:\\docume~1\\jstenar\\lokala~1\\temp\\tmp3xrtkk'

In [10]: win32api.GetLongPathName(f.name)
Out[10]: 'c:\\Documents and Settings\\jstenar\\Lokala
inst\xe4llningar\\Temp\\tmp3xrtkk'

In [11]: get_long_path_name(f.name)
Out[11]: u'c:\\Documents and Settings\\jstenar\\Lokala
Inst\xe4llningar\\Temp\\tmp3xrtkk'

In [12]: win32api.GetLongPathNameW(f.name)
Out[12]: u'c:\\Documents and Settings\\jstenar\\Lokala
inst\xe4llningar\\Temp\\tmp3xrtkk'

[Brian Granger (ellisonbg)]

Jorgen,

Thanks!!! I will give this a shot. One thing to note, we are already
using pywin32 in IPython. But maybe we can replace our usage of it
with ctypes versions. Nice!

Brian

On Sat, Apr 25, 2009 at 11:00 AM, Jörgen Stenarson
<email address hidden> wrote:
> Brian Granger skrev:
>> Here is the actual error and the path names that seem to be problematic:
>>
>> In [1]: import tempfile
>>
>> In [2]: f = tempfile.NamedTemporaryFile()
>>
>> In [3]: f.name
>> Out[3]: 'c:\\docume~1\\admini~1\\locals~1\\temp\\tmpfydqw_'
>>
>> In [4]: f.write('pass\n')
>>
>> In [5]: %run $f.name
>> ERROR: File `c:docume~1admini~1locals~1temptmpfydqw_.py` not found.
>>
>
> Perhaps you can use something like this to get the full pathname first.
>
> The code below was based on a function from
> <https://svn.participatoryculture.org/svn/dtv/tags/Democracy-Player-0.9.1.1/tv/platform/windows-xul/platformutils.py>
>
> import ctypes
> _GetLongPathName = ctypes.windll.kernel32.GetLongPathNameW
> _GetLongPathName.argtypes = [ctypes.c_wchar_p, ctypes.c_wchar_p,
> ctypes.c_uint ]
>
> def get_long_path_name(path):
>     buf = ctypes.create_unicode_buffer(260)
>     rv = _GetLongPathName(path, buf, 260)
>     if rv == 0 or rv > 260:
>         return path
>     else:
>         return buf.value
>
>
> or using win32api (which is not a standard library but comes with pywin32)
>
> import win32api
> win32api.GetShortPathName (path_name)
>
> I would probably choose the first approach because ctypes is a
> dependency of pyreadline which most users of ipython would have anyway.
> But pywin32 is not.
>
>
> using either of these solutions you can do
>
> In [7]: import win32api, tempfile
>
> In [8]: f = tempfile.NamedTemporaryFile()
>
> In [9]: f.name
> Out[9]: 'c:\\docume~1\\jstenar\\lokala~1\\temp\\tmp3xrtkk'
>
> In [10]: win32api.GetLongPathName(f.name)
> Out[10]: 'c:\\Documents and Settings\\jstenar\\Lokala
> inst\xe4llningar\\Temp\\tmp3xrtkk'
>
> In [11]: get_long_path_name(f.name)
> Out[11]: u'c:\\Documents and Settings\\jstenar\\Lokala
> Inst\xe4llningar\\Temp\\tmp3xrtkk'
>
> In [12]: win32api.GetLongPathNameW(f.name)
> Out[12]: u'c:\\Documents and Settings\\jstenar\\Lokala
> inst\xe4llningar\\Temp\\tmp3xrtkk'
>
> --
> %run doesn't work with paths returned by tempfile.NamedTemporaryFile()
> https://bugs.launchpad.net/bugs/366353
> You received this bug notification because you are a member of IPython
> Developers, which is subscribed to IPython.
>
> Status in IPython - Enhanced Interactive Python: New
>
> Bug description:
> In test_magic we use tempfile.NamedTemporaryFile() to create files that are used with %run during the test suite.  The paths created by tempfile don't play well with %run:
>
> To reproduce, do:
>
> import tempfile
> f = tempfile.NamedTemporaryFile()
> f.write('pass\n')
> %run $f.name
>

--
Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo
<email address hidden>
<email address hidden>

[Jörgen Stenarson (jorgen-stenarson)]

Brian Granger skrev:
> Jorgen,
>
> Thanks!!! I will give this a shot. One thing to note, we are already
> using pywin32 in IPython. But maybe we can replace our usage of it
> with ctypes versions. Nice!
>
Oh, I didn't know that. It must be for some optional use because I have
been running without it at home until I tried that win32api call:)

/Jörgen

[Brian Granger (ellisonbg)]

I have added a new function to platutils that can expand the "~" in
Windows paths:

In [1]: import tempfile

In [2]: import os

In [3]: from IPython.platutils import get_long_path_name

In [4]: f = tempfile.NamedTemporaryFile(suffix='.py')

In [5]: f.write('pass\n')

In [6]: name = f.name

In [7]: bname = get_long_path_name(name)

In [10]: name
Out[10]: 'c:\\docume~1\\admini~1\\locals~1\\temp\\tmpixcmwt.py'

In [11]: bname
Out[11]: u'c:\\Documents and Settings\\Administrator\\Local
Settings\\Temp\\tmpixcmwt.py'

In [8]: %run $name
ERROR: File `c:docume~1admini~1locals~1temptmpixcmwt.py` not found.

In [9]: %run $bname
ERROR: File `c:Documents.py` not found.

Run chokes on both forms of Windows paths. I have tracked the problem
down to line 1557 of Magic.py:

        opts,arg_lst = self.parse_options(parameter_s,'nidtN:b:pD:l:rs:T:e',
                                          mode='list',list_all=1)

Here the parse_options method is where the paths get messed up. If
the path has spaces, parse_options is splitting on the spaces and only
returning the first part of the path. If the path has the "~" chars,
it somehow deletes the ""\\"" path separators. I don't have time to
fix this now, but that is what is wrong.

[Robert Kern (robert-kern)]

Try %run "$bname"

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco

[Fernando Perez (fdo.perez)]

Robert is right in his suggestion, since that prevents the magic stripping of the name. In general, whenever one is using a name with spaces in it, it's best to quote it. However, the problem is actually deeper. I made a little script from Brian's code:

import tempfile
import os
from IPython.platutils import get_long_path_name

f = tempfile.NamedTemporaryFile(suffix='.py')
f.write('print "In temp file"\n')
f.flush()

name = f.name
bname = get_long_path_name(name)

print 'name:',name
_ip.magic('run %s' % name)
print
print 'bname:',bname
_ip.magic('run %s' % bname)
print
print '"bname" "%s"' % bname
_ip.magic('run "%s"' % bname)

### EOF

and now we get this:

In [3]: run t366353.py
name: c:\docume~1\fperez\locals~1\temp\tmpiovihn.py
ERROR: File `c:docume~1fperezlocals~1temptmpiovihn.py` not found.

bname: c:\Documents and Settings\fperez\Local Settings\Temp\tmpiovihn.py
ERROR: File `c:Documents.py` not found.

"bname" "c:\Documents and Settings\fperez\Local Settings\Temp\tmpiovihn.py"
Could not open file <c:\Documents and Settings\fperez\Local Settings\Temp\tmpiovihn.py> fo
r safe execution.

So even the third option (Robert's quoting suggestion) still doesn't work, even though now %run gets the correct path. The problem is actually quite fundamental:

In [7]: bname
Out[7]: u'c:\\Documents and Settings\\fperez\\Local Settings\\Temp\\tmp1o8msd.py'

In [8]: execfile(bname)
---------------------------------------------------------------------------
IOError Traceback (most recent call last)

H:\ipython\ipython\IPython\tests\t366353.py in <module>()
----> 1
      2
      3
      4
      5

IOError: [Errno 13] Permission denied: 'c:\\Documents and Settings\\fperez\\Local Settings
\\Temp\\tmp1o8msd.py'

In [9]: !more "$bname"
Cannot access file C:\Documents and Settings\fperez\Local Settings\Temp\tmp1o8msd.py

Even though the file is perfectly writable:

In [10]: f
Out[10]: <open file '<fdopen>', mode 'w+b' at 0x012152F0>

In [11]: f.write('x=1\n')

In [12]: f.flush()

In [13]: f.seek(0)

I think the issue is this:

http://docs.python.org/library/tempfile.html#tempfile.NamedTemporaryFile

in particular:

Whether the name can be used to open the file a second time, while the named temporary file is still open, varies across platforms (it can be so used on Unix; it cannot on Windows NT or later)

So basically that function should only be used if you are only going to ever open the file once.

Since this isn't really a %run bug but a fundamental limitation of the Windows NamedTemporaryFile API, I'm closing this bug. I will fix the test suite so we use something else instead in our tests, so the *tests* also run fine in windows, but we don't have a real bug there.

This problem also bit us recently in the NIPY test suite, which is why this time I knew what to look for :)

[Fernando Perez (fdo.perez)]

Closing as invalid because there isn't really a bug in our %run. I'll adapt the test suite, and any user who might be using NamedTemporaryFile under Windows must be aware of this limitation, but it's a generic Python problem, not an ipython one.