%run doesn't work with paths returned by tempfile.NamedTemporaryFile()
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
IPython |
Invalid
|
Low
|
Fernando Perez |
Bug Description
In test_magic we use tempfile.
To reproduce, do:
import tempfile
f = tempfile.
f.write('pass\n')
%run $f.name
Brian Granger (ellisonbg) wrote : | #1 |
Fernando Perez (fdo.perez) wrote : | #2 |
Brian, I don't see this bug:
In [5]: import tempfile
In [6]: f = tempfile.
In [7]: f.write('print 999\n')
In [8]: f.flush()
In [9]: %run $f.name
999
This works fine for me on linux. Is it an issue with spaces in the paths you're seeing instead? It's possible that the problem is that %run doesn't handle paths with spaces in them...
Brian Granger (ellisonbg) wrote : Re: [Bug 366353] Re: %run doesn't work with paths returned by tempfile.NamedTemporaryFile() | #3 |
I see paths that have the ~ in them (used to shorten the path). I
don't know how to fix this and I need to keep moving to get this out.
Brian
On Fri, Apr 24, 2009 at 3:42 PM, Fernando Perez <email address hidden> wrote:
> Brian, I don't see this bug:
>
> In [5]: import tempfile
>
> In [6]: f = tempfile.
>
> In [7]: f.write('print 999\n')
>
> In [8]: f.flush()
>
> In [9]: %run $f.name
> 999
>
>
> This works fine for me on linux. Is it an issue with spaces in the paths you're seeing instead? It's possible that the problem is that %run doesn't handle paths with spaces in them...
>
> --
> %run doesn't work with paths returned by tempfile.
> https:/
> You received this bug notification because you are a member of IPython
> Developers, which is subscribed to IPython.
>
> Status in IPython - Enhanced Interactive Python: New
>
> Bug description:
> In test_magic we use tempfile.
>
> To reproduce, do:
>
> import tempfile
> f = tempfile.
> f.write('pass\n')
> %run $f.name
>
--
Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo
<email address hidden>
<email address hidden>
Robert Kern (robert-kern) wrote : | #4 |
On Fri, Apr 24, 2009 at 17:57, Brian Granger <email address hidden> wrote:
> I see paths that have the ~ in them (used to shorten the path). I
> don't know how to fix this and I need to keep moving to get this out.
%run "$f.name"
--
Robert Kern
"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
-- Umberto Eco
Fernando Perez (fdo.perez) wrote : | #5 |
On Fri, Apr 24, 2009 at 4:13 PM, Robert Kern
<email address hidden> wrote:
> On Fri, Apr 24, 2009 at 17:57, Brian Granger <email address hidden> wrote:
>> I see paths that have the ~ in them (used to shorten the path). I
>> don't know how to fix this and I need to keep moving to get this out.
>
> %run "$f.name"
Thanks, Robert. Brian, did that work?
f
Brian Granger (ellisonbg) wrote : | #6 |
> %run "$f.name"
This doesn't work - I get the same error as before, that it can't find
the file. I think this is an issue of how the path is written, but I
haven't had a chance to really investigate it. If I could copy and
paste from the retarded Windows Command Prompt, I would show you the
error.
Brian
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
> -- Umberto Eco
>
> --
> %run doesn't work with paths returned by tempfile.
> https:/
> You received this bug notification because you are a member of IPython
> Developers, which is subscribed to IPython.
>
> Status in IPython - Enhanced Interactive Python: New
>
> Bug description:
> In test_magic we use tempfile.
>
> To reproduce, do:
>
> import tempfile
> f = tempfile.
> f.write('pass\n')
> %run $f.name
>
--
Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo
<email address hidden>
<email address hidden>
Robert Kern (robert-kern) wrote : | #7 |
On Fri, Apr 24, 2009 at 22:43, Brian Granger <email address hidden> wrote:
>> %run "$f.name"
>
> This doesn't work - I get the same error as before, that it can't find
> the file. I think this is an issue of how the path is written, but I
> haven't had a chance to really investigate it. If I could copy and
> paste from the retarded Windows Command Prompt, I would show you the
> error.
You can via the menu:
http://
But this is much, much better than the Windows Command Prompt:
http://
--
Robert Kern
"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
-- Umberto Eco
Brian Granger (ellisonbg) wrote : | #8 |
> You can via the menu:
>
> http://
>
> But this is much, much better than the Windows Command Prompt:
>
> http://
Thanks, that will help a lot!
Brian
Brian Granger (ellisonbg) wrote : | #9 |
Here is the actual error and the path names that seem to be problematic:
In [1]: import tempfile
In [2]: f = tempfile.
In [3]: f.name
Out[3]: 'c:\\docume~
In [4]: f.write('pass\n')
In [5]: %run $f.name
ERROR: File `c:docume~
On Fri, Apr 24, 2009 at 9:18 PM, Brian Granger <email address hidden> wrote:
>> You can via the menu:
>>
>> http://
>>
>> But this is much, much better than the Windows Command Prompt:
>>
>> http://
>
> Thanks, that will help a lot!
>
> Brian
>
--
Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo
<email address hidden>
<email address hidden>
Jörgen Stenarson (jorgen-stenarson) wrote : Re: [Bug 366353] Re: %run doesn't work with paths returned by tempfile.NamedTemporaryFile() | #10 |
Brian Granger skrev:
> Here is the actual error and the path names that seem to be problematic:
>
> In [1]: import tempfile
>
> In [2]: f = tempfile.
>
> In [3]: f.name
> Out[3]: 'c:\\docume~
>
> In [4]: f.write('pass\n')
>
> In [5]: %run $f.name
> ERROR: File `c:docume~
>
Perhaps you can use something like this to get the full pathname first.
The code below was based on a function from
<https:/
import ctypes
_GetLongPathName = ctypes.
_GetLongPathNam
ctypes.c_uint ]
def get_long_
buf = ctypes.
rv = _GetLongPathNam
if rv == 0 or rv > 260:
return path
else:
return buf.value
or using win32api (which is not a standard library but comes with pywin32)
import win32api
win32api.
I would probably choose the first approach because ctypes is a
dependency of pyreadline which most users of ipython would have anyway.
But pywin32 is not.
using either of these solutions you can do
In [7]: import win32api, tempfile
In [8]: f = tempfile.
In [9]: f.name
Out[9]: 'c:\\docume~
In [10]: win32api.
Out[10]: 'c:\\Documents and Settings\
inst\xe4llninga
In [11]: get_long_
Out[11]: u'c:\\Documents and Settings\
Inst\xe4llninga
In [12]: win32api.
Out[12]: u'c:\\Documents and Settings\
inst\xe4llninga
Brian Granger (ellisonbg) wrote : Re: [Bug 366353] Re: %run doesn't work with paths returned by tempfile.NamedTemporaryFile() | #11 |
Jorgen,
Thanks!!! I will give this a shot. One thing to note, we are already
using pywin32 in IPython. But maybe we can replace our usage of it
with ctypes versions. Nice!
Brian
On Sat, Apr 25, 2009 at 11:00 AM, Jörgen Stenarson
<email address hidden> wrote:
> Brian Granger skrev:
>> Here is the actual error and the path names that seem to be problematic:
>>
>> In [1]: import tempfile
>>
>> In [2]: f = tempfile.
>>
>> In [3]: f.name
>> Out[3]: 'c:\\docume~
>>
>> In [4]: f.write('pass\n')
>>
>> In [5]: %run $f.name
>> ERROR: File `c:docume~
>>
>
> Perhaps you can use something like this to get the full pathname first.
>
> The code below was based on a function from
> <https:/
>
> import ctypes
> _GetLongPathName = ctypes.
> _GetLongPathNam
> ctypes.c_uint ]
>
> def get_long_
> buf = ctypes.
> rv = _GetLongPathNam
> if rv == 0 or rv > 260:
> return path
> else:
> return buf.value
>
>
> or using win32api (which is not a standard library but comes with pywin32)
>
> import win32api
> win32api.
>
> I would probably choose the first approach because ctypes is a
> dependency of pyreadline which most users of ipython would have anyway.
> But pywin32 is not.
>
>
> using either of these solutions you can do
>
> In [7]: import win32api, tempfile
>
> In [8]: f = tempfile.
>
> In [9]: f.name
> Out[9]: 'c:\\docume~
>
> In [10]: win32api.
> Out[10]: 'c:\\Documents and Settings\
> inst\xe4llninga
>
> In [11]: get_long_
> Out[11]: u'c:\\Documents and Settings\
> Inst\xe4llninga
>
> In [12]: win32api.
> Out[12]: u'c:\\Documents and Settings\
> inst\xe4llninga
>
> --
> %run doesn't work with paths returned by tempfile.
> https:/
> You received this bug notification because you are a member of IPython
> Developers, which is subscribed to IPython.
>
> Status in IPython - Enhanced Interactive Python: New
>
> Bug description:
> In test_magic we use tempfile.
>
> To reproduce, do:
>
> import tempfile
> f = tempfile.
> f.write('pass\n')
> %run $f.name
>
--
Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo
<email address hidden>
<email address hidden>
Changed in ipython: | |
assignee: | nobody → ellisonbg |
importance: | Undecided → Critical |
status: | New → Confirmed |
Jörgen Stenarson (jorgen-stenarson) wrote : Re: [Bug 366353] Re: %run doesn't work with paths returned by tempfile.NamedTemporaryFile() | #12 |
Brian Granger skrev:
> Jorgen,
>
> Thanks!!! I will give this a shot. One thing to note, we are already
> using pywin32 in IPython. But maybe we can replace our usage of it
> with ctypes versions. Nice!
>
Oh, I didn't know that. It must be for some optional use because I have
been running without it at home until I tried that win32api call:)
/Jörgen
Brian Granger (ellisonbg) wrote : Re: [Bug 366353] Re: %run doesn't work with paths returned by tempfile.NamedTemporaryFile() | #13 |
I have added a new function to platutils that can expand the "~" in
Windows paths:
In [1]: import tempfile
In [2]: import os
In [3]: from IPython.platutils import get_long_path_name
In [4]: f = tempfile.
In [5]: f.write('pass\n')
In [6]: name = f.name
In [7]: bname = get_long_
In [10]: name
Out[10]: 'c:\\docume~
In [11]: bname
Out[11]: u'c:\\Documents and Settings\
Settings\
In [8]: %run $name
ERROR: File `c:docume~
In [9]: %run $bname
ERROR: File `c:Documents.py` not found.
Run chokes on both forms of Windows paths. I have tracked the problem
down to line 1557 of Magic.py:
Here the parse_options method is where the paths get messed up. If
the path has spaces, parse_options is splitting on the spaces and only
returning the first part of the path. If the path has the "~" chars,
it somehow deletes the ""\\"" path separators. I don't have time to
fix this now, but that is what is wrong.
Robert Kern (robert-kern) wrote : | #14 |
Try %run "$bname"
--
Robert Kern
"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
-- Umberto Eco
Fernando Perez (fdo.perez) wrote : | #15 |
Robert is right in his suggestion, since that prevents the magic stripping of the name. In general, whenever one is using a name with spaces in it, it's best to quote it. However, the problem is actually deeper. I made a little script from Brian's code:
import tempfile
import os
from IPython.platutils import get_long_path_name
f = tempfile.
f.write('print "In temp file"\n')
f.flush()
name = f.name
bname = get_long_
print 'name:',name
_ip.magic('run %s' % name)
print
print 'bname:',bname
_ip.magic('run %s' % bname)
print
print '"bname" "%s"' % bname
_ip.magic('run "%s"' % bname)
### EOF
and now we get this:
In [3]: run t366353.py
name: c:\docume~
ERROR: File `c:docume~
bname: c:\Documents and Settings\
ERROR: File `c:Documents.py` not found.
"bname" "c:\Documents and Settings\
Could not open file <c:\Documents and Settings\
r safe execution.
So even the third option (Robert's quoting suggestion) still doesn't work, even though now %run gets the correct path. The problem is actually quite fundamental:
In [7]: bname
Out[7]: u'c:\\Documents and Settings\
In [8]: execfile(bname)
-------
IOError Traceback (most recent call last)
H:\ipython\
----> 1
2
3
4
5
IOError: [Errno 13] Permission denied: 'c:\\Documents and Settings\
\\Temp\
In [9]: !more "$bname"
Cannot access file C:\Documents and Settings\
Even though the file is perfectly writable:
In [10]: f
Out[10]: <open file '<fdopen>', mode 'w+b' at 0x012152F0>
In [11]: f.write('x=1\n')
In [12]: f.flush()
In [13]: f.seek(0)
I think the issue is this:
http://
in particular:
Whether the name can be used to open the file a second time, while the named temporary file is still open, varies across platforms (it can be so used on Unix; it cannot on Windows NT or later)
So basically that function should only be used if you are only going to ever open the file once.
Since this isn't really a %run bug but a fundamental limitation of the Windows NamedTemporaryFile API, I'm closing this bug. I will fix the test suite so we use something else instead in our tests, so the *tests* also run fine in windows, but we don't have a real bug there.
This problem also bit us recently in the NIPY test suite, which is why this time I knew what to look for :)
Fernando Perez (fdo.perez) wrote : | #16 |
Closing as invalid because there isn't really a bug in our %run. I'll adapt the test suite, and any user who might be using NamedTemporaryFile under Windows must be aware of this limitation, but it's a generic Python problem, not an ipython one.
Changed in ipython: | |
assignee: | Brian Granger (ellisonbg) → Fernando Perez (fdo.perez) |
importance: | Critical → Low |
status: | Confirmed → Invalid |
Because of this bug, many tests in test_magic are disabled in win32 because they use %run with tempfiles. When this bug is fixed, these tests need to be un-skipped.