python-mode breaks for python 3

[Rustom]

python 3 has removed execfile
This makes python-mode stop working

In function py-execute-file changing the line

(cmd (format "execfile(r'%s') # PYTHON-MODE\n" filename)))

to
(cmd (format "exec(open(r'%s').read()) # PYTHON-MODE\n" filename)))

seems to solve the problem

[Andreas Roehler (a-roehler)]

Rustom wrote:
> Public bug reported:
>
> python 3 has removed execfile
> This makes python-mode stop working
>
> In function py-execute-file changing the line
>
> (cmd (format "execfile(r'%s') # PYTHON-MODE\n" filename)))
>
> to
> (cmd (format "exec(open(r'%s').read()) # PYTHON-MODE\n" filename)))
>
> seems to solve the problem
>
> ** Affects: python-mode
> Importance: Undecided
> Status: New
>

IMO it's not to cure by changing a line in general.

More bugs will be around due to different flavours of python.

Versions of python 2.4.+ are in use still AFAIK.
2.5.+ or 2.6.+ will remain widely used

Will a single python-mode be able to cope with this differences?

IMHO: yes, however it must to address some issues.

Herewith some outlines what seems necessary:

- specifying by a variable the version where edits are for (customizable)

- some guessing, which python-version might be in used, if user didn't set it

- displaying python-versions-variable in mode-line

- auto-adapting code, editing and execution resp. to the mentioned version.

- possible switch between versions-specifics, updating the modeline etc.

Introducing a first item in this line might read:

(defcustom py-adressed-python-version ""
  "*With different Python versions, changes have been made, which affect python execution as editing likewise.
If a version is specified here, python-mode will adapt its proceeding to it.
Otherwise python-mode will try some guess from the installed system.
You may switch the addressed python-version with M-x py-switch-addressed-version during your emacs-session. "
  :type 'string
  :group 'python)

Any hints, objections, ideas?

Thanks!

Andreas

--
https://code.launchpad.net/s-x-emacs-werkstatt/
http://bazaar.launchpad.net/~a-roehler/python-mode/python-mode.el/

[Barry Warsaw (barry)]

On Oct 17, 2009, at 11:54 AM, Andreas Roehler wrote:
>
> (defcustom py-adressed-python-version ""
> "*With different Python versions, changes have been made, which
> affect python execution as editing likewise.
> If a version is specified here, python-mode will adapt its
> proceeding to it.
> Otherwise python-mode will try some guess from the installed system.
> You may switch the addressed python-version with M-x py-switch-
> addressed-version during your emacs-session. "
> :type 'string
> :group 'python)
>
> Any hints, objections, ideas?

I'd definitely love to see some enhancements to python-mode to deal
with Python 3, but of course I have no time for that myself. ;)

I'd suggest two variables:

(defvar py-python-major-version ...)

This would be a buffer-local variable, defaulting to "2", specifying
the Python major version of the code in the current buffer.

(defcustom py-guess-python-version ...)

This would be a global customizable variable specifying whether to
apply heuristics to guess the py-python-major-version when a buffer is
visited. It should probably default to t.

+1 for working on this!
-Barry

[Rustom (rustompmody)]

Right now the mc Im on does not have python 2.x and python3

But when I was on a debian box with 2.5 and 3
changing

(cmd (format "execfile(r'%s') # PYTHON-MODE\n" filename)))

to

(cmd (format "exec(compile(open('%s').read(), '%s', 'exec')) #
PYTHON-MODE\n" filename filename)))

[ not

(cmd (format "exec(open(r'%s').read()) # PYTHON-MODE\n" filename)))
as my original post suggested]

seemed to work for both 2.5 and 3

Whether other changes from 2.x to 3 will introduce breakage in python-mode I dont know

The more important question remaining about my suggested fix is about windows.
Evidently there is some stupidity that exec expects its input to have unix line endings even on windows so the original code may not work on windows but the one with compile should.

But Ive not tested on windows

[Andreas Roehler (a-roehler)]

Rustom wrote:
> Right now the mc Im on does not have python 2.x and python3
>
> But when I was on a debian box with 2.5 and 3
> changing
>
> (cmd (format "execfile(r'%s') # PYTHON-MODE\n" filename)))
>
> to
>
> (cmd (format "exec(compile(open('%s').read(), '%s', 'exec')) #
> PYTHON-MODE\n" filename filename)))
>
> [ not
>
> (cmd (format "exec(open(r'%s').read()) # PYTHON-MODE\n" filename)))
> as my original post suggested]
>
> seemed to work for both 2.5 and 3
>
> Whether other changes from 2.x to 3 will introduce breakage in python-
> mode I dont know
>
> The more important question remaining about my suggested fix is about windows.
> Evidently there is some stupidity that exec expects its input to have unix line endings even on windows so the original code may not work on windows but the one with compile should.
>
> But Ive not tested on windows
>

Hi Rustom,

looks like good news. Think you are right.

Checked a little bit with
Python 2.5.1
on Linux 2.6.22.19-0.2-default #1 SMP 2008-12-18 10:17:03 +0100 i686 athlon i386 GNU/Linux

works fine AFAIS.

What puzzles me still is a pure python question -

do we need this `read()' here, i.e. if a file opened is
delivered to exec, will it not being read anyway?

Thanks

Andreas

--
https://code.launchpad.net/s-x-emacs-werkstatt/
http://bazaar.launchpad.net/~a-roehler/python-mode/python-mode.el/

[Rustom (rustompmody)]

On Tue, Oct 20, 2009 at 1:59 PM, Andreas Roehler
<email address hidden>wrote:

> Rustom wrote:
> > (cmd (format "exec(compile(open('%s').read(), '%s', 'exec')) #
> > PYTHON-MODE\n" filename filename)))
> What puzzles me still is a pure python question -
>
> do we need this `read()' here, i.e. if a file opened is
> delivered to exec, will it not being read anyway?
>
> Thanks
>
>
See Guido's 2 to 3 doc
http://docs.python.org/dev/3.0/whatsnew/3.0.html#removed-syntax
says stream argument not taken

[Andreas Roehler (a-roehler)]

Rustom Mody wrote:
> On Tue, Oct 20, 2009 at 1:59 PM, Andreas Roehler
> <<email address hidden> <mailto:<email address hidden>>> wrote:
>
> Rustom wrote:
> > (cmd (format "exec(compile(open('%s').read(), '%s', 'exec')) #
> > PYTHON-MODE\n" filename filename)))

For me both of your variants are working, see output of checks below.

uname -a && python --version && cat 2+4.py && cat exec-read.py && python exec-read.py &&
                                      cat exec-compile-read.py && python exec-compile-read.py

==>

Linux ... 2.6.22.19-0.2-default #1 SMP 2008-12-18 10:17:03 +0100 i686 athlon i386 GNU/Linux
Python 2.5.1
#! /usr/bin/env python
 # -*- coding: utf-8 -*-

print 2 + 4
##################
#! /usr/bin/env python
 # -*- coding: utf-8 -*-

exec(open('2+4.py').read())
######################

6
#! /usr/bin/env python
 # -*- coding: utf-8 -*-

exec(compile(open('2+4.py').read(), '2+4.py', 'exec'))
#################

6

;;;;;;;;;;;;;;;;;

BTW can you tell whats the us of `compile' here for you?

Do you have some tests for python-mode.el?

> What puzzles me still is a pure python question -
>
> do we need this `read()' here, i.e. if a file opened is
> delivered to exec, will it not being read anyway?
>
> Thanks
>
>
> See Guido's 2 to 3 doc
> http://docs.python.org/dev/3.0/whatsnew/3.0.html#removed-syntax
> says stream argument not taken
>
>
Ah, thanks

Andreas

[Rustom (rustompmody)]

On Thu, Oct 22, 2009 at 5:43 PM, Andreas Roehler
<email address hidden>wrote:

> Rustom Mody wrote:
> > On Tue, Oct 20, 2009 at 1:59 PM, Andreas Roehler
> > <<email address hidden> <mailto:<email address hidden>>> wrote:
> >
> > Rustom wrote:
> > > (cmd (format "exec(compile(open('%s').read(), '%s', 'exec')) #
> > > PYTHON-MODE\n" filename filename)))
>
> For me both of your variants are working, see output of checks below.
>
> You probably need to try on windows.
See
http://bugs.python.org/issue5524

> uname -a && python --version && cat 2+4.py && cat exec-read.py && python
> exec-read.py &&
> cat exec-compile-read.py && python
> exec-compile-read.py
>
> ==>
>
> Linux ... 2.6.22.19-0.2-default #1 SMP 2008-12-18 10:17:03 +0100 i686
> athlon i386 GNU/Linux
> Python 2.5.1
> #! /usr/bin/env python
> # -*- coding: utf-8 -*-
>
> print 2 + 4
> ##################
> #! /usr/bin/env python
> # -*- coding: utf-8 -*-
>
> exec(open('2+4.py').read())
> ######################
>
> 6
> #! /usr/bin/env python
> # -*- coding: utf-8 -*-
>
> exec(compile(open('2+4.py').read(), '2+4.py', 'exec'))
> #################
>
> 6
>
>
> ;;;;;;;;;;;;;;;;;
>
> BTW can you tell whats the us of `compile' here for you?
>
> See
http://stackoverflow.com/questions/436198/what-is-an-alternative-to-execfile-in-python-3-0

> Do you have some tests for python-mode.el?
>
As in automated el/py etc? No
As in biological? ...
Heres yours truly :-)

[Andreas Roehler (a-roehler)]

...
> > Rustom wrote:
> > > (cmd (format "exec(compile(open('%s').read(), '%s', 'exec')) #
> > > PYTHON-MODE\n" filename filename)))
>
> For me both of your variants are working, see output of checks below.
>
> You probably need to try on windows.
> See
> http://bugs.python.org/issue5524

Hmm, AFAIU the use of `compile' here on windows is to signal an error if the file contains "\r" chars?
Right?

>
> uname -a && python --version && cat 2+4.py && cat exec-read.py &&
> python exec-read.py &&
> cat exec-compile-read.py &&
> python exec-compile-read.py
>
> ==>
>
> Linux ... 2.6.22.19-0.2-default #1 SMP 2008-12-18 10:17:03 +0100
> i686 athlon i386 GNU/Linux
> Python 2.5.1
> #! /usr/bin/env python
> # -*- coding: utf-8 -*-
>
> print 2 + 4
> ##################
> #! /usr/bin/env python
> # -*- coding: utf-8 -*-
>
> exec(open('2+4.py').read())
> ######################
>
> 6
> #! /usr/bin/env python
> # -*- coding: utf-8 -*-
>
> exec(compile(open('2+4.py').read(), '2+4.py', 'exec'))
> #################
>
> 6
>
>
> ;;;;;;;;;;;;;;;;;
>
> BTW can you tell whats the us of `compile' here for you?
>
> See
> http://stackoverflow.com/questions/436198/what-is-an-alternative-to-execfile-in-python-3-0
>
>
>
> Do you have some tests for python-mode.el?
>
> As in automated el/py etc? No
> As in biological? ...

If you may deliver a simple test case, how to call interactively it from inside emacs,
it might be helpful - at least for me... :)

Andreas

> Heres yours truly :-)

[Rustom (rustompmody)]

On Thu, Oct 22, 2009 at 8:32 PM, Andreas Roehler
<email address hidden>wrote:

> ...
> > > Rustom wrote:
> > > > (cmd (format "exec(compile(open('%s').read(), '%s', 'exec'))
> #
> > > > PYTHON-MODE\n" filename filename)))
> >
> > For me both of your variants are working, see output of checks below.
> >
> > You probably need to try on windows.
> > See
> > http://bugs.python.org/issue5524
>
> Hmm, AFAIU the use of `compile' here on windows is to signal an error if
> the file contains "\r" chars?
> Right?
>
The way I understood it the exec want Unix-only lineendings and compile
avoids the whole issue by supplying exec with a code object and not a
compilable text.
But I may be wrong

>
> > Do you have some tests for python-mode.el?
> >
> > As in automated el/py etc? No
> > As in biological? ...
>
> If you may deliver a simple test case, how to call interactively it from
> inside emacs,
> it might be helpful - at least for me... :)
>

Not sure what you are asking for.
Your emacs startup should contain something like

(autoload 'py-shell "python-mode" "Python Inferior Mode." t)
(autoload 'python-mode "python-mode" "Python Mode." t)
(add-to-list 'auto-mode-alist '("\\.py\\'" . python-mode))
(add-to-list 'interpreter-mode-alist '("python" . python-mode))

Also assuming that python-mode.el is in your path

After that opening a file with .py extension should start in python mode
From there C-c ! should start the python interpreter
After that (from the file buffer) C-c C-c should read the file into the
python interpreter buffer.

But as I said I am not sure what you are asking for :-)

[Andreas Roehler (a-roehler)]

Rustom wrote:
> On Thu, Oct 22, 2009 at 8:32 PM, Andreas Roehler
> <email address hidden>wrote:
>
>> ...
>>> > Rustom wrote:
>>> > > (cmd (format "exec(compile(open('%s').read(), '%s', 'exec'))
>> #
>>> > > PYTHON-MODE\n" filename filename)))
>>>
>>> For me both of your variants are working, see output of checks below.
>>>
>>> You probably need to try on windows.
>>> See
>>> http://bugs.python.org/issue5524
>> Hmm, AFAIU the use of `compile' here on windows is to signal an error if
>> the file contains "\r" chars?
>> Right?
>>
> The way I understood it the exec want Unix-only lineendings and compile
> avoids the whole issue by supplying exec with a code object and not a
> compilable text.
> But I may be wrong

Probably you are right - if the `\r'-error doesn't occur
with `compile'. Think we should make a comment, saying
"compile" here is introduced for this side-effect.

>
>>> Do you have some tests for python-mode.el?
>>>
>>> As in automated el/py etc? No
>>> As in biological? ...
>> If you may deliver a simple test case, how to call interactively it from
>> inside emacs,
>> it might be helpful - at least for me... :)
>>
>
> Not sure what you are asking for.

> Your emacs startup should contain something like
>
> (autoload 'py-shell "python-mode" "Python Inferior Mode." t)
> (autoload 'python-mode "python-mode" "Python Mode." t)
> (add-to-list 'auto-mode-alist '("\\.py\\'" . python-mode))
> (add-to-list 'interpreter-mode-alist '("python" . python-mode))
>
> Also assuming that python-mode.el is in your path
>
> After that opening a file with .py extension should start in python mode
>>From there C-c ! should start the python interpreter
> After that (from the file buffer) C-c C-c should read the file into the
> python interpreter buffer.
>
> But as I said I am not sure what you are asking for :-)
>

Sorry. Simply tried to catch the execution of your form with edebug.
The ways I tried, it wasn't called.

Thanks anyway.

Andreas

[Rustom (rustompmody)]

On Fri, Oct 23, 2009 at 12:50 AM, Andreas Roehler
<email address hidden> wrote:
>
> Sorry. Simply tried to catch the execution of your form with edebug.
> The ways I tried, it wasn't called.

Ok -- Here goes

1. Take some trivial python file (say foo.py) in python-mode
A single line

x = 1

will do

[I assume it opens with python-mode as major mode]

2. Set edebug to break function py-execute-file (in my case its line no 1358)

3. From foo.py do C-c ! (py-shell) to start python

4. Go back to foo.py buffer

5. Load using C-c C-c (py-execute-buffer)

Why are 3 and 4 necessary? Dunno -- but they are at least in my case

edebug should be at the breakpoint.

Please tell me if this does not work.

Rustom

[Andreas Roehler (a-roehler)]

...

Hi Rustom,

seems I have a ipython-bug here.

with
print 3 * 4
in foo.py

I get the correct result only first time in a separate buffer without ipython.

Afterwards always strange output:

IPython 0.8.1 -- An enhanced Interactive Python.
....
object? -> Details about 'object'. ?object also works, ?? prints more.

In [
1
]:
12
Last number 12 is the correct output.
Seems some encoding from shell-processes wrong...

;;;;;;;;;;;;;;;;;

However, don't think it's related to our patch - attached.

May you try it? Thanks!

Hi Barry,

so far a first draft how to do it. Comments welcome.

Andreas

[Rustom (rustompmody)]

Hello Andreas.

On Tue, Nov 3, 2009 at 2:51 AM, Andreas Roehler
<email address hidden> wrote:
> ...
>
> Hi Rustom,
>
> seems I have a ipython-bug here.
>
> with
> print 3 * 4
> in foo.py
>
> I get the correct result only first time in a separate buffer without
> ipython.

Ive never used IPython.

>
> Afterwards always strange output:
>
> IPython 0.8.1 -- An enhanced Interactive Python.
> ....
> object? -> Details about 'object'. ?object also works, ?? prints more.
>
>  [0;32m In [  [1;32m 1  [0;32m ]:   [0m 12
> Last number 12 is the correct output.
> Seems some encoding from shell-processes wrong...

Ive seen some discussions about encoding issues on the emacs list recently.
Personally Ive never been able to wrap my head round unicode matters
though Ive tried many times :-(

As for IPython I could install it and try but is it related to the
earlier bug viz that python3 removing execfile has broken basic
functionality? If you think so I could poke around. If not I guess we
should deal with separate bugs separately -- No??

One last point -- youve sent (or launchpad has automatically attached)
a patch file.
How does one use it? emacs? bzr? launchpad? Whats the workflow?

Rustom

[Andreas Roehler (a-roehler)]

Rustom wrote:
> Hello Andreas.
>
> On Tue, Nov 3, 2009 at 2:51 AM, Andreas Roehler
> <email address hidden> wrote:
>> ...
>>
>> Hi Rustom,
>>
>> seems I have a ipython-bug here.
>>
>> with
>> print 3 * 4
>> in foo.py
>>
>> I get the correct result only first time in a separate buffer without
>> ipython.
>
> Ive never used IPython.
>
>> Afterwards always strange output:
>>
>> IPython 0.8.1 -- An enhanced Interactive Python.
>> ....
>> object? -> Details about 'object'. ?object also works, ?? prints more.
>>
>> [0;32m In [ [1;32m 1 [0;32m ]: [0m 12
>> Last number 12 is the correct output.
>> Seems some encoding from shell-processes wrong...
>
> Ive seen some discussions about encoding issues on the emacs list recently.
> Personally Ive never been able to wrap my head round unicode matters
> though Ive tried many times :-(
>
> As for IPython I could install it and try but is it related to the
> earlier bug viz that python3 removing execfile has broken basic
> functionality? If you think so

No. Watched my problem in other circumstances too. Might be my own fault. Still need
examining it.

I could poke around. If not I guess we
> should deal with separate bugs separately -- No??
>
> One last point -- youve sent (or launchpad has automatically attached)
> a patch file.
> How does one use it? emacs? bzr? launchpad? Whats the workflow?

Probably many possibilities exists.
A patch is nothing else than a diff.
But exists a programm "patch" already designed to read in that diff.

See: # man patch

or from Emacs: M-x woman patch

Basically it enough to know:

     patch [options] [originalfile [patchfile]]

Don't forget to make a copy of your original or RCS it before...

Enjoy!

Andreas

>
> Rustom
>

[Rustom (rustompmody)]

One suggestion regarding the encoding bug:
As everything becomes encoding/unicode many variables are changing.
In particular emacs has moved from ASCII to UTF-8 default in emacs 23
Likewise python 3 has revamped its entire notion of encoding.

So try and change only 1 variable:
If things were working with emacs22 + python2.x
change only one of emacs/python and see the results.

On Tue, Nov 3, 2009 at 1:00 PM, Andreas Roehler
<email address hidden> wrote:
> Rustom wrote:
>> Hello Andreas.
>>
>> On Tue, Nov 3, 2009 at 2:51 AM, Andreas Roehler
>> <email address hidden> wrote:
>>> ...
>>>
>>> Hi Rustom,
>>>
>>> seems I have a ipython-bug here.
>>>
>>> with
>>> print 3 * 4
>>> in foo.py
>>>
>>> I get the correct result only first time in a separate buffer without
>>> ipython.
>>
>> Ive never used IPython.
>>
>>> Afterwards always strange output:
>>>
>>> IPython 0.8.1 -- An enhanced Interactive Python.
>>> ....
>>> object? -> Details about 'object'. ?object also works, ?? prints more.
>>>
>>>  [0;32m In [  [1;32m 1  [0;32m ]:   [0m 12
>>> Last number 12 is the correct output.
>>> Seems some encoding from shell-processes wrong...
>>
>> Ive seen some discussions about encoding issues on the emacs list recently.
>> Personally Ive never been able to wrap my head round unicode matters
>> though Ive tried many times :-(
>>
>> As for IPython I could install it and try but is it related to the
>> earlier bug viz that python3 removing execfile has broken basic
>> functionality? If you think so
>
> No. Watched my problem in other circumstances too. Might be my own fault. Still need
> examining it.
>
> I could poke around. If not I guess we
>> should deal with separate bugs separately -- No??
>>
>> One last point -- youve sent (or launchpad has automatically attached)
>> a patch file.
>> How does one use it? emacs? bzr? launchpad? Whats the workflow?
>
> Probably many possibilities exists.
> A patch is nothing else than a diff.
> But exists a programm "patch" already designed to read in that diff.
>
> See:  # man patch
>
> or from Emacs: M-x woman patch
>
> Basically it enough to know:
>
>     patch [options] [originalfile [patchfile]]
>
> Don't forget to make a copy of your original or RCS it before...

I would have thought that there is some bzr specific mantras to be chanted :-)

[Andreas Roehler (a-roehler)]

Rustom wrote:
> One suggestion regarding the encoding bug:
> As everything becomes encoding/unicode many variables are changing.
> In particular emacs has moved from ASCII to UTF-8 default in emacs 23
> Likewise python 3 has revamped its entire notion of encoding.
>
> So try and change only 1 variable:
> If things were working with emacs22 + python2.x
> change only one of emacs/python and see the results.

Being here on

GNU Emacs 23.1.50.1 (i686-pc-linux-gnu, GTK+ Version 2.12.0) of 2009-10-27

and Python 2.5.1

Bug only occurs with some kind of shell-command on buffer/region.

For the moment it's not an issue.

Let's wait for next versions with newer bugs... :)

BTW patch is against python-mode.el base, not against your fix.

Afterwards it should load your fix at your machine (3.0),
while not here.

Andreas

[herbuz (herbuz)]

Hello.
I can't apply 20091102-bug-450552-exec.patch for python-mode 6.0.12
Output:
Hunk #1 FAILED at 368.
Hunk #2 succeeded at 14311 with fuzz 2 (offset 12962 lines).

changing the line

(cmd (format "execfile(r'%s') # PYTHON-MODE\n" filename)))

to
(cmd (format "exec(open(r'%s').read()) # PYTHON-MODE\n" filename))) not help.

GNU Emacs 24.2.1 (i386-mingw-nt6.1.7600)
python-mode-6.0.12

[Andreas Roehler (a-roehler)]

Am 27.10.2012 17:37, schrieb herbuz:
> Hello.
> I can't apply 20091102-bug-450552-exec.patch for python-mode 6.0.12
> Output:
> Hunk #1 FAILED at 368.
> Hunk #2 succeeded at 14311 with fuzz 2 (offset 12962 lines).
>
> changing the line
>
> (cmd (format "execfile(r'%s') # PYTHON-MODE\n" filename)))
>
> to
> (cmd (format "exec(open(r'%s').read()) # PYTHON-MODE\n" filename))) not help.
>
> GNU Emacs 24.2.1 (i386-mingw-nt6.1.7600)
> python-mode-6.0.12
>

this bug should not exist any more, so no need to apply that patch now.
Please file a new report, should anything look wrong with current trunk.