Line entry edge case error

[Stephan Peijnik]

This bug came in via bugs.debian.org and I am forwarding it to you as it seems to be an IPython issue. The Debian bug in question is http://bugs.debian.org/473018.
Could you please have a look at the bug report and comment on it? This problem seems to also affect IPython 0.9.1 which I am currently packaging for Debian.

The original problem description can be found below.

Regards,

Stephan

-- original problem description --
Interactively entering the following, allowing ipython
to automatically provide indentation:

def a(b):
  b()
 return True

causes an indentation error (as expected). I press "up"
to retrieve the history, and go back to add in the
additional space before "return". Pressing enter causes:

NameError: name 'e' is not defined

Weirder still: if True is replaced with 'q' and some value
is assigned to the variable q, ipython returns that value
as the next "out" result. Find transcript below.

The "call" b() appears to not matter (it can be replaced
with other code, for instance "b=b").

[Ville M. Vainio (villemvainio)]

Reproduced. Seems like readline only gets "e" from the input line
'return True' when you edit it. I'll investigate.

On Thu, Oct 30, 2008 at 8:37 AM, Stephan Peijnik <email address hidden> wrote:
> Public bug reported:
>
> This bug came in via bugs.debian.org and I am forwarding it to you as it seems to be an IPython issue. The Debian bug in question is http://bugs.debian.org/473018.
> Could you please have a look at the bug report and comment on it? This problem seems to also affect IPython 0.9.1 which I am currently packaging for Debian.

--
Ville M. Vainio
http://tinyurl.com/vainio

[Ville M. Vainio (villemvainio)]

On Wed, Nov 5, 2008 at 12:01 PM, Ville M. Vainio <email address hidden> wrote:

> Reproduced. Seems like readline only gets "e" from the input line
> 'return True' when you edit it. I'll investigate.

It appears that the problem is all the way down in readline - we get
the "e" from raw_input(). It seems this can't be fixed in ipython
side, unfortunately.

--
Ville M. Vainio
http://tinyurl.com/vainio

[Stephan Peijnik (speijnik)]

OK, I'm thinking about marking this bug (in dbts) as "wontfix" then.

[Stephan Peijnik (speijnik)]

I know it's been a while, but I again tested against this bug and I'm not sure it's really readline-related.

Without doing a check on this I expect that ipython, and not readline, executes the python code entered. If that's the case it sure looks like ipython causing the problem, and not readline.

This is the result of another test of mine:

--begin test--

In [1]: def a(b):
   ...: q=5
   ...: return q
------------------------------------------------------------
IndentationError: unindent does not match any outer indentation level (<ipython console>, line 3)

In [2]: def a(b):
    q=5
   return q
---------------------------------------------------------------------------
NameError Traceback (most recent call last)

/home/sp/<ipython console> in <module>()

NameError: name 'q' is not defined

In [3]: def a(b):
    q=5
   return q
------------------------------------------------------------
IndentationError: unindent does not match any outer indentation level (<ipython console>, line 3)

In [6]: def a(b):
    q=5
   return q
---------------------------------------------------------------------------
NameError Traceback (most recent call last)

/home/sp/<ipython console> in <module>()

NameError: name 'q' is not defined
--end test--

Note how every command with an odd line number works correctly and how ones with an even line number do not work.
Also, I did copy&paste the complete sets of commands I entered, why does the fourth input end up at "In [6]"?

So again, I am suspecting an ipython-internal issue, rather than a readline one. Could someone please try to reproduce this?

-- Stephan

[Ville M. Vainio (villemvainio)]

On Tue, Feb 24, 2009 at 1:28 PM, Stephan Peijnik <email address hidden> wrote:

> Without doing a check on this I expect that ipython, and not readline,
> executes the python code entered. If that's the case it sure looks like
> ipython causing the problem, and not readline.

Try "%hist -r" to see what IPython thinks you actually typed.

--
Ville M. Vainio
http://tinyurl.com/vainio

[Stephan Peijnik (speijnik)]

On Tue, 2009-02-24 at 13:24 +0000, Ville M. Vainio wrote:
> On Tue, Feb 24, 2009 at 1:28 PM, Stephan Peijnik <email address hidden>
> wrote:
>
> > Without doing a check on this I expect that ipython, and not readline,
> > executes the python code entered. If that's the case it sure looks like
> > ipython causing the problem, and not readline.
>
> Try "%hist -r" to see what IPython thinks you actually typed.

Okay, looking at this makes me think it's readline again, thanks for the
hint.

--begin: %hist -r output--
1:
def a(b):
    q=5
   return q
2: q
3: %hist -r
--end: %hist -r output--

-- Stephan

[Stephan Peijnik (speijnik)]

On Tue, 2009-02-24 at 13:24 +0000, Ville M. Vainio wrote:
> On Tue, Feb 24, 2009 at 1:28 PM, Stephan Peijnik <email address hidden>
> wrote:
>
> > Without doing a check on this I expect that ipython, and not readline,
> > executes the python code entered. If that's the case it sure looks like
> > ipython causing the problem, and not readline.
>
> Try "%hist -r" to see what IPython thinks you actually typed.

Oh, and yet another test:

--begin: test--
In [1]: def a(b):
   ...: q=5
   ...: return q
------------------------------------------------------------
IndentationError: unindent does not match any outer indentation level
(<ipython console>, line 3)

In [2]: %hist -r
Executing: def a(b):
    q=5
   return q
------------------------------------------------------------
IndentationError: unindent does not match any outer indentation level
(<ipython console>, line 3)
--end: test--

Now this looks even weirder than my previous test with '%hist -r' as you
suggested.

-- Stephan

[Ville M. Vainio (villemvainio)]

Ok, this makes me think it may be ipython bug after all.

I can't debug this properly now, but iplib.py has this snippet in raw_input:

                if self.has_readline: # and some config option is set?
                    try:
                        histlen = self.readline.get_current_history_length()
                        if histlen > 1:
                            newhist = self.input_hist_raw[-1].rstrip()
                            self.readline.remove_history_item(histlen-1)
                            self.readline.replace_history_item(histlen-2,
                                            newhist.encode(self.stdin_encoding))
                    except AttributeError:
                        pass # re{move,place}_history_item are new in 2.4.

Try disabling that code (if 0:). It's used to support multiline history editing, which may be buggy.

[Stephan Peijnik (speijnik)]

I disabled multiline history editing, as you suggested, but that doesn't change the result.

I am somewhat wondering why "%hist -r" executes the piece of code in its history again, as it shouldn't do that.
Right now I am wondering whether this bug isn't more severe than I initially thought. If %hist executes commands this could cause bad things to happen (think of a call to os.remove being executed again, after changing the working directory).

[Ville M. Vainio (villemvainio)]

On Fri, Feb 27, 2009 at 7:56 PM, Stephan Peijnik <email address hidden> wrote:

> I am somewhat wondering why "%hist -r" executes the piece of code in its history again, as it shouldn't do that.

It doesn't - it's just that readline is feeding our raw_input
characters that it shouldn't. You should run %hist -r (and %hist
without -r) to get a hint about what's happening.

> Right now I am wondering whether this bug isn't more severe than I initially thought. If %hist executes commands this could cause bad things to
happen (think of a call to os.remove being executed again, after
changing the working directory).

It's certainly a very serious issue - unfortunately it's something we
probably can't easily get rid of, since we are blindly relying on the
hope that we get sensible input from the terminal.

--
Ville M. Vainio
http://tinyurl.com/vainio

[Brian Granger (ellisonbg)]

Ville,

Have you decided that this is a readline issue? If so, can we close this ticket?