commit fails to invoke external editor in non-ascii directory

[helios]

Hello,

I am just posting my bug report to the mailinglist here. I will forward up with the requested informations in a moment.

Hello!

Now that it happens again I do so as advised and send this error message.

It happens on commit after I changed a file and added some more.

---------------------------------------------------------------------
martin@shambala:> bzr commit
bzr: ERROR: exceptions.UnicodeDecodeError: 'ascii' codec can't decode byte
0xfc in position 30: ordinal not in range(128)

Traceback (most recent call last):
  File "/usr/lib/python2.4/site-packages/bzrlib/commands.py", line 626, in
run_bzr_catch_errors
    return run_bzr(argv)
  File "/usr/lib/python2.4/site-packages/bzrlib/commands.py", line 588, in
run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.4/site-packages/bzrlib/commands.py", line 292, in
run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.4/site-packages/bzrlib/builtins.py", line 1853,
in run
    message = edit_commit_message(template)
  File "/usr/lib/python2.4/site-packages/bzrlib/msgeditor.py", line 98, in
edit_commit_message
    tmp_fileno, msgfilename = tempfile.mkstemp(prefix='bzr_log.',
dir=u'.')
  File "tempfile.py", line 302, in mkstemp
    return _mkstemp_inner(dir, prefix, suffix, flags)
  File "tempfile.py", line 238, in _mkstemp_inner
    return (fd, _os.path.abspath(file))
  File "posixpath.py", line 403, in abspath
    path = join(os.getcwd(), path)
  File "posixpath.py", line 65, in join
    path += '/' + b
UnicodeDecodeError: 'ascii' codec can't decode byte 0xfc in position 30:
ordinal not in range(128)

bzr 0.13.0 on python 2.4.4.final.0 (linux2)
arguments: ['/usr/bin/bzr', 'commit']

** please send this report to <email address hidden>
martin@shambala:~/Geburtstag 2007/Wünsche#3> bzr commit
bzr: ERROR: exceptions.UnicodeDecodeError: 'ascii' codec can't decode byte
0xfc in position 30: ordinal not in range(128)

Traceback (most recent call last):
  File "/usr/lib/python2.4/site-packages/bzrlib/commands.py", line 626, in
run_bzr_catch_errors
    return run_bzr(argv)
  File "/usr/lib/python2.4/site-packages/bzrlib/commands.py", line 588, in
run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.4/site-packages/bzrlib/commands.py", line 292, in
run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.4/site-packages/bzrlib/builtins.py", line 1853,
in run
    message = edit_commit_message(template)
  File "/usr/lib/python2.4/site-packages/bzrlib/msgeditor.py", line 98, in
edit_commit_message
    tmp_fileno, msgfilename = tempfile.mkstemp(prefix='bzr_log.',
dir=u'.')
  File "tempfile.py", line 302, in mkstemp
    return _mkstemp_inner(dir, prefix, suffix, flags)
  File "tempfile.py", line 238, in _mkstemp_inner
    return (fd, _os.path.abspath(file))
  File "posixpath.py", line 403, in abspath
    path = join(os.getcwd(), path)
  File "posixpath.py", line 65, in join
    path += '/' + b
UnicodeDecodeError: 'ascii' codec can't decode byte 0xfc in position 30:
ordinal not in range(128)

bzr 0.13.0 on python 2.4.4.final.0 (linux2)
arguments: ['/usr/bin/bzr', 'commit']

** please send this report to <email address hidden>
---------------------------------------------------------------------

Happens on Debian (mostly etch):

---------------------------------------------------------------------
martin@shambala:~> dpkg -l | egrep "(bzr|python)" | cut -c1-70
ii bzr 0.13-0.1
ii bzr-svn 0.2-1
ii ipython 0.7.2-5
ii libboost-python1.33.1 1.33.1-10
ii python 2.4.4-2
ii python-apt 0.6.19
ii python-cairo 1.2.0-1
ii python-celementtree 1.0.5-8
ii python-central 0.5.12
ii python-crypto 2.0.1+dfsg1-1.2
ii python-dbus 0.71-3
ii python-elementtree 1.2.6-10
ii python-glade2 2.8.6-8
ii python-gmenu 2.16.1-2
ii python-gnome2 2.12.4-6
ii python-gnome2-desktop 2.14.0-3
ii python-gnome2-extras 2.14.2-1+b1
ii python-gtk2 2.8.6-8
ii python-imaging 1.1.5-11
ii python-imaging-tk 1.1.5-11
ii python-libxml2 2.6.27.dfsg-1
ii python-minimal 2.4.4-2
ii python-numeric 24.2-7
ii python-paramiko 1.5.2-0.1
ii python-pycurl 7.15.5-1
ii python-pygame 1.7.1release-4.1
ii python-pylibacl 0.2.1-3.1
ii python-pyorbit 2.0.1-5
ii python-pysqlite2 2.3.2-1
ii python-pyxattr 0.2.1-1.1
ii python-qt3 3.16-1.2
ii python-sip4 4.4.5-4
ii python-subversion 1.4.2dfsg1-2
ii python-support 0.5.6
ii python-tk 2.4.4-1
ii python-uno 2.0.4.dfsg.2-4
ii python-urwid 0.9.7.1-1
ii python-xml 0.8.4-6
ii python2.4 2.4.4-2
ii python2.4-minimal 2.4.4-2
---------------------------------------------------------------------

Also with bzr 0.14:

---------------------------------------------------------------------
bzr: ERROR: exceptions.UnicodeDecodeError: 'ascii' codec can't decode byte
0xfc in position 30: ordinal not in range(128)

Traceback (most recent call last):
  File "/usr/lib/python2.4/site-packages/bzrlib/commands.py", line 650, in
run_bzr_catch_errors
    return run_bzr(argv)
  File "/usr/lib/python2.4/site-packages/bzrlib/commands.py", line 612, in
run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.4/site-packages/bzrlib/commands.py", line 304, in
run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.4/site-packages/bzrlib/builtins.py", line 1919,
in run
    reporter=reporter)
  File "/usr/lib/python2.4/site-packages/bzrlib/decorators.py", line 51,
in write_locked
    return unbound(self, *args, **kwargs)
  File "/usr/lib/python2.4/site-packages/bzrlib/mutabletree.py", line 126,
in commit
    revprops=revprops, *args, **kwargs)
  File "/usr/lib/python2.4/site-packages/bzrlib/commit.py", line 307, in
commit
    message = message_callback(self)
  File "/usr/lib/python2.4/site-packages/bzrlib/builtins.py", line 1896,
in get_message
    my_message = edit_commit_message(template)
  File "/usr/lib/python2.4/site-packages/bzrlib/msgeditor.py", line 98, in
edit_commit_message
    tmp_fileno, msgfilename = tempfile.mkstemp(prefix='bzr_log.',
dir=u'.')
  File "tempfile.py", line 302, in mkstemp
    return _mkstemp_inner(dir, prefix, suffix, flags)
  File "tempfile.py", line 238, in _mkstemp_inner
    return (fd, _os.path.abspath(file))
  File "posixpath.py", line 403, in abspath
    path = join(os.getcwd(), path)
  File "posixpath.py", line 65, in join
    path += '/' + b
UnicodeDecodeError: 'ascii' codec can't decode byte 0xfc in position 30:
ordinal not in range(128)

bzr 0.14.0 on python 2.4.4.final.0 (linux2)
arguments: ['/usr/bin/bzr', 'commit']

** please send this report to <email address hidden>
---------------------------------------------------------------------

With this repostory:

---------------------------------------------------------------------
martin@shambala:~/Geburtstag 2007/Wünsche#3> bzr info
Location:
  branch root: file:///home/martin/Geburtstag%202007/

Format:
       control: Meta directory format 1
  working tree: Working tree format 3
        branch: Branch format 5
    repository: Knit repository format 1

In the working tree:
         6 unchanged
         1 modified
         2 added
         0 removed
         0 renamed
         5 unknown
         0 ignored
         1 versioned subdirectory

Branch history:
         5 revisions
        19 days old
   first revision: Sun 2007-01-14 18:42:47 +0100
  latest revision: Thu 2007-01-18 22:56:48 +0100

Revision store:
         5 revisions
        89 KiB
---------------------------------------------------------------------

And this locale setting:

---------------------------------------------------------------------
martin@shambala:~> locale
LANG=de_DE@euro
LC_CTYPE="de_DE@euro"
LC_NUMERIC="de_DE@euro"
LC_TIME="de_DE@euro"
LC_COLLATE="de_DE@euro"
LC_MONETARY="de_DE@euro"
LC_MESSAGES="de_DE@euro"
LC_PAPER="de_DE@euro"
LC_NAME="de_DE@euro"
LC_ADDRESS="de_DE@euro"
LC_TELEPHONE="de_DE@euro"
LC_MEASUREMENT="de_DE@euro"
LC_IDENTIFICATION="de_DE@euro"
LC_ALL=
---------------------------------------------------------------------

All the files I want to update / add are same locale.

I have no time for further investigation right now or to write a bug
report... so this will have to do it for now.

I worked around the bug by issuing:

bzr mv Wünsche Wuensche

(i.e. removing the umlaut from the directory name)

Regards,
--
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7

[helios (martin-lichtvoll)]

Answer from John Arbash Meinel:

> Martin Steigerwald wrote:
> > Please also CC to my personal address as I am not subcribed to list.
> >
> >
> > Hello!
> >
> > Now that it happens again I do so as advised and send this error
> > message.
> >
> > It happens on commit after I changed a file and added some more.
>
> hmm...
>
> Well, as a workaround you can use 'bzr commit -m "message"' to avoid
> this bug, but certainly we need to fix this.
>
> > File "/usr/lib/python2.4/site-packages/bzrlib/msgeditor.py", line
> > 98, in edit_commit_message
> > tmp_fileno, msgfilename = tempfile.mkstemp(prefix='bzr_log.',
> > dir=u'.')
>
> ...
>
> v- It seems like os.path.abspath() is having a problem with getcwd()
> since it is thinking in terms of 8-bit string path, while we are
> thinking in terms of unicode path.
>
> > File "tempfile.py", line 238, in _mkstemp_inner
> > return (fd, _os.path.abspath(file))
> > File "posixpath.py", line 403, in abspath
> > path = join(os.getcwd(), path)
> > File "posixpath.py", line 65, in join
> > path += '/' + b
> > UnicodeDecodeError: 'ascii' codec can't decode byte 0xfc in position
> > 30: ordinal not in range(128)
> >
> > All the files I want to update / add are same locale.
> >
> > I have no time for further investigation right now or to write a bug
> > report... so this will have to do it for now.
> >
> > I worked around the bug by issuing:
> >
> > bzr mv Wünsche Wuensche
> >
> > (i.e. removing the umlaut from the directory name)
> >
> > Regards,
>
> The other way of avoiding this bug is to cd to the root of your project
> and run commit from there, rather than running commit in a
> subdirectory.
>
> The specific bug is in python's 'tempfile.mkstemp' function. Which
> doesn't seem to handle non-ascii paths very well.
>
> I wonder what python is making of your locale, could you put the
> attached file into ~/.bazaar/plugins/
>
> And then run a bzr command ('bzr rocks' is fine)

> It should print out what encoding bzr thinks you are running in.

Hello John,

bzr seems to detect the right encoding:

martin@shambala:Zeit/Bazaar-Tests/Bazaar> bzr rocks
bzr-svn is not up to date with installed bzr version 0.14.0.
There should be a newer version available, e.g. 0.14.
Bzr using encoding: 'ISO-8859-15'
it sure does!

> Otherwise, my suggested patch is the simple one, which may mean things
> don't work properly in certain paths on Windows
>
> === modified file 'bzrlib/msgeditor.py'
> --- bzrlib/msgeditor.py 2006-11-11 19:25:25 +0000
> +++ bzrlib/msgeditor.py 2007-02-07 16:53:59 +0000
> @@ -95,7 +95,7 @@
>
> msgfilename = None
> try:
> - tmp_fileno, msgfilename = tempfile.mkstemp(prefix='bzr_log.',
> dir=u'.')
> + tmp_fileno, msgfilename = tempfile.mkstemp(prefix='bzr_log.',
> dir='.')
> msgfile = os.close(tmp_fileno)
> if infotext is not None and infotext != "":
> hasinfo = True
>
> Or the attached one which is more involved, but cleans up a bunch of
> other stuff. The big difference is just that it uses:
> dir=osutils.getcwd()
>
> Which returns the full unicode path to the current directory, which
> should just be a pass-through for os.abspath(), r...

[helios (martin-lichtvoll)]

As written above this gives me an error message.

[Alexander Belchenko (bialix)]

things are worse on win32

[Alexander Belchenko (bialix)]

On win32 bzr in such situation fails in subprocess module:

Traceback (most recent call last):
  File "bzrlib\commands.pyc", line 650, in run_bzr_catch_errors
  File "bzrlib\commands.pyc", line 612, in run_bzr
  File "bzrlib\commands.pyc", line 304, in run_argv_aliases
  File "bzrlib\builtins.pyc", line 1919, in run
  File "bzrlib\decorators.pyc", line 51, in write_locked
  File "bzrlib\mutabletree.pyc", line 126, in commit
  File "bzrlib\commit.pyc", line 307, in commit
  File "bzrlib\builtins.pyc", line 1896, in get_message
  File "bzrlib\msgeditor.pyc", line 109, in edit_commit_message
  File "bzrlib\msgeditor.pyc", line 61, in _run_editor
  File "subprocess.pyc", line 443, in call
  File "subprocess.pyc", line 593, in __init__
  File "subprocess.pyc", line 793, in _execute_child
UnicodeEncodeError: 'ascii' codec can't encode characters in position 22-27: ordinal not in range(128)

bzr 0.14.0 on python 2.5.0.final.0 (win32)
arguments: ['C:\\Program Files\\Bazaar\\bzr.EXE', 'ci']

[Alexander Belchenko (bialix)]

The problem in another place too: if I set up editor in bazaar.conf then _get_editor() will always return unicode path. And therefore if filename contains non-ascii (but not unicode) it also will fail in subprocess module in list2cmdline() function.

[John A Meinel (jameinel)]

One interesting possible workaround.

    tmp_fileno, msgfilename = tempfile.mkstemp(prefix='bzr_log.',
                                               dir=u'.',
                                               text=True)
    msgfile = os.fdopen(tmp_fileno, 'w')

When we create the file we know that we are creating it in the current working directory.

mkstemp always returns the full path of the new file. But if we know we used "dir='.'" then we can turn around and use:
msgfilename = osutils.basename(msgfilename)

Would someone be able to try this?

[Alexander Belchenko (bialix)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

John A Meinel пишет:
> One interesting possible workaround.
>
> tmp_fileno, msgfilename = tempfile.mkstemp(prefix='bzr_log.',
> dir=u'.',
> text=True)
> msgfile = os.fdopen(tmp_fileno, 'w')
>
> When we create the file we know that we are creating it in the current
> working directory.
>
> mkstemp always returns the full path of the new file. But if we know we used "dir='.'" then we can turn around and use:
> msgfilename = osutils.basename(msgfilename)
>
> Would someone be able to try this?

Yes, I'm able, I'll try it tonight.
But I think that we anyway should write non-ascii test for this.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHPct/zYr338mxwCURAte/AJ9VrpO+RatAN6n5wV3thnXE1rlywgCfXFIG
dq3vWfm1CeyfolVCyV4sQnw=
=NT79
-----END PGP SIGNATURE-----

[John A Meinel (jameinel)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Alexander Belchenko wrote:
> John A Meinel ?8H5B:
>> One interesting possible workaround.
>
>> tmp_fileno, msgfilename = tempfile.mkstemp(prefix='bzr_log.',
>> dir=u'.',
>> text=True)
>> msgfile = os.fdopen(tmp_fileno, 'w')
>
>> When we create the file we know that we are creating it in the current
>> working directory.
>
>> mkstemp always returns the full path of the new file. But if we know we used "dir='.'" then we can turn around and use:
>> msgfilename = osutils.basename(msgfilename)
>
>> Would someone be able to try this?
>
> Yes, I'm able, I'll try it tonight.
> But I think that we anyway should write non-ascii test for this.

Sure, but it will only fail on platforms that can't pass around the path on the
command line.

So I agree that we need a test, but I also need someone to run that test on a
platform where it will fail.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHPc3jJdeBCYSNAAMRAr/mAKDZ9JuZ4oxFP0UD88yohqfPUHpyFQCgzTBJ
BZ7rSZw0moEDSh8yWJSVfj8=
=EuZN
-----END PGP SIGNATURE-----

[Dan Watkins (oddbloke)]

I've just tested John's suggestion, and, with an entirely non-rigourous test, it does seem to work. I haven't looked around for potential side-effects, which I will do now.

[Alexander Belchenko (bialix)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

John A Meinel пишет:
> Alexander Belchenko wrote:
>> John A Meinel ?8H5B:
>>> One interesting possible workaround.
>>> tmp_fileno, msgfilename = tempfile.mkstemp(prefix='bzr_log.',
>>> dir=u'.',
>>> text=True)
>>> msgfile = os.fdopen(tmp_fileno, 'w')
>>> When we create the file we know that we are creating it in the current
>>> working directory.
>>> mkstemp always returns the full path of the new file. But if we know we used "dir='.'" then we can turn around and use:
>>> msgfilename = osutils.basename(msgfilename)
>>> Would someone be able to try this?
>> Yes, I'm able, I'll try it tonight.
>> But I think that we anyway should write non-ascii test for this.
>
> Sure, but it will only fail on platforms that can't pass around the path on the
> command line.
>
> So I agree that we need a test, but I also need someone to run that test on a
> platform where it will fail.

Hmmm, my first thought is that you talking about Windows.
But looking at this duplicate (https://bugs.launchpad.net/bugs/163123)
I see that it's occurs and on Linux too.
What do you mean when you said: "I also need someone to run that test on a
> platform where it will fail"?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHPdG2zYr338mxwCURAu9VAJ48REVDqGVN/SJw92sNbGJBKBQuPACgjkvZ
5Q4a0FxPQUmbe1cyv8h+zPY=
=v23Y
-----END PGP SIGNATURE-----

[GrdScarabe (grdscarabe)]

Hi, here is what I did try ... from what I understood of John A Meinel workaround :

--- msgeditor.py 2007-11-16 18:32:15.000000000 +0100
+++ /usr/lib/python2.5/site-packages/bzrlib/msgeditor.py 2007-11-16 18:32:57.000000000 +0100
@@ -172,6 +172,7 @@
                                                dir=u'.',
                                                text=True)
     msgfile = os.fdopen(tmp_fileno, 'w')
+ msgfilename = osutils.basename(msgfilename)
     try:
         if start_message is not None:
             msgfile.write("%s\n" % start_message.encode(

Maybe I misunderstood the workaround, so please tell me, but with that modification in the code, the bug is still present :

$ bzr commit
added somefile
bzr: ERROR: exceptions.UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 6: ordinal not in range(128)

Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 817, in run_bzr_catch_errors
    return run_bzr(argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 779, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 477, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.5/site-packages/bzrlib/builtins.py", line 2283, in run
    reporter=reporter, revprops=properties)
  File "/usr/lib/python2.5/site-packages/bzrlib/decorators.py", line 165, in write_locked
    return unbound(self, *args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/workingtree_4.py", line 246, in commit
    result = WorkingTree3.commit(self, message, revprops, *args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/decorators.py", line 165, in write_locked
    return unbound(self, *args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/mutabletree.py", line 207, in commit
    revprops=revprops, *args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/commit.py", line 311, in commit
    message = message_callback(self)
  File "/usr/lib/python2.5/site-packages/bzrlib/builtins.py", line 2260, in get_message
    my_message = edit_commit_message(template)
  File "/usr/lib/python2.5/site-packages/bzrlib/msgeditor.py", line 105, in edit_commit_message
    infotext, ignoreline, start_message)
  File "/usr/lib/python2.5/site-packages/bzrlib/msgeditor.py", line 173, in _create_temp_file_with_commit_template
    text=True)
  File "tempfile.py", line 302, in mkstemp
    return _mkstemp_inner(dir, prefix, suffix, flags)
  File "tempfile.py", line 238, in _mkstemp_inner
    return (fd, _os.path.abspath(file))
  File "posixpath.py", line 403, in abspath
    path = join(os.getcwd(), path)
  File "posixpath.py", line 65, in join
    path += '/' + b
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 6: ordinal not in range(128)

bzr 0.90.0 on python 2.5.1.final.0 (linux2)
arguments: ['/usr/bin/bzr', 'commit']

** please send this report to <email address hidden>

[Dan Watkins (oddbloke)]

The attached branch will contain, once it finished uploading, a fix and a basic test. I'll send it to the ML for review in a few minutes.

[GrdScarabe (grdscarabe)]

Hi,

I've applied the following patch http://codebrowse.launchpad.net/~daniel-thewatkins/bzr/84043/revision/3006 to my installation of Bazaar. And it seems to work.

What I did :
 - mkdir /tmp/Thèse
 - cd /tmp/Thèse
 - bzr init
 - echo "just a test" > test
 - bzr add test
 - bzr commit test

Before the fix was applied, Bazaar crashed with the non-ascii error, now it just works as expected. Thanks !

I can write a test if you point me out a HOWTO to do so ;)

[Dan Watkins (oddbloke)]

The tests have already been improved (and I'm currently pushing the more up-to-date branch). :)

[Dan Watkins (oddbloke)]

Fixed in bzr.dev r3013.