Ubuntu

locale._parse_localename fails when localename does not contain encoding information (was: alacarte crashed with ValueError in _parse_localename() )

Reported by Zoran Savic on 2008-04-27
116
This bug affects 16 people
Affects Status Importance Assigned to Milestone
Python
New
Unknown
Ubuntu Translations
Medium
Unassigned
python2.6 (Ubuntu)
Medium
Unassigned
Karmic
Undecided
Unassigned

Bug Description

Binary package hint: alacarte

"Main Menu" worked with first ubuntu 8.04 installation. After serbian language pack installed program did not working any more and in the terminal generate errors such like this:

user@computer:~$ alacarte
Traceback (most recent call last):
  File "/usr/bin/alacarte", line 36, in <module>
    main()
  File "/usr/bin/alacarte", line 32, in main
    app = MainWindow(datadir, version, sys.argv)
  File "/usr/lib/python2.5/site-packages/Alacarte/MainWindow.py", line 49, in __init__
    self.editor = MenuEditor()
  File "/usr/lib/python2.5/site-packages/Alacarte/MenuEditor.py", line 35, in __init__
    self.locale = locale.getdefaultlocale()[0]
  File "/usr/lib/python2.5/locale.py", line 443, in getdefaultlocale
    return _parse_localename(localename)
  File "/usr/lib/python2.5/locale.py", line 375, in _parse_localename
    raise ValueError, 'unknown locale: %s' % localename
ValueError: unknown locale: sr_ME
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/apport_python_hook.py", line 78, in apport_excepthook
    report_file = open(pr_filename, 'wt')
IOError: [Errno 13] Permission denied: '/var/crash/_usr_bin_alacarte.1000.crash'

Original exception was:
Traceback (most recent call last):
  File "/usr/bin/alacarte", line 36, in <module>
    main()
  File "/usr/bin/alacarte", line 32, in main
    app = MainWindow(datadir, version, sys.argv)
  File "/usr/lib/python2.5/site-packages/Alacarte/MainWindow.py", line 49, in __init__
    self.editor = MenuEditor()
  File "/usr/lib/python2.5/site-packages/Alacarte/MenuEditor.py", line 35, in __init__
    self.locale = locale.getdefaultlocale()[0]
  File "/usr/lib/python2.5/locale.py", line 443, in getdefaultlocale
    return _parse_localename(localename)
  File "/usr/lib/python2.5/locale.py", line 375, in _parse_localename
    raise ValueError, 'unknown locale: %s' % localename
ValueError: unknown locale: sr_ME

ProblemType: Crash
Architecture: i386
Date: Sun Apr 27 23:08:17 2008
DistroRelease: Ubuntu 8.04
ExecutablePath: /usr/bin/alacarte
InterpreterPath: /usr/bin/python2.5
Package: alacarte 0.11.5-0ubuntu1
PackageArchitecture: all
ProcCmdline: /usr/bin/python -OOt /usr/bin/alacarte
ProcEnviron:
 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/X11R6/bin
 LANG=sr_ME
 SHELL=/bin/bash
PythonArgs: ['/usr/bin/alacarte']
SourcePackage: alacarte
Title: alacarte crashed with ValueError in _parse_localename()
Uname: Linux 2.6.24-16-generic i686
UserGroups:

-------------------

INFORMATION FOR SRU CONSIDERATION FOLLOWS:

1. A statement explaining the impact of the bug on users and justification for backporting the fix to the stable release:

This bug prevents users in some locales (notable Serbia and Antigua, sr_ME and en_AG) from running locale-aware Python software in Ubuntu, including alacarte and software-center. The latter is particularly problematic, as it is likely to affect many novice users.

The proposed fix is small, clear, in the upstream bugtracker already, and is isolated to one function in one file, and has been used with success by users of Ubuntu Karmic in both the mentioned locales from a PPA.

This appears to qualify as a bug ... "which represents a severe regression from the previous release of Ubuntu. This includes packages which are totally unusable, like being uninstallable or crashing on startup."

It could perhaps be argued that these locales did not exist in earlier versions so this is not technically a regression?

2. An explanation of how the bug has been addressed in the development branch, including the relevant version numbers of packages modified in order to implement the fix.

An attached debdiff for Ubuntu Lucid is attached to this bug report. It awaits sponsorship.

3. A minimal patch applicable to the stable version of the package.

See attached debdiffs. The original patch is at http://bugs.python.org/file14881/locale.py-parselocale-patch.diff

4. Detailed instructions how to reproduce the bug.

TEST CASE:

4.1 Install Ubuntu 9.10 Karmic Desktop (i386 and amd64 have both been tested).

4.2 At the GDM login screen, set the locale to English/Antigua and log in (start a session) in that locale.

4.3 Click on Applications -> Ubuntu Software Center

4.4 To see the Python traceback, open a Terminal window and run software-center from it. A smaller test case would be to run

  LANG=en_AG python2.6 -c "import locale ; print locale.getdefaultlocale()"

and testing all locales in this way can therefore be done by running

  sudo apt-get install ^language-pack-\* -y # Install all locales
  for i in $(locale -a)
  do
    echo $i
    LANG=$i python2.6 -c "import locale ; print locale.getdefaultlocale()" || break
  done

4.5 If desired, obtain the patched python2.6 package(s) from the PPA at https://launchpad.net/~jmarsden/+archive/ppa/ and repeat the tests. software-center will now run and operate correctly.

5. A discussion of the regression potential of the patch and how users could get inadvertently affected.

This patch has been tested for every locale in Ubuntu Karmic, see 4.4 above, but not every Python app has been tested with this patch. So, by definition, there is a non-zero regression potential.

Nevertheless, the patch is so small, and can be reviewed by anyone who reads Python for correctness. It affects just one function, and then only in locales without a period in them (there are few such locales). On that basis, it is hard to see how it can really introduce a regression in practice. This issue has been around for several versions of Ubuntu, so the current behaviour with the patch in place (for locale-aware Python applications at least) is an obvious improvement over earlier behaviour -- the applications actually run. So far, no Python apps have been found which this patch affects negatively. The patch has been tested in several commonly used locales, including en_US.UTF-8, de_DE.utf8, es_ES.utf8 and fr_FR.utf8, as well as the affected locales sr_ME and en_AG.

-------------------

Zoran Savic (zorans-ordinatio) wrote :
Pedro Villavicencio (pedro) wrote :

thanks for the report, is this still an issue with intrepid?

Changed in alacarte:
assignee: nobody → desktop-bugs
importance: Undecided → Medium
status: New → Incomplete
Pedro Villavicencio (pedro) wrote :

We are closing this bug report because it lacks the information we need to investigate the problem, as described in the previous comments. Please reopen it if you can give us the missing information, and don't hesitate to submit bug reports in the future. To reopen the bug report you can click on the current status, under the Status column, and change the Status back to New. Thanks again!.

Changed in alacarte:
status: Incomplete → Invalid
Nicksha (nicksha) wrote :

The bug has reappeared in Jaunty beta. Alacarte worked in Intrepid.

Changed in alacarte:
status: Invalid → New
Nicksha (nicksha) wrote :

I am also using serbian translation (the unknown locale it complains about is sr_RS). Alacarte works in terminal if the command is preceded with LANG=en. Don't know if it's relevant, since Alacarte is a Python program, but I'm using the 64bit Ubuntu variant.

Nicksha (nicksha) wrote :

It appears that this is related to an upstream Python bug (http://bugs.python.org/issue6895). There is a patch for it. The problem is that the serbian locale doesn't have encoding information in it's name (instead of "sr_RS.UTF-8" it's just "sr_RS"), and _parse_localename() function doesn't accept that.

Petar Vasić (p-vasic) wrote :

According to last comment, and numerous duplicates, this seems to be an issue with python parsing locales without encoding information.

summary: - alacarte crashed with ValueError in _parse_localename()
+ locale._parse_localename fails when localename does not contain encoding
+ information (was: alacarte crashed with ValueError in
+ _parse_localename() )
affects: alacarte (Ubuntu) → python2.6 (Ubuntu)
Changed in python2.6 (Ubuntu):
status: New → Confirmed
Jonathan Marsden (jmarsden) wrote :

The upstream patch to locale.py suggested by nicsabacovic does solve the problem when applied to locale.py, at least in Ubuntu 9.10 Karmic. After applying it (by hand, since it is for python 3.x and does not apply to 2.6 using patch), alacarte runs in the sr_RS@latin and sr_RS locales.

debdiff (and packages in my PPA for brave people to test!) will follow shortly, for Jaunty and Karmic, I hope.

Jonathan

Changed in python2.6 (Ubuntu):
status: Confirmed → In Progress
Jonathan Marsden (jmarsden) wrote :

Attached is my initial attempt at a debdiff for the Jaunty python2.6 package.

This will hopefully build in my PPA at https://launchpad.net/~jmarsden/+archive/ppa/ overnight. Brave souls who want to download and test the updated Python packages from there are very much welcome to do so.

Jonathan

Nicksha (nicksha) wrote :

Thanks for the effort, Jonathan! I'll be sure to check it out.

Don't know if it's important, but all the version numbers in the debdiff are 2.6.2 (Jaunty's version), and the current vesrion in Karmic is 2.6.4.

Jonathan Marsden (jmarsden) wrote :

> Don't know if it's important, but all the version numbers in the debdiff are 2.6.2 (Jaunty's version),
> and the current vesrion in Karmic is 2.6.4.

Yes, here is a separate debdiff for the Karmic version. This one is building in my PPA now.

Jonathan

Nicksha (nicksha) wrote :

With the updated Python packages from your PPA, Alacarte now starts with no errors or warning messages of any kind. Same applies to software-center.

Thanks!

Adi Roiban (adiroiban) on 2009-11-11
Changed in ubuntu-translations:
status: New → In Progress
importance: Undecided → Medium
tags: added: locale

"Thanks Jonathan, the package from your ppa fixes errors I've been seeing on my Ubuntu Karmic installation."

David Planella (dpm) wrote :

Jonathan,

Thanks for the great work. If you think the fix is ready for release, it might be worth considering it for a stable release update (SRU).

Do you (or anyone else) think you could follow the https://wiki.ubuntu.com/StableReleaseUpdates procedure so that this fix can be considered for Karmic?

Thanks!

Download full text (3.4 KiB)

yes , i think so ....

2009/11/11 David Planella <email address hidden>

> Jonathan,
>
> Thanks for the great work. If you think the fix is ready for release, it
> might be worth considering it for a stable release update (SRU).
>
> Do you (or anyone else) think you could follow the
> https://wiki.ubuntu.com/StableReleaseUpdates procedure so that this fix
> can be considered for Karmic?
>
> Thanks!
>
> --
> locale._parse_localename fails when localename does not contain encoding
> information (was: alacarte crashed with ValueError in _parse_localename() )
> https://bugs.launchpad.net/bugs/223281
> You received this bug notification because you are a direct subscriber
> of the bug.
>
> Status in Ubuntu Translations (internationalization and localization): In
> Progress
> Status in “python2.6” package in Ubuntu: In Progress
>
> Bug description:
> Binary package hint: alacarte
>
> "Main Menu" worked with first ubuntu 8.04 installation. After serbian
> language pack installed program did not working any more and in the terminal
> generate errors such like this:
>
> user@computer:~$ alacarte
> Traceback (most recent call last):
> File "/usr/bin/alacarte", line 36, in <module>
> main()
> File "/usr/bin/alacarte", line 32, in main
> app = MainWindow(datadir, version, sys.argv)
> File "/usr/lib/python2.5/site-packages/Alacarte/MainWindow.py", line 49,
> in __init__
> self.editor = MenuEditor()
> File "/usr/lib/python2.5/site-packages/Alacarte/MenuEditor.py", line 35,
> in __init__
> self.locale = locale.getdefaultlocale()[0]
> File "/usr/lib/python2.5/locale.py", line 443, in getdefaultlocale
> return _parse_localename(localename)
> File "/usr/lib/python2.5/locale.py", line 375, in _parse_localename
> raise ValueError, 'unknown locale: %s' % localename
> ValueError: unknown locale: sr_ME
> Error in sys.excepthook:
> Traceback (most recent call last):
> File "/usr/lib/python2.5/site-packages/apport_python_hook.py", line 78, in
> apport_excepthook
> report_file = open(pr_filename, 'wt')
> IOError: [Errno 13] Permission denied:
> '/var/crash/_usr_bin_alacarte.1000.crash'
>
> Original exception was:
> Traceback (most recent call last):
> File "/usr/bin/alacarte", line 36, in <module>
> main()
> File "/usr/bin/alacarte", line 32, in main
> app = MainWindow(datadir, version, sys.argv)
> File "/usr/lib/python2.5/site-packages/Alacarte/MainWindow.py", line 49,
> in __init__
> self.editor = MenuEditor()
> File "/usr/lib/python2.5/site-packages/Alacarte/MenuEditor.py", line 35,
> in __init__
> self.locale = locale.getdefaultlocale()[0]
> File "/usr/lib/python2.5/locale.py", line 443, in getdefaultlocale
> return _parse_localename(localename)
> File "/usr/lib/python2.5/locale.py", line 375, in _parse_localename
> raise ValueError, 'unknown locale: %s' % localename
> ValueError: unknown locale: sr_ME
>
> ProblemType: Crash
> Architecture: i386
> Date: Sun Apr 27 23:08:17 2008
> DistroRelease: Ubuntu 8.04
> ExecutablePath: /usr/bin/alacarte
> InterpreterPath: /usr/bin/python2.5
> Package: alacarte 0.11.5-0ubuntu1
> PackageArchitecture: all
> ProcCmdline: /usr/bin/python -OOt /usr/bin/alacarte
> ProcEnviron:...

Read more...

Dragan (dkostadinovic) wrote :

Thanks Jonathan, the package from your ppa fixes errors I've been seeing on my Ubuntu Karmic installation

Jonathan Marsden (jmarsden) wrote :

> If you think the fix is ready for release, it might be worth considering it for a stable release update (SRU).

> Do you (or anyone else) think you could follow the https://wiki.ubuntu.com/StableReleaseUpdates
> procedure so that this fix can be considered for Karmic?

OK. I'll give it a go, possibly working on it over this coming weekend. Because the affected package is an interpreter (Python), I think the standards for what it takes to get an SRU included are very high (a mistake in a Python package would affect many users of many different Python applications!). But I'll follow the process and see what feedback we get.

michael603 (michael603) wrote :

Thanks a million for this patch :D ... another thanks from Serbia, Jonathan...

Jonathan Marsden (jmarsden) wrote :

Apparently for an SRU we need to get this change accepted into Lucid. So, here is a debdiff for the Lucid version.

Changed in python2.6 (Ubuntu):
status: In Progress → Confirmed
Changed in ubuntu-translations:
status: In Progress → Confirmed
Jonathan Marsden (jmarsden) wrote :

Per https://wiki.ubuntu.com/SponsorshipProcess I am setting this to be assigned to noone and subscribing ubuntu-main-sponsors .

Changed in python2.6 (Ubuntu):
assignee: Ubuntu Desktop Bugs (desktop-bugs) → nobody
Jonathan Marsden (jmarsden) wrote :

I have updated the bug description with the info the SRU process requests, to the best of my ability.

Jonathan

description: updated
description: updated
Changed in python:
status: Unknown → New

It seems the fix would still fail for locales which contain the @modifier, eg. sr_RS.UTF-8@latin locale, if that was used. Though, since Ubuntu only sets the locale to sr_RS@latin, it at least wouldn't crash with this fix.

Actually, I was wrong: @modifier is handled correctly there, but just the diff doesn't show it.

Jonathan Marsden (jmarsden) wrote :

Testing confirms that this code works fine for locales containing @modifier. On Ubuntu 9.10 Karmic Desktop, I ran

  # Install all available language packs (may take a while!)
  sudo apt-get install ^language-pack-\*
  # Test every locale on the system
   for i in $(locale -a) ; do echo $i ; LANG=$i python2.6 -c "import locale ; print locale.getdefaultlocale()" || break ; done

This works (runs to completion, no python exceptions) with the patched python2.6 installed.

description: updated
Martin Pitt (pitti) on 2009-12-07
Changed in python2.6 (Ubuntu):
status: Confirmed → Fix Committed
Martin Pitt (pitti) wrote :

Jonathan, I locally built your lucid python update, and the bug still happens:

$ LANG=en_AG python -c "import locale ; print locale.getdefaultlocale()"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python2.6/locale.py", line 478, in getdefaultlocale
    return _parse_localename(localename)
  File "/usr/lib/python2.6/locale.py", line 410, in _parse_localename
    raise ValueError, 'unknown locale: %s' % localename
ValueError: unknown locale: en_AG

Seems it doesn't work?

(And yes, I did create the local before: sudo locale-gen en_AG)

Changed in python2.6 (Ubuntu):
status: Fix Committed → Incomplete
Martin Pitt (pitti) wrote :

Looking at the build log it seems that the patch doesn't actually get applied?

Jonathan Marsden (jmarsden) wrote :

Martin, Thanks so much for looking at this. I noticed you used the command python, whereas was using the command python2.6 for my tests. Is the default Python in Lucid Python 3.0.x maybe? Just a guess at this point. It may be a silly mistake in my packaging, too! The Karmic one definitely works well for several end users...

I am currently creating a new Lucid VM from the daily ISO and will retest there as soon as I can and report what I find. Probably in the next hour or so. I am also on #ubuntu-motu if you want to help or talk about this that way.

Jonathan Marsden (jmarsden) wrote :

Found it; there is a separate list of patches to apply in debian/rules. I forgot to add my patch to it. I don't understand why the debian/patch/series file is not sufficient, but that's how this package seems to work.

New debdiff coming very soon!

Jonathan Marsden (jmarsden) wrote :

Debdiff for lucid that actually applies the patch attached. How embarrassing!

In mitigation, why does a debian/patch/ directory that uses dpatch have a series file in it? That's very confusing, because as far as I can tell the series file never gets used.

Jonathan Marsden (jmarsden) wrote :
Changed in python2.6 (Ubuntu):
status: Incomplete → Confirmed

Jonathan Marsden [2009-12-07 17:39 -0000]:
> Martin, Thanks so much for looking at this. I noticed you used the
> command python, whereas was using the command python2.6 for my tests.
> Is the default Python in Lucid Python 3.0.x maybe?

No, it's 2.6.

> Just a guess at this point. It may be a silly mistake in my
> packaging, too! The Karmic one definitely works well for several
> end users...

Right, I didn't test the karmic update. I guess the karmic->lucid
packaging changed a bit. The python2.6 packaging is very weird and
uses a custom patch system. I think it's simply not applied on build.

Accepted python2.6 into karmic-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Changed in python2.6 (Ubuntu Karmic):
status: New → Fix Committed
tags: added: verification-needed
Jonathan Marsden (jmarsden) wrote :

Added karmic-proposed repository per instructions, sudo apt-get upgraded, tested.

Seems to work fine for me on Ubuntu 9.10 Karmic amd64.

Specifics:

dpkg -l python2.6 show3s it installed:
ii python2.6 2.6.4-0ubuntu3 An interactive high-level object-oriented language (version 2.6)

for i in $(locale -a) ; do echo $i ; LANG=$i python -c "import locale; print locale.getdefaultlocale()" || break ; done

runs to completion, on a machine with all available language packs installed (so LOTS of locales). So does

for i in $(locale -a) ; do echo $i ; LANG=$i python -c "import locale; locale.setlocale(locale.LC_ALL, "")" || break ; done

Running software-center works as expected.

No issues observed (in admittedly fairly minimal testing).

Martin Pitt (pitti) wrote :

I also tested the karmic update with the command above and confirm that it works fine now. It also still works normally with de_DE.UTF-8 and C.

tags: added: verification-done
removed: verification-needed
Martin Pitt (pitti) wrote :

> How embarrassing!

... but not for you, just the packaging is. :-)

Martin Pitt (pitti) wrote :

Hm, doesn't build for me:

sh -e debian/patches/locales-without-encoding.dpatch -patch
debian/patches/locales-without-encoding.dpatch: 10: @DPATCH@: not found

then it fails with some other error. I think it doesn't grok the dpatch-run thing.

I massaged the patch to be in old-style dpatch format, and will do a test build now.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python2.6 - 2.6.4-1ubuntu2

---------------
python2.6 (2.6.4-1ubuntu2) lucid; urgency=low

  * debian/patches/locales-without-encoding.dpatch:
    - Allow locales with no specified encoding. (Closes LP: #223281)
 -- Jonathan Marsden <email address hidden> Thu, 26 Nov 2009 01:19:32 -0800

Changed in python2.6 (Ubuntu):
status: Confirmed → Fix Released
Martin Pitt (pitti) on 2009-12-08
Changed in ubuntu-translations:
status: Confirmed → Fix Committed
Jonathan Marsden (jmarsden) wrote :

What remains to be done to get this into Lucid? Unfortunately, I don't seem to be doing very well at getting us a really truly working Lucid patch/debdiff!

I tried to set up a Lucid VM yesterday, to test my revised Lucid debdiff with, but there were X issues going on and I never got as far as a working Lucid desktop VM to use. I sent it to my PPA, and the build failed for the same reason you saw,Martin. Odd that dpatch-edit-patch on Karmic created a dpatch file that Lucid can't handle... if that's what happened, and it looks that way at first glance..

Martin Pitt (pitti) on 2009-12-15
Changed in ubuntu-translations:
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python2.6 - 2.6.4-0ubuntu3

---------------
python2.6 (2.6.4-0ubuntu3) karmic-proposed; urgency=low

  * debian/patches/locales-without-encoding.dpatch:
    - Allow locales with no specified encoding. (LP: #223281)
 -- Jonathan Marsden <email address hidden> Sun, 08 Nov 2009 18:11:57 -0800

Changed in python2.6 (Ubuntu Karmic):
status: Fix Committed → Fix Released
Matthias Klose (doko) wrote :

reopening, the patch just hides the original problem

Changed in python2.6 (Ubuntu):
status: Fix Released → New
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python2.6 - 2.6.4-4ubuntu1

---------------
python2.6 (2.6.4-4ubuntu1) lucid; urgency=low

  * Update to 20100122, taken from the 2.6 release branch.
    - Fix DoS via XML document with malformed UTF-8 sequences (CVE_2009_3560).
      Closes: #566233.
  * python2.6-doc: Fix searching in local documentation. LP: #456025.
  * Update locale module from the trunk. LP: #223281.
  * Merge with Debian (2.6.4-4).

python2.6 (2.6.4-4) unstable; urgency=low

  * Update to 20100122, taken from the 2.6 release branch.
    - Fix DoS via XML document with malformed UTF-8 sequences (CVE_2009_3560).
      Closes: #566233.
  * Hurd fixes (Pino Toscano). Closes: #565693:
    - hurd-broken-poll.dpatch: ported from 2.5.
    - hurd-disable-nonworking-constants.dpatch: disable a few constants from
      the public API whose C counterparts are not implemented, so using them
      either always blocks or always fails (caused issues in the test suite).
    - Exclude the profiled build for hurd.
    - Disable four blocking tests from the test suite.

python2.6 (2.6.4-3) unstable; urgency=low

  * Disable the profiled build on s390, mips, mipsel.
  * Fix symbol files for kfreebsd-amd64 and sparc64.

python2.6 (2.6.4-2) unstable; urgency=low

  * Update to 20100116, taken from the 2.6 release branch.
  * Fix bashism in makesetup shell script. Closes: #530170, #530171.
  * Fix build issues on avr (Bradley Smith). Closes: #528439.
    - Configure --without-ffi.
    - Don't run lengthly tests.
  * locale.py: Update locale aliases from the 2.7 branch.
 -- Matthias Klose <email address hidden> Fri, 22 Jan 2010 11:37:29 +0100

Changed in python2.6 (Ubuntu):
status: New → Fix Released

Bug is still there for me, with python2.6.5 installed.

A temporary workaround. Type:

export LANG=en_US

... do your python stuff ...
When finished, type

export LANG=sr_RS

I've opened a new bug about this: bug 645785.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Related questions

Remote bug watches

Bug watches keep track of this bug in other bug trackers.