Confusing new locale-gen behavior

Bug #1560577 reported by Gunnar Hjalmarsson on 2016-03-22
34
This bug affects 7 people
Affects Status Importance Assigned to Milestone
glibc (Ubuntu)
High
Gunnar Hjalmarsson

Bug Description

When the locales package was provided by the langpack-locales source package, you could call locale-gen with arguments. Examples:

sudo locale-gen en_US.UTF-8
(to generate that locale only)

sudo locale-gen --purge
(to purge all existing locales and create all locales)

With the locales package provided by glibc, the equivalent to the latter seems to be:

sudo locale-gen

What's problematic is that if you add an argument (any argument), the existing locales seem to be purged, but nothing is generated. I'd consider this to be a pretty serious regression, which will likely result in locale issues for many users, if something isn't done to fix it. I'm not sure how to best deal with the problem, though.

CVE References

Gunnar Hjalmarsson (gunnarhj) wrote :

Well, after having looked at the /usr/sbin/locale-gen script and done some further testing, I see that not *all* arguments are ignored. The synopsis seems to be:

locale-gen [[--keep-existing] LANGUAGE]

where LANGUAGE is the name of a file in /var/lib/locales/supported.d. (If you state multiple LANGUAGE arguments, only the last is recognized.) The --keep-existing option prevents purging of previously generated locales.

So basically you can

* Purge all existing locales and generate all locales in
  /var/lib/locales/supported.d

  sudo locale-gen

* Purge all existing locales and generate the locales of *one* file
  in /var/lib/locales/supported.d

  sudo locale-gen LANGUAGE

* [Re-]generate the locales of *one* file in
  /var/lib/locales/supported.d language-pack without purging

  sudo locale-gen --keep-existing LANGUAGE

This is quite different compared to the way the old locale-gen worked.

* All locales are purged by default

* No simple way to generate locales included in SUPPORTED in addition
  to the locales belonging to installed language packs.

* The man page is pretty useless.

I fear that this will result in a lot user confusion. Possibly there is some point with the new behavior, but at this time I fail to see it. In any case the change should be accompanied by some proper documentation.

I would suggest that we use the old locale-gen script (and the related man page) in 16.04. Then, post Xenial, we could consider more thoroughly what to do going forward.

Gunnar Hjalmarsson (gunnarhj) wrote :

I made a couple of minor changes to the script (attached). Those changes would improve it a bit IMO, if going back to the old script is considered 'too much'. Especially they would prevent that a command like

sudo locale-gen en_US.UTF-8

or

sudo locale-gen --purge

results in a broken locale without the user being aware of what happened. The modified script also allows multiple (valid) language code arguments, so

sudo locale-gen en ja

generates the English and Japanese locales, and not only the Japanese.

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in glibc (Ubuntu):
status: New → Confirmed
Gunnar Hjalmarsson (gunnarhj) wrote :

glibc has been updated, and the locale-gen script, which is shipped with locales 2.23-0ubuntu2, addresses most of my concern.

I still found a reason to propose some minor changes:

* Allow multiple language arguments
* Make sure (via '|| true') that the error message is displayed if
  the language/locale isn't supported.

New diff attached.

axel (axel334) wrote :

Can you please have a look at
https://bugs.launchpad.net/ubuntu/+source/language-pack-en-base/+bug/1562279

Although I installed
language-pack-gnome-pl
language-pack-gnome-pl-base
language-pack-pl
language-pack-pl-base

I still get:

sudo locale-gen "pl_PL.UTF-8"/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
Generating locales (this might take a while)...
  pl_PL.UTF-8...cannot open locale definition file `pl_PL': No such file or directory
 done
Generation complete.

Gunnar Hjalmarsson (gunnarhj) wrote :

@axel: I suspect that the locales package hasn't been properly installed. Try:

sudo apt-get install --reinstall locales
sudo locale-gen pl_PL.UTF-8

axel (axel334) wrote :

locale -a
C
C.UTF-8
en_AG
en_AG.utf8
en_AU.utf8
en_BW.utf8
en_CA.utf8
en_DK.utf8
en_GB.utf8
en_HK.utf8
en_IE.utf8
en_IN
en_IN.utf8
en_NG
en_NG.utf8
en_NZ.utf8
en_PH.utf8
en_SG.utf8
en_US.utf8
en_ZA.utf8
en_ZM
en_ZM.utf8
en_ZW.utf8
pl_PL.utf8
POSIX

locale
LANG=pl_PL.UTF-8
LANGUAGE=pl_PL:pl:en_GB:en
LC_CTYPE="pl_PL.UTF-8"
LC_NUMERIC="pl_PL.UTF-8"
LC_TIME="pl_PL.UTF-8"
LC_COLLATE="pl_PL.UTF-8"
LC_MONETARY="pl_PL.UTF-8"
LC_MESSAGES="pl_PL.UTF-8"
LC_PAPER="pl_PL.UTF-8"
LC_NAME="pl_PL.UTF-8"
LC_ADDRESS="pl_PL.UTF-8"
LC_TELEPHONE="pl_PL.UTF-8"
LC_MEASUREMENT="pl_PL.UTF-8"
LC_IDENTIFICATION="pl_PL.UTF-8"
LC_ALL=pl_PL.UTF-8

I also did
sudo dpkg-reconfigure locales

and it helped. It is ok now, also in wine. Thank you very much.

Gunnar Hjalmarsson (gunnarhj) wrote :

I encountered another issue at <http://askubuntu.com/q/753516>: If you for instance pass the argument "en_IN.UTF-8", locale-gen currently sees it as unsupported, since the real name in SUPPORTED is just "en_IN". New locales tend to be UTF-8 only, and I think locale-gen should accept arguments with the codeset appended in these cases.

@Adam: I made a patch out of my proposed changes. If other glibc stuff is on its way in before final freeze, you may want to consider it together with them. Otherwise I think the proposed changes would motivate a separate upload.

Changed in glibc (Ubuntu):
assignee: nobody → Gunnar Hjalmarsson (gunnarhj)
status: Confirmed → In Progress
tags: added: patch
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package glibc - 2.23-0ubuntu3

---------------
glibc (2.23-0ubuntu3) xenial; urgency=medium

  * Merge with 2.23 from experimental, bringing in upstream updates:
    - Save/restore fprs/vrs while resolving symbols (LP: #1564918)
    - Fix _nss_dns_getnetbyname_r() stack overflow (CVE-2016-3075)
    - Merge libnss-dns-udeb and libnss-files-udeb into libc6-udeb.
  * Tidy up locale-gen, thanks to Gunnar Hjalmarsson (LP: #1560577):
    - Fix thinko that broke handling of multiple locale arguments.
    - Recognize UTF-8 locales without charset suffix in SUPPORTED.
    - Fix bug that led to the unsupported message not being shown.
  * Show reboot-required notification for all updates (LP: #1546457)

 -- Adam Conrad <email address hidden> Thu, 14 Apr 2016 10:26:16 -0600

Changed in glibc (Ubuntu):
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers