apt warns about wrong language code: eo.utf8

[Aisano]

TEST CASE:
1. run "sudo apt-get install language-pack-eo"
2. run "sudo LANG=eo.utf8 apt-get update"
3. very a bunch of "W: Wrong language code eo.UTF-8" at the end
4. install apt from lucid-proposed
5. repeat step 2
6. verify that no errors are displayed this time

I installed Ubuntu 10.04 beta 2, specifying "Esperanto" as the default language. This works quite well but every now and then I get a window saying "Eraro okazis" ("an error occurred"), with several (more than 10) lines saying "W: Wrong language code eo.utf8". A reproducible way of producing this window is starting the Synaptic package manager; when it is started the window pops up every time.

The window can be closed without affecting Synaptic, which even displays its texts in Esperanto without problems. Still it is somewhat annoying. I attach a screen copy of the window. (Weird thing was that Alt-Print produced just a black rectangle; I had to copy the whole screen using Print without Alt and then cut out the window. That need not be related to the problem but I thought I'd better mention it.)

[Arturo Torres Sánchez (r2d2.art2005)]

Same problem here. Using Ubuntu 10.04 beta 2, with Esperanto as default language.

[Michael Moroni (airon90)]

Same as Arthuro

[Aisano (info-ais-sanmarino)]

I suspect that the problem is related to the fact that "eo.utf8" does not contain an underscore. All other locales in my /usr/lib/locale contain a country specification, such as "CH" in "de_CH.utf8". I remember that we once used "XX" for Esperanto but this XX seems to have gone.

If my assumption is correct then synaptic and/or a ibrary it uses require the underscore in the locale name. The message "wrong language code" suggests that something tries to parse the locale name, taking everything before the underscore as the "language code".

I cannot say if there is a rule saying that locale names must contain an underscore. We could try with an underscore and an empty country code, such as "eo_.utf8". To test my hypothesis I created such a locale in /usr/lib/locales and called synaptic with "LANG=eo_.utf8". No error window. But that does not tell us whose fault the problem is and which side (synaptic or the locale definition) should be changed.

[Aisano (info-ais-sanmarino)]

Checking my /usr/share/i18n/SUPPORTED file I found that at present it lists only three locales without an underscore: "eo.UTF-8" (with uppercase "UTF", while all locales under /usr/lib/locale have an encoding name in lowercase), "eo" and "ia" (Interlingua). To my amazement even the Klingon locale has a country code: "tlh_GB.UTF-8". Seems to be British Klingon.

The case difference suggests that the list in SUPPORTED does not really matter. Still the list shows that, if we insist on having no country code, we might have few allies. Maybe Esperanto should go back to a pseudo country. ISO 3166 says that the following two-letter codes may be user-assigned: AA, QM to QZ, XA to XZ, ZZ. How about "eo_AA.utf8"?

[Aisano (info-ais-sanmarino)]

Given that three people reported the same problem I set this to "confirmed". I found that it can easily be reproduced even without having an Esperanto locale installed, using the little script I attach. You need administrator privileges, of course.

[Aisano (info-ais-sanmarino)]

I did a little digging in C++ code and found that the error message ("Wrong language code") is produced in libapt; the problem affects not only synaptic but also its command-line twin "apt-get". The message is produced in "apt-pkg/indexfile.cc" in line 99, in the "pkgIndexFile::CheckLanguageCode()" method. This method's comment does not give a pattern for language codes but a list of "common cases": "de_DE, de_DE@euro, de_DE.UTF-8, de_DE.UTF-8@euro, de_DE.ISO8859-1, tig_ER", adding "more in /etc/gdm/locale.conf" (which I cannot find on my system).

The problem, however, is not in "CheckLanguageCode()" but in the "LanguageCode()" method, which parses the language code from the LANG value (only the result is then checked in "CheckLanguageCode()"). Unless the language is one of "en", "pt", "sv", or "zh" it uses the underscore to extract the language code. With no underscore found it uses the whole value of LANG, including the encoding. The obvious fix would be to look for a period when no underscore is found. I will try if I can get this to work; should not be too hard.

[Aisano (info-ais-sanmarino)]

The approach works. It could still be discussed if this heuristic checking of language codes is a good idea -- if translation files are found for a certain locale then why bother if the locale name is correct. But that is another story.

I include a patch for apt-pkg/indexfile.cc. I suppose I will have to find another package to assign this bug to, as it does not seem to be related to "langpack-locales". Unfortunately Launchpad does not know anything called "apt-pkg", "libapt" or similar.

[Aisano (info-ais-sanmarino)]

Michael Vogt (apt) has accepted the patch into the 10.10 (maverick) branch, so I set the status to "fix committed". This does not mean that there will be an immediate fix for 10.04 (lucid lynx). Users affected by this bug should take care that

(1) The message ("wrong language code") is just a warning, synaptic and apt-get are doing fine despite this (somewhat annoying) problem.
(2) The problem can be circumvented by creating a new locale "/usr/lib/locales/eo_.utf8" and calling synaptic as "LANG=eo_.utf8 synaptic". I suspect, however, that many will consider this more annoying than the warnings.
(3) Michael suggested that we might request a "StableReleaseUpdate" (SRU) for Ubuntu 10.04. Given that Ubuntu 10.04 is a Long-Term Support version we may even get it.

As users of the eo.utf8 locale seem to be the only ones affected I repeat the essence in Esperanto:

La problemo estas klarigita, kaj ekzistas riparo (flikaĵo), kiu tamen aperos eble nur en eldono 10.10 (maverick meerkat). Bonvolu noti:

(1) La mesaĝo ("wrong language code") estas nur averto; la programoj synaptic kaj apt-get funkcias malgraŭ tiu ĉi ĝenaĵo.
(2) Eblas ĉirkaŭiri la problemon kreante novan lokaĵaron "/usr/lib/locales/eo_.utf8" kaj rulante synaptic per "LANG=eo_.utf8 synaptic". Tamen eble multaj konsideras tion ĉi pli ĝena ol la avertojn.
(3) Michael Vogt, zorganto de "apt", proponis peti SRU-on (ŝanĝon de stabila eldono) por Ubuntu 10.04. Eble do estos korektaĵo.

[Michael Moroni (airon90)]

2010/5/3 Aisano <email address hidden>

> La problemo estas klarigita, kaj ekzistas riparo (flikaĵo), kiu tamen
> aperos eble nur en eldono 10.10 (maverick meerkat). Bonvolu noti:
>
> (1) La mesaĝo ("wrong language code") estas nur averto; la programoj
> synaptic kaj apt-get funkcias malgraŭ tiu ĉi ĝenaĵo.
> (2) Eblas ĉirkaŭiri la problemon kreante novan lokaĵaron
> "/usr/lib/locales/eo_.utf8" kaj rulante synaptic per "LANG=eo_.utf8
> synaptic". Tamen eble multaj konsideras tion ĉi pli ĝena ol la avertojn.
> (3) Michael Vogt, zorganto de "apt", proponis peti SRU-on (ŝanĝon de
> stabila eldono) por Ubuntu 10.04. Eble do estos korektaĵo.
>

Dankegon Aisano por via atentigo pri la problemo. Mi ne sciis kion fari krom
raporti la cimon. :S
Eble estas aliaj lingvoj, kiu ne havas nacisufikson (ekzemple: en_US, it_IT,
...). Do, mi sugestas peti pri SRU kaj ŝanĝi la dosiero(j)n tiel: eo_EO,
xx_XX, ... por ke apt rekonas la dosierojn.

Aldone, Ubuntu havas ankaŭ Usonan version de Esperanto (???): eo_US.
Mi ĉifoje pensas ĉu estas malsamecoj inter Esperanto kaj "Usona
Esperanto"...
Eble ĉu estas solvo de nia problemo?
-- Michael

--
Primus in orbe deos fecit timor.
La unua estis timo, kio kreis diojn en la mondo.
-- Petronio Arbitro, ''Satyricon''

[Aisano (info-ais-sanmarino)]

=== I received this from <email address hidden> ===
=== in Esperanto so I answer in this language. ===
=== If you cannot read Esperanto please delete this message ===
=== The bug presently seems to affect only the Esperanto locale ===

Je 2010-05-03 21:05 Michael Moroni skribis:
> 2010/5/3 Aisano<email address hidden>
>
>
>> ...
> Dankegon Aisano por via atentigo pri la problemo. Mi ne sciis kion fari krom
> raporti la cimon. :S
> Eble estas aliaj lingvoj, kiu ne havas nacisufikson (ekzemple: en_US, it_IT,
> ...). Do, mi sugestas peti pri SRU kaj ŝanĝi la dosiero(j)n tiel: eo_EO,
> xx_XX, ... por ke apt rekonas la dosierojn.
>
Ne dankinde. Antaŭ multaj jaroj mi multe laboris pri Uniksoj, do
kiam mi sukcesis elŝuti la fonto-kodon de libapt, ne estis malfacile
trovi la kaŭzon de la eraro kaj korekti ĝin. Se mi iom fieras, estas
pro la fakto ke mi jam antaŭe ĝuste analizis la kaŭzon,
nome ke iu biblioteko provas trovi substrekon (_) en $LANG.

Mi nun en kelkaj tagoj lernis kiel krei flikaĵon, do mi esperas
lerni ankaŭ peti pri SRU, kun la helpo de Michael Vogt.

Kiel mi jam skribis en Launchpad, laŭ
/usr/share/i18n/SUPPORTED estas nur unu alia lingvo sen
lando: Interlingvao (ia). Eĉ la klingona lingvo havas landon:
Britio (tlh_GB). Sed evidente Michael Vogt ne dubas, ke lingvo
sen lando estas io tute normala.
> Aldone, Ubuntu havas ankaŭ Usonan version de Esperanto (???): eo_US.
> Mi ĉifoje pensas ĉu estas malsamecoj inter Esperanto kaj "Usona
> Esperanto"...
> Eble ĉu estas solvo de nia problemo?
> -- Michael
>
Jes, vi pravas, ekzistas "usona Esperanto", sed ĝi ŝajnas al
mi same stranga kiel "brita klingona lingvo". Mi preferus
havi "veran" solvon. (Eble eo_US ekzistas, ĉar Usono simple
ne estas tia, kiaj la aliaj landoj... oni diras.)

Kiam mi lernos peti pri SRU mi faros, kaj espereble ni ĝin
ricevos. Mi povus sendi al vi riparitan version de
/usr/lib/libapt-pkg-libc6.10-6.so.4.8.0,
sed ĝi estus nur provizora.

Ĉion bonan,
Reinhard (Aisano)

[Aisano (info-ais-sanmarino)]

**** Information relevant for a STABLE RELEASE UPDATE nomination ****

1. The bug outputs a lot of unjustified warnings to users of synaptic or apt-get who have a countryless locale ($LANG value) with an encoding, such as "eo.utf8". Synaptic displays these warnings in a separate window which has to be closed by the user. Although the function of the programs (which are critical for system administration) is not affected the user has no way of telling.

2. The bug has been patched for "apt-0.7.25". So far, libapt did not correctly parse locales without an underscore but with an encoding. The parsing algorithm has been extended to consider a period (dot) as a separator as well. This extension comes into effect only after and if the traditional approach has failed.

3. The patch (attached to #8 above) adds two lines to the indexfile.cc source file. Michael Vogt (responsible for the apt package) has approved it.

TEST CASE
4. (A test script to reproduce the error was attached to #6 above before the patch was begun.) On a computer with the "eo.utf8" (Esperanto) locale installed it is suffcient to call "sudo LANG=eo.utf8 synaptic", and synaptic will display a warning window saying "W: Wrong language code eo.utf8" (several times; picture attached above under #1). With the patch installed this warning window does not appear.

If the "eo.utf8" locale is not installed a pseudo locale can be created by performing three commands:
$ cd /usr/lib/locale
$ sudo mkdir eo.utf8
$ sudo cp -r en_US.utf8/* eo.utf8
Afterwards, the test command ("sudo LANG=eo.utf8 synaptic") can be called as above. The pseudo locale can then be removed by "sudo rm -r eo.utf8".

5. Regression potential. If somebody had created a locale of their own with a period inside the language code, and without a country code (such"LANG=x..utf8"), then the patch would affect (not break) the use of such a locale, as the user would start getting the warnings described above. As language codes are internationally standardized (ISO 639) to consist of letters only, the warnings would be justified in this case.

Michael Vogt is going to nominate this patch for an SRU and upload it.

[Martin Pitt (pitti)]

Accepted apt into lucid-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Fabio Marconi (fabiomarconi)]

Testcase verified: OK, no errors.

[Launchpad Janitor (janitor)]

This bug was fixed in the package apt - 0.7.25.3ubuntu8

---------------
apt (0.7.25.3ubuntu8) lucid-proposed; urgency=low

  [ Loïc Minier ]
  * Use https:// in Vcs-Bzr URL.

  [ Michael Vogt ]
  * apt-pkg/deb/debrecords.cc:
    - fix max tag buffer size (LP: #545336, closes: #578959)
  * apt-pkg/indexfile.cc:
    - If no "_" is found in the language code, try to find a "."
      This is required for languages like Esperanto that have no
      county associated with them (LP: #560956)
      Thanks to "Aisano" for the fix
 -- Michael Vogt <email address hidden> Wed, 05 May 2010 10:33:46 +0200

[Martin Pitt (pitti)]

Copied to maverick.