Installation information cause letter format instead of DIN A4

[JMB]

Dear ladies and gentlemen,

I switched to Debian unstable after problems with TeXLive installations introduce about Oneiric.
As experienced Unix administrator (AIX and RedHat certified) I was astonished not to be able
to solve the problem.
After several attempts I found out that my choices given to the installer:
       English - Berlin timezone - German eliminate deadkeys
yield without any reason to the assumption that the paper format is "letter".
E.g. scientists in Germany allways use English language and Berlin time (i.e. the real place being Germay),
which should clearly qualify to DIN A4 choice, so the choice for /etc/papersize (letter) should be altered
to "a4" under these (and similar) selections.
The behaviour is there as already stated since Oneiric - before it worked as expected -
and is present in the latest flavour.
I currently work under Xubuntu 12.10 but have several installations (including Ubuntu 12.10
[which is really unstable and Unity still not configurable] and Debian Wheezy [testing,
the latter working out of the box without changing /etc/papersize manually]).
As this bug is quite clear I hope you can fix it before 13.04 - this bug is really mean as the
real problem is hidden for the user and even when trying to change problems relvealed in
TeXLive installation relying on the wrong set of papersize the TeXLive scripts for that purpose
are not working (Debian related).
But this is _NOT_ a TeXLive problem - when in Germany (Berlin time) and setting English
(i.e. /etc/default/locale being: LANG="en_US.UTF-8"), papersize should be a4 and NOT letter.
Otherwise one should ask for papersize seperately.
If other data is needed (not think so - but anyway) please contact me:
   <email address hidden>

Many thanks in advance for your help!

JMB

P.S.: If I missed something - like an expert mode of the installer - please inform me.
   But as unexperienced users may have similar selections the problem should be fixed anyway.

ProblemType: Bug
DistroRelease: Ubuntu 12.10
Package: ubiquity (not installed)
ProcVersionSignature: Ubuntu 3.5.0-21.32-generic 3.5.7.1
Uname: Linux 3.5.0-21-generic x86_64
ApportVersion: 2.6.1-0ubuntu9
Architecture: amd64
Date: Mon Dec 31 13:24:16 2012
InstallCmdLine: file=/cdrom/preseed/xubuntu.seed boot=casper initrd=/casper/initrd.lz quiet splash -- maybe-ubiquity
InstallationDate: Installed on 2012-10-23 (68 days ago)
InstallationMedia: Xubuntu 12.10 "Quantal Quetzal" - Release amd64 (20121017.1)
MarkForUpload: True
SourcePackage: ubiquity
UpgradeStatus: No upgrade log present (probably fresh install)

[Colin Watson (cjwatson)]

This is very odd since we have explicit code in the installer to reconfigure the paper size according to the locale. Some part of this must have bitrotted, and I will investigate. Thanks for your report.

[Colin Watson (cjwatson)]

Ah, of course it is possible that this is because there's no en_DE locale; such cases always lead to difficulties. In this situation we use two different locales distributed among the various locale categories (LC_PAPER, among others, is set to reflect the location, as opposed to something like LC_MESSAGES). One thought I have is that we may only be generating the language-based locale and not the location-based locale, which would probably cause this problem since libpaper1.config looks at 'locale width' and 'locale height'.

[Colin Watson (cjwatson)]

Oh. In fact we don't even set LC_PAPER as I thought; that was, I think, a fairly late change in some release cycle so I only did it for the peculiar cases of Chinese and Portuguese. I suspect I need to extend this.

[Colin Watson (cjwatson)]

There's some history on this in comments #11 etc. of bug 590108.

[Colin Watson (cjwatson)]

I did some analysis of the cases where a country has multiple languages available in /usr/share/i18n/SUPPORTED. The summary of my conclusion is that I do not think that this is in general amenable to automation, and will require a table of special cases.

In some cases there seems to be to be a reasonably clear correct choice. These include cases where there are special-purpose locales, "joke" locales, cases where the country has minority language communities but there is nevertheless a clear majority language that you might reasonably guess a migrant to that country would adopt, and cases where there's an official national language which you might therefore see on road signs etc. (Note that this decision will not come into effect if the user's selected language has a locale in the selected country, so it doesn't affect e.g. Spanish-speaking Mexicans living in the United States.)

AU: en la -> en
CN: bo ug zh -> zh
DE: de fy hsb nds -> de
DK: da en -> da
DZ: ar ber -> ar
ES: an ast ca es eu gl -> es
ET: aa am gez om sid so ti wal -> am (probably)
FI: fi sv -> fi
FR: br ca eu fr oc -> fr
GB: cy en gd gv kw tlh -> en
IE: en ga -> en
IT: ca fur it lij sc -> it
MA: ar ber -> ar
MK: mk sq -> mk
NG: en ha ig yo -> en
NL: fy li nds nl -> nl
NZ: en mi -> en
IL: he iw -> he
PH: en fil tl -> fil (probably)
PK: pa sd ur -> ur
PL: csb pl -> pl
RU: cv mhr os ru tt -> ru
SG: en zh -> en (probably)
SN: ff wo -> wo
TR: ku tr -> tr
TW: nan zh -> zh
UA: crh ru uk -> uk
US: en eo es unm yi -> en
ZM: bem en -> en

In some cases there appears to be no clear right choice, sometimes due to geopolitical disputes:

BE: de fr li nl wa
CA: en fr ik iu shs
CH: de fr it wae
CY: el tr

In some cases I'm simply not sure:

DJ: aa so
ER: aa byn gez ti tig
HK: en yue zh
IN: ar as bho bn bo brx en gu hi hne kn kok ks mai ml mr or pa sa sd ta te ur
KE: om so sw
LK: si ta
LU: de fr lb
NO: nb nn se
ZA: af en nr nso ss st tn ts ve xh zu

[JMB (jmb-tux)]

I got a little confused - actually I have not looked at the dependencies and variables of
the installation process but from my point of view the site location - given in the Ubuntu
installation process for setting the time zone - should be responsible for the choice for
paper size (which seems to be the case till ~Ubuntu 10.10), i.e.:
      Berlin time / CET -> DIN A4 paper.
Selecting English locale (i.e. language and cultural conventions like day namings etc.)
is in international environments alway EN (GB or US) - so this is not qualified for the
paper size.

As TeX Live in Debian packaging can not be changed to another paper
format with standard scripts (I looked into it - they just don't work - and
finally reinstalled the system - much less effort), this installation bug is
a real problem under those circumstances.
I found nothing in WWW and even on forums the answer was not clear:
    http://ubuntuforums.org/showthread.php?p=11755454#post11755454

In case the installer can not be fixed due to other users/customers needs
it would be nice to advise people installing TeX Live to look at /etc/papersize
and change it (to a4 or letter ...) if necessary _before_ the installation.
Maybe you can - if appropriate - redirect/fork the case to documentation
and/or TeX Live maintainer.

Thanks!

[Colin Watson (cjwatson)]

Please don't worry - I understand your problem just fine and I agree that the installer should take care of it automatically. However it ties into things that are more complicated than just the paper size, and I want to solve it properly since the underlying problem has remained unsolved for far too long. The reason I am leaving long and complicated notes on this bug report is that I want to record as much context as possible for my future self in case I am interrupted in dealing with this problem.

[Gunnar Hjalmarsson (gunnarhj)]

Let me first comment on the discussion about locale settings in the installer. I'm inclined to think that the installer UI is not really sufficient as a base for setting all the locale categories. You are asked about the time zone, and that leads me to assume that my answer affects /etc/timezone and such stuff. Consequently I have thought that the installer typically sets LANG based on the language choice, while language-selector is meant to be used afterwards if needed to complete the locale. However, when I mentioned this at bug 1035498, Steve Langasek objected.

Different ways to state the default paper size seem to cause quite some confusion; see for instance bug 393818. I don't know to which extent various apps care about LC_PAPER directly. LibreOffice seems to use /etc/papersize and ignore LC_PAPER. However, it would be reasonable IMO that the default paper size is derived from the paper height and width in the LC_PAPER locale (which info ought to be possible to access via nl_langinfo).

Irrespective of how the paper size issue is dealt with in the installer, I can think that it should be possible to change /etc/papersize afterwards from the desktop. One thought is that when regional formats are changed system-wide via language-selector, /etc/papersize is updated accordingly. That way user specific settings of LC_PAPER would still be ignored, but it would probably be a step in the right direction.

[Gunnar Hjalmarsson (gunnarhj)]

I wrote a script, that it would be nice to have in PATH. It could be used by both the installer and language-selector to set /etc/papersize to a reasonable value as derived from an applicable locale. As regards language-selector, it could be called when the system wide setting of regional formats (in /etc/default/locale) is updated.

@Colin: If you think this is a good idea, is there a suitable package where the script could reside?

[Colin Watson (cjwatson)]

I don't see any need to change the handling of /etc/papersize. Reconfiguring libpaper works fine once the locale issues have been sorted out. Don't change things that aren't broken.

[Colin Watson (cjwatson)]

Reconfiguring libpaper1, that is. The installer already does this; I've tested it in combination with my locale fixes and it's working fine.

[Colin Watson (cjwatson)]

There is a problem in langpack-locales, though. If you do the following sequence:

  apt-get install language-pack-de-base
  locale-gen de_DE.UTF-8
  apt-get remove language-pack-de-base

... you end up with /var/lib/locales/supported.d/local still containing de_DE.UTF-8, but 'locale -a' doesn't list it. I think this is a clear bug in remove-language-pack. The installer ends up effectively executing this sequence with my rearrangements, because it only keeps the language pack for the locale effective for translations but we still want the libc locale to be present for the locale effective for non-translation-like categories.

[Gunnar Hjalmarsson (gunnarhj)]

Using the papersize command instead of editing /etc/papersize directly may be an option. But please note that the main point with the attached script is to point at a way to go from locale to papersize. It makes sense IMO to use the locale info that is available. Maybe that's what you do in the installer as well... I'd like to see a way to keep the papersize setting in sync with the regional formats settings as done from language-selector.

As regards locales, if I understand it correctly the Ubuntu model so far has been that only locales corresponding to installed languages are available. New locales are created when a language is installed, and they are removed when a language is removed. That's why remove-language-pack works the way it does. There is no other UI besides language-selector for managing the generated locales as shown with "locale -a", is there?

So if you would change it somehow, I think we would need to reconsider things beyond langpack-locales.

[Colin Watson (cjwatson)]

On Wed, Feb 20, 2013 at 12:18:57PM -0000, Gunnar Hjalmarsson wrote:
> Using the papersize command instead of editing /etc/papersize directly
> may be an option. But please note that the main point with the attached
> script is to point at a way to go from locale to papersize. It makes
> sense IMO to use the locale info that is available. Maybe that's what
> you do in the installer as well... I'd like to see a way to keep the
> papersize setting in sync with the regional formats settings as done
> from language-selector.

I don't mind what you do in language-selector for this, but it really
isn't related to this bug so I'd like it if you could keep it separate.
The installer can and does manage just fine by running 'dpkg-reconfigure
libpaper1', which already has the necessary logic to go from locale to
papersize. I'm getting the impression that you're unaware of that code
that already exists in libpaper1.config?

> As regards locales, if I understand it correctly the Ubuntu model so far
> has been that only locales corresponding to installed languages are
> available. New locales are created when a language is installed, and
> they are removed when a language is removed. That's why remove-language-
> pack works the way it does. There is no other UI besides language-
> selector for managing the generated locales as shown with "locale -a",
> is there?

UI or not, it has always been possible and supported to use the
locale-gen program directly to generate additional locales, which is why
/var/lib/locales/supported.d/local exists. It's a straightforward bug
that remove-language-pack wasn't honouring that properly. Martin
approved my patch to remove-language-pack and I've uploaded it.

[Launchpad Janitor (janitor)]

This bug was fixed in the package langpack-locales - 2.13+git20120306-7

---------------
langpack-locales (2.13+git20120306-7) raring; urgency=low

  * debian/local/remove-language-pack: Keep any locales for that language
    which are listed in /var/lib/locales/supported.d/local (LP: #1094872).
 -- Colin Watson <email address hidden> Wed, 20 Feb 2013 12:45:29 +0000

[Gunnar Hjalmarsson (gunnarhj)]

On 2013-02-20 13:50, Colin Watson wrote:
> I don't mind what you do in language-selector for this, but it really
> isn't related to this bug so I'd like it if you could keep it separate.

That's fine; I just filed bug 1130690 for the language-selector side of it, and invalidated the language-selector task in this bug.

> The installer can and does manage just fine by running 'dpkg-reconfigure
> libpaper1', which already has the necessary logic to go from locale to
> papersize. I'm getting the impression that you're unaware of that code
> that already exists in libpaper1.config?

Yes, I was unaware of that; thanks for the pointer! Don't understand, though, how "dpkg-reconfigure libpaper1" can be used in a non-interactive way. Hope you don't mind if I get back to you for advice, if needed, when dealing with bug 1130690.

> UI or not, it has always been possible and supported to use the
> locale-gen program directly to generate additional locales, which is why
> /var/lib/locales/supported.d/local exists. It's a straightforward bug
> that remove-language-pack wasn't honouring that properly. Martin
> approved my patch to remove-language-pack and I've uploaded it.

Thanks for letting me know. I will do some testing with installing/removing languages via language-selector to check if there is a need for further adjustments due to the change. Possibly it's already taken care of through
https://launchpad.net/ubuntu/+source/accountsservice/0.6.29-1ubuntu5

[Colin Watson (cjwatson)]

On Wed, Feb 20, 2013 at 02:51:17PM -0000, Gunnar Hjalmarsson wrote:
> That's fine; I just filed bug 1130690 for the language-selector side of
> it, and invalidated the language-selector task in this bug.

Thanks.

> Yes, I was unaware of that; thanks for the pointer! Don't understand,
> though, how "dpkg-reconfigure libpaper1" can be used in a non-
> interactive way. Hope you don't mind if I get back to you for advice, if
> needed, when dealing with bug 1130690.

Sorry, I forgot to mention the -fnoninteractive option.

[Gunnar Hjalmarsson (gunnarhj)]

Sorry for being a pain, Colin, but

sudo dpkg-reconfigure -fnoninteractive libpaper1

(in Quantal) does not seem to change anything. I tested with both changing the session environment and editing /etc/default/locale, but /etc/papersize remained unchanged.

[Colin Watson (cjwatson)]

You might need to make sure that sudo doesn't change the environment, or
else do something like sudo sh -c '. /etc/default/locale;
dpkg-reconfigure -fnoninteractive libpaper1'.

[Gunnar Hjalmarsson (gunnarhj)]

On 2013-02-20 22:04, Colin Watson wrote:
> On 2013-02-20 21:39, Gunnar Hjalmarsson wrote:
>>
>> sudo dpkg-reconfigure -fnoninteractive libpaper1
>>
>> (in Quantal) does not seem to change anything.
>
> You might need to make sure that sudo doesn't change the environment,
> or else do something like sudo sh -c '. /etc/default/locale;
> dpkg-reconfigure -fnoninteractive libpaper1'.

Tried that too with no success, i.e. /etc/papersize remains unchanged. What kind of magic do you use in the installer? ;-)

From libpaper1.config I learned (the undocumented?) "locale height" and "locale width", which replace the need for nl_langinfo. The attached simplistic shell script (run as sudo) works as expected for me. It should be possible to use the paperconfig command instead of writing /etc/papersize directly, but if TeX Live is installed, interactivity is automatically (and incorrectly?) triggered sometimes.

Do we need other default papersize options but "a4" and "letter"? A user with special needs can add

  export PAPERSIZE=[whatever]

to e.g. ~/.profile.

[Colin Watson (cjwatson)]

Ah, sorry, you have to do something like this first:

  sudo rm -f /etc/papersize
  sudo ucf --purge /etc/papersize
  echo SET libpaper/defaultpaper | sudo debconf-communicate

See ubiquity/scripts/plugininstall.py. I wrote all that nearly six
years ago so had largely forgotten :-)

libpaper1.config has to operate under the constraint that libpaper1
hasn't even been unpacked yet, so it does have to be pretty
conservative. It's certainly possible that you can do something easier
elsewhere; I'm really just saying that what we have now works well in
the installer context and I don't feel the need to change it. I don't
mind what language-selector does as long as it's consistent; your shell
script is at least simple.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubiquity - 2.13.12

---------------
ubiquity (2.13.12) raring; urgency=low

  [ Colin Watson ]
  * While displaying question or error dialogs, only change the busy cursor
    state rather than also allowing changing step (LP: #1095684).
  * Use an action group when marking packages for install or delete in the
    apt cache; this is much faster when many packages are involved, since it
    avoids repeating autoremoval calculations over and over again.
  * Sort and consolidate imports.
  * Fix ordering of encryption password strength indications (LP: #1068391).
  * At the end of the language install plugin, read all locale-related keys
    from (/target)/etc/default/locale and set them in the installer's own
    environment.
  * If the combination of language and location does not identify a
    supported combined locale, use a locale based on the language for
    translation-like locale categories and pick one based on the location
    for other locale categories (LP: #1094872). There are still some
    locations where we cannot pick a language element of a location-based
    locale because this is unclear or contentious, and in those cases we
    stick to the old behaviour of having the locale only reflect the
    country.

  [ Aurélien Gâteau ]
  * KDE frontend: show name of installed OS as partition name if
    available

  [ Dmitrijs Ledkovs ]
  * Make panel.c aware of screen & display changes, prevents visual
    artefacts on nexus7 when screen is changing or external monitor is
    plugged in and resolution is changes.
  * Update disable/enable ctrl+alt+t terminal to use gsettings keys,
    instead of gconf.
  * Migrate remaining gconf settings to gsettings in ubiquity-dm.
  * Make sure to correctly set picture-uri. (LP: #1128597)
  * Refactor ubiquity-dm, now that there is less code.
  * Remove gconftool.py.
  * Automatic update of included source packages: grub-installer
    1.78ubuntu6, hw-detect 1.92ubuntu1.
 -- Dmitrijs Ledkovs <email address hidden> Thu, 21 Feb 2013 02:53:43 +0000

[Gunnar Hjalmarsson (gunnarhj)]

Hi again, Colin!

Yeah, right now I'm inclined to keep it simple. ;-) libpaper1.config seems to not be installed by the package - maybe an installer exclusive thing?

I have proposed a tiny script "locale2papersize", similar to the shell script attached here, and it's aimed to be used by accountsservice, language-selector and g-c-c (patch at bug 1130690). The approach, i.e. LC_PAPER locale => papersize, is indeed consistent with how it's done in the installer.

I noticed a small bug in remove-language-pack as regards the Chinese languages; fix proposed in the linked MP.

Also, when adding a couple of extra locales via locale-gen, I was prompted by check-language-support (run without arguments) and the language-selector UI to install more language support. Fix proposed; I asked Martin to review it since he wrote the code.
https://code.launchpad.net/~gunnarhj/ubuntu/raring/language-selector/avail-lang/+merge/149975
Don't know if this is of any significance to the installer.

Cheers,

[Launchpad Janitor (janitor)]

This bug was fixed in the package langpack-locales - 2.13+git20120306-8

---------------
langpack-locales (2.13+git20120306-8) raring; urgency=low

  * debian/local/remove-language-pack: Fix typo preventing removing
    both locales in case of zh-han[st] (LP: #1094872).
 -- Gunnar Hjalmarsson <email address hidden> Sat, 23 Feb 2013 18:12:00 +0100

[Simos Xenitellis  (simosx)]

We (Ubuntu-gr - Greece) are facing an issue that we believe is related to the above change in the code.

See https://bugs.launchpad.net/ubuntu/+source/ubiquity/+bug/1158750

Specifically, what happens is that currently the code sets (by mistake) the locale to 'el_GR' instead of 'el_GR.UTF-8' (or 'el_GR.utf8').

The default for a 'el_GR' locale (without .utf8) is to use the old 8-bit deprecated encodings, which cause severe issues in a 13.04 installation. For example, LibreOffice simply refuses to start!
The reason is that Greek text for Mon/Tue/Wed/Thu/etc that come from the Greek locale, are being read in the deprecated ISO-8859-7 encoding.

We are trying to test if this issue exists with other locales as well,
and also how to solve. It appears that a small code addition (adding '.utf8') should suffice.

If you have any input, it would be helpful to get this resolved quickly.

[Gunnar Hjalmarsson (gunnarhj)]

I suspect that also the crash report at bug 1165030 has something to do with the issue described by Simon in comment #26.

[Colin Watson (cjwatson)]

No need to have this bug reopened to track the regression - bug 1158750 is good enough for that. Closing again.