Locale issues with beta-1/2 cloud-images

[James Page]

Default locale of image appears to be en_US.UTF-8 - however when I logon from a en_GB.UTF-8 client via SSH I get the following errors - springs up in a few places while doing things on the server.

This was from a local openstack test instance; however I get the same error in ec2 instances with the latest daily snapshot.

ubuntu@server-3:~$ locale
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE=en_GB.UTF-8
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE=en_GB.UTF-8
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES=en_GB.UTF-8
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

Related Bugs:
 * Bug 920601: cloud-images don't accept LANG settings
 * Bug 969462: postgresql-9.1 fails to start after install if invalid locale is set

ProblemType: Bug
DistroRelease: Ubuntu 11.10
Package: cloud-init 0.6.1-0ubuntu15
ProcVersionSignature: User Name 3.0.0-9.15-virtual 3.0.3
Uname: Linux 3.0.0-9-virtual x86_64
Architecture: amd64
Date: Mon Sep 26 17:09:24 2011
Ec2AMI: ami-00000002
Ec2AMIManifest: FIXME
Ec2AvailabilityZone: nova
Ec2InstanceType: m1.small
Ec2Kernel: aki-00000001
Ec2Ramdisk: unavailable
PackageArchitecture: all
ProcEnviron:
 LC_CTYPE=en_GB.UTF-8
 LC_COLLATE=en_GB.UTF-8
 LANG=en_US.UTF-8
 LC_MESSAGES=en_GB.UTF-8
 SHELL=/bin/bash
SourcePackage: cloud-init
UpgradeStatus: No upgrade log present (probably fresh install)

[Scott Moser (smoser)]

I can confirm this, and also confirm it on 11.04. I suspect it is the case with even 10.04.

$ env LC_ALL=en_GB.UTF-8 ssh 10.55.60.59 'cat /etc/cloud/build.info; echo ===; locale'
bash: warning: setlocale: LC_ALL: cannot change locale (en_GB.UTF-8)
build_name: server
serial: 20110426
===
locale: Cannot set LC_CTYPE to default localeLANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE="en_GB.UTF-8"
LC_NUMERIC="en_GB.UTF-8"
LC_TIME="en_GB.UTF-8"
LC_COLLATE="en_GB.UTF-8"
LC_MONETARY="en_GB.UTF-8"
LC_MESSAGES="en_GB.UTF-8"
LC_PAPER="en_GB.UTF-8"
LC_NAME="en_GB.UTF-8"
LC_ADDRESS="en_GB.UTF-8"
LC_TELEPHONE="en_GB.UTF-8"
LC_MEASUREMENT="en_GB.UTF-8"
LC_IDENTIFICATION="en_GB.UTF-8"
LC_ALL=en_GB.UTF-8
: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory

I also suspect this is the case with *any* server that does not have locale X installed if you ssh in with a locale set locally.

[Dave Walker (davewalker)]

ubuntu@server-1251:~$ cat /etc/cloud/build.info
build_name: server
serial: 20110921.1

ubuntu@server-1251:~$ locale
LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

[Scott Moser (smoser)]

There was some discussion on this, but i believe the only real way to fix it is to add locales to the image.

[Launchpad Janitor (janitor)]

[Expired for cloud-init (Ubuntu) because there has been no activity for 60 days.]

[Ben Howard (darkmuggle-deactivatedaccount)]

So a couple of findings here.

We are doing locales wrong: generating the locales is only part of the process of localizations. So while Cloud-Init generates the locales, the images are far from localized. In order to have a localized image, the base language pack (i.e. language-pack-{en,fr,de,...} needs to be installed otherwise the base language remains in English. The default "en_US.UTF8" locale is incomplete in the images since the language pack is not installed.

Therefore, I think that we should do the following:

1. The images should have the "language-pack-en" installed. Any English speaker who is exporting variable via SSH will automatically get localization as long as the localization is standard. Since Ubuntu's language of collaberation is English, this should fix the vast majority of users encountering localization problems, unless they are exporting non-standard chacter sets (i.e. exporting en_GB instead of en_GB.UTF8).

2. Cloud-init will should support the installation of language packs when it parses "locale: <locale>" during the cloud-config section.

3. Cloud-init should have a /etc/profile.d message that tells people if their locale settings are wrong and how to fix it.

[Scott Moser (smoser)]

Some more data here, the following results in postgresql not started on installation:

$ LC_ALL=en_GB.UTF-8 ssh ubuntu@$HOSTNAME
$ sudo apt-get install postgresql-9.1
...
Adding user postgres to group ssl-cert
Building PostgreSQL dictionaries from installed myspell/hunspell packages...
Setting up postgresql-9.1 (9.1.3-2) ...
Error: The locale requested by the environment is invalid.
Error: could not create default cluster. Please create it manually with

  pg_createcluster 9.1 main --start

or a similar command (see 'man pg_createcluster').

If you run 'apt-get install language-pack-en' prior to running the above, it will work.

It will still fail, though if you do:
 $ LC_ALL=fr_FR.UTF-8 ssh ....
 $ sudo apt-get install postgresql-9.1
(which would be fixed by 'apt-get install language-pack-fr').

[Ben Howard (darkmuggle-deactivatedaccount)]

Added "language-pack-en" to all Precise builds.

[Scott Moser (smoser)]

Ben, the right way to do this is for us to add 'language-pack-en' to the cloud-image seed at http://bazaar.launchpad.net/~ubuntu-core-dev/ubuntu-seeds/ubuntu.precise/files .

[Scott Moser (smoser)]

I proposed that for merging at https://code.launchpad.net/~smoser/ubuntu-seeds/cloud-images-language-pack-en/+merge/100411

[Scott Moser (smoser)]

i've added language-pack-en to the cloud-images seed. So we can back out the automated-ec2-builds changes.

[Scott Moser (smoser)]

Ug. whenever we think we're going down the right track, we find more information. But I guess, better to know it now then later.
http://irclogs.ubuntu.com/2012/04/03/%23ubuntu-devel.html#t00:19 is a conversation between cjwatson, myself and infinity.

There, I find out
 a. it is perfectly valid and supported to run 'localegen' for locales where no corresponding language-pack-xx is installed
 b. an easy way to check "is this locale currently valid" is to do:
    import locale
    try:
       locale.setlocale(locale.LC_ALL, '') # or a specific locale as the second argument
    except locale.Error

That wont' involve apt (and thus will be quicker).

Because of 'a', cloud-init is not invalid in doing its 'locale-gen' on possibly un-installed locales, so I think i'd just leave that as it is for now.

And i think we can just handle package installation via informing the user on login. This means really that the more difficult to get right portion of this change (the package installation) can just be skipped.

What do you think, Ben?

[Scott Moser (smoser)]

OK.
  So in talking with Ben, we've come up with the following plan:
 * remove the language-pack-en from the cloud-image seed (i just did this and pushed to lp:~ubuntu-core-dev/ubuntu-seeds/ubuntu.precise/ revision 2022).
 * leave cloud-init's locale generation code alone for 12.04
 * add the profile.d warning based on lp:~utlemming/cloud-init/cloud-init.profile.d and show that the first time the user logs in with invalid locale.

[Scott Moser (smoser)]

Fix-committed in revision 553.

[Scott Moser (smoser)]

Ugh.
I put a bad bug number in the changelog, but
cloud-init (0.6.3~bzr554-0ubuntu1) precise; urgency=low
    * Provide user-friendly message when a user ssh's in with an invalid
      locale (LP: #960547)

Marking fix-released in ubuntu

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/2211