Failure when using ssh with a locale that is not configured on the server

[Julian Edwards]

If LC_ALL is not set (which seems to be the default on a few server installations I've done now), mid way through installing, you get this backtrace and then the installation just hangs. You can ctrl-c out of it but the package is left half configured.

Traceback (most recent call last):
  File "/usr/bin/django-admin", line 5, in <module>
    management.execute_from_command_line()
  File "/usr/lib/python2.7/dist-packages/django/core/management/__init__.py", line 443, in execute_from_command_line
    utility.execute()
  File "/usr/lib/python2.7/dist-packages/django/core/management/__init__.py", line 382, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/lib/python2.7/dist-packages/django/core/management/base.py", line 196, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/usr/lib/python2.7/dist-packages/django/core/management/base.py", line 232, in execute
    output = self.handle(*args, **options)
  File "/usr/lib/python2.7/dist-packages/django/core/management/base.py", line 371, in handle
    return self.handle_noargs(**options)
  File "/usr/lib/python2.7/dist-packages/south/management/commands/syncdb.py", line 90, in handle_noargs
    syncdb.Command().execute(**options)
  File "/usr/lib/python2.7/dist-packages/django/core/management/base.py", line 232, in execute
    output = self.handle(*args, **options)
  File "/usr/lib/python2.7/dist-packages/django/core/management/base.py", line 371, in handle
    return self.handle_noargs(**options)
  File "/usr/lib/python2.7/dist-packages/django/core/management/commands/syncdb.py", line 57, in handle_noargs
    cursor = connection.cursor()
  File "/usr/lib/python2.7/dist-packages/django/db/backends/__init__.py", line 308, in cursor
    cursor = util.CursorWrapper(self._cursor(), self)
  File "/usr/lib/python2.7/dist-packages/django/db/backends/postgresql_psycopg2/base.py", line 177, in _cursor
    self.connection = Database.connect(**conn_params)
  File "/usr/lib/python2.7/dist-packages/psycopg2/__init__.py", line 179, in connect
    connection_factory=connection_factory, async=async)
psycopg2.OperationalError: FATAL: password authentication failed for user "maas"
FATAL: password authentication failed for user "maas"

Related bugs:
 * bug 969462: [postgres] fails to start after install if invalid locale is set

[Raphaël Badin (rvb)]

Can you tell us with which package you're getting this?

I installed successfully the daily packages on quantal and raring canonistack instances (http://paste.ubuntu.com/5570250/ / http://paste.ubuntu.com/5570252/).

[Julian Edwards (julian-edwards)]

Your LC_ALL must be set then.

[Raphaël Badin (rvb)]

No it's not, see the two links I pasted.

[Julian Edwards (julian-edwards)]

The links aren't that useful because it's showing the post-facto state. If you could deliberately unset LC_ALL and then install the packages from scratch it would be closer to my own experience.

If it ends up installing ok for you, then there's some deep voodoo at work here! It's very repeatable on my quantal server installation.

[Dave Walker (davewalker)]

We have seen this previously on a bunch of packages, where the locale
isn't configured. We particularly saw this on cloud images, which
wouldn't have an understanding of the required locale until first
login (& it might not be installed.)

Personally, i feel the bug is the locale not being configured properly
on te platform.. but I appreciate the packaging should handle this
with more grace.

[Julian Edwards (julian-edwards)]

On 01/03/13 09:47, Dave Walker wrote:
> We have seen this previously on a bunch of packages, where the locale
> isn't configured. We particularly saw this on cloud images, which
> wouldn't have an understanding of the required locale until first
> login (& it might not be installed.)
>
> Personally, i feel the bug is the locale not being configured properly
> on te platform.. but I appreciate the packaging should handle this
> with more grace.
>

Completely agree on all counts.

[Raphaël Badin (rvb)]

> The links aren't that useful because it's showing the post-facto state. If you could deliberately unset LC_ALL and then install the
> packages from scratch it would be closer to my own experience.

That's what I did and it seems fine (easy to recreate on a canonistack instance): http://paste.ubuntu.com/5575795/

Am I missing something?

[Julian Edwards (julian-edwards)]

On 01/03/13 19:12, Raphaël Badin wrote:
>> The links aren't that useful because it's showing the post-facto state. If you could deliberately unset LC_ALL and then install the
>> packages from scratch it would be closer to my own experience.
>
> That's what I did and it seems fine (easy to recreate on a canonistack
> instance): http://paste.ubuntu.com/5575795/
>
> Am I missing something?
>

No. It's a mystery. :/

[Jeroen T. Vermeulen (jtv)]

Looks like this was caused by a missing en_US entry in Julian's /usr/share/locale.

I'm seeing another, but possibly related failure where the error is that the database is not running; filed as bug 1171696.

[Robie Basak (racb)]

> Personally, i feel the bug is the locale not being configured properly
> on te platform.. but I appreciate the packaging should handle this
> with more grace.

If LC_* isn't set, then AIUI locale "C" is implied; this should not need any further configuration and not break anything. I may be wrong though.

[Julian Edwards (julian-edwards)]

On 16/05/13 01:57, Robie Basak wrote:
>> Personally, i feel the bug is the locale not being configured properly
>> on te platform.. but I appreciate the packaging should handle this
>> with more grace.
>
> If LC_* isn't set, then AIUI locale "C" is implied; this should not need
> any further configuration and not break anything. I may be wrong though.
>

I know what is going wrong now.

ssh sets my LC_ to my client-side locale which happens to not exist on
the server side.

That is what screws up packaging - it fails when it gets to the postgres
parts.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Robie Basak (racb)]

When I ssh in to a Precise or Saucy server using:

LC_ALL=en_DK.UTF-8 ssh <server>

I get the message:

WARNING! Your environment specifies an invalid locale.
 This can affect your user experience significantly, including the
 ability to manage packages. You may install the locales by running:

   sudo apt-get install language-pack-en
     or
   sudo locale-gen en_DK.UTF-8

To see all available language packs, run:
   apt-cache search "^language-pack-[a-z][a-z]$"
To disable this message for all users, run:
   sudo touch /var/lib/cloud/instance/locale-check.skip

Do you guys see this?

I think it's a general problem that doesn't just apply to maas here, and so perhaps an appropriate fix should be applied more generally?

If the server doesn't have a particular locale configured, it cannot be expected to work. I think the solutions are:
    1) Tell the user to generate the locale.
    2) Switch the locale to C or C.UTF-8 (not sure which) so that at least things will work until the system locale is configured as required.

I was going to suggest (1) but then found that there's already something in place for this. Is this working for you?

[Julian Edwards (julian-edwards)]

No, I never saw this sort of message at all. I concur with falling back to a C locale though, because as it stands package installation of Postgres just *hangs*.

[Robie Basak (racb)]

@Julian

Do you have a ~/.cloud-locale-test.skip file present? If not, do you see a /etc/profile.d/Z99-cloud-locale-test.sh file? The warning is provided by cloud-init; I wonder what version provides it, and if we need to SRU it? What version of cloud-init do you have?

I think the root cause is:

1. A user remote to the system has a different locale set that isn't configured on the system. This seems reasonable to me, and perhaps is going to become even more common that it is now.

2. This user uses ssh to install maas.

3. The postgresql-9.1 postinst fails with:

    Error: The locale requested by the environment is invalid.
    Error: could not create default cluster. Please create it manually with

      pg_createcluster 9.1 main --start

    or a similar command (see 'man pg_createcluster').

4. The subsequent DB initialisation needed by maas fails.

Solutions (one more added):

1) Tell the user to generate the locale, and do nothing else.

2) Switch the locale to C or C.UTF-8 (not sure which) so that at least things will work until the system locale is configured as required.

3) Require that everything functions correctly with an invalid locale set, including the postgresql-9.1 postinst.

The second option would require a modification to cloud-init, so adding a task for that and for feedback.

[Scott Moser (smoser)]

imo, this is either a bug in postgres because it fails to install when locale is set to invalid, or its user error.
I'm not sure if whether or not it will become more or less common.

More generally, there is a question of:
 is installing packages on ubuntu with an invalid locale a supported operation?

If so, then postgres has a bug (as do some other packages).

If not, then I'm not really sure what to do.

Martin Pitt (postgres maintainer for ubuntu) clearly stated it is not when closing bug 969462 as 'Invalid'

[Julian Edwards (julian-edwards)]

It's clear that PG has a packaging bug, it should not hang the postinst script because of an invalid locale. That bug ought to be triaged and this can be a dupe of it.

@Robie
Yes I see a /etc/profile.d/Z99-cloud-locale-test.sh but not the skip file. I can recreate this trivially with new canonistack instances:

ed@beast:~$ echo $LC_TIME
en_GB.utf8
ed@beast:~$ ssh 10.55.60.195
The authenticity of host '10.55.60.195 (<no hostip for proxy command>)' can't be established.
ECDSA key fingerprint is 42:7b:4f:79:b2:2c:95:d7:ce:32:90:d4:c5:6c:f3:3f.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.55.60.195' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 12.04.2 LTS (GNU/Linux 3.2.0-43-virtual x86_64)

 * Documentation: https://help.ubuntu.com/
...
[snip]
...
ubuntu@server-72566baa-6d58-40c5-a382-0fef613e7256:~$ echo $LC_TIME
en_GB.utf8
ubuntu@server-72566baa-6d58-40c5-a382-0fef613e7256:~$ locale
locale: Cannot set LC_ALL to default locale: No such file or directory
LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME=en_GB.utf8
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=
ubuntu@server-72566baa-6d58-40c5-a382-0fef613e7256:~$ cat ~/.cloud-locale-test.skip
cat: /home/ubuntu/.cloud-locale-test.skip: No such file or directory
ubuntu@server-72566baa-6d58-40c5-a382-0fef613e7256:~$ ls /etc/profile.d/Z99-cloud-locale-test.sh
/etc/profile.d/Z99-cloud-locale-test.sh

[Robie Basak (racb)]

I've started a discussion thread at: https://lists.ubuntu.com/archives/ubuntu-devel/2013-June/037195.html

[Robie Basak (racb)]

Following discussion in that thread and on IRC, we have a compromised, explained in bug 969462. The postgres package will ignore the environmental locale and use /etc/default/locale or /etc/environment or fall back to C. This should fix the issue.

In case the workaround isn't clear, it is:

LC_ALL=C apt-get install maas