Failure when using ssh with a locale that is not configured on the server

[Julian Edwards]

[impact]
The setup:

1) The user has LC_ALL set to a locale on their machine (or some other LC_* variable but it's usually LC_ALL).
2) This locale does not exist on an Ubuntu server.
3) The user sshes into the server.

ssh (at least with the settings that are default on Debian and Ubuntu and Fedora) send the value of LC_ALL to the remote server, where the default settings on Ubuntu allow this to be set in the environment of the process it launches.

Because we are now in a situation where LC_ALL is set to a value that is not valid for the system, various undesirable things happen, including ugly messages from cloud-init and perl complaining every time a perl process starts and the bug that the original description below describes.

[test case]
Configure the setup as described above. Ssh into the server and run "perl < /dev/null" and check for warnings.

[regression potential]
This adds a step to the startup of every login shell, which obviously is not entirely trivial. However, the new executable being invoked is very simple and even if it fails, the startup of the shell should not be inhibited.

[original description]

If LC_ALL is not set (which seems to be the default on a few server installations I've done now), mid way through installing, you get this backtrace and then the installation just hangs. You can ctrl-c out of it but the package is left half configured.

Traceback (most recent call last):
  File "/usr/bin/django-admin", line 5, in <module>
    management.execute_from_command_line()
  File "/usr/lib/python2.7/dist-packages/django/core/management/__init__.py", line 443, in execute_from_command_line
    utility.execute()
  File "/usr/lib/python2.7/dist-packages/django/core/management/__init__.py", line 382, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/lib/python2.7/dist-packages/django/core/management/base.py", line 196, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/usr/lib/python2.7/dist-packages/django/core/management/base.py", line 232, in execute
    output = self.handle(*args, **options)
  File "/usr/lib/python2.7/dist-packages/django/core/management/base.py", line 371, in handle
    return self.handle_noargs(**options)
  File "/usr/lib/python2.7/dist-packages/south/management/commands/syncdb.py", line 90, in handle_noargs
    syncdb.Command().execute(**options)
  File "/usr/lib/python2.7/dist-packages/django/core/management/base.py", line 232, in execute
    output = self.handle(*args, **options)
  File "/usr/lib/python2.7/dist-packages/django/core/management/base.py", line 371, in handle
    return self.handle_noargs(**options)
  File "/usr/lib/python2.7/dist-packages/django/core/management/commands/syncdb.py", line 57, in handle_noargs
    cursor = connection.cursor()
  File "/usr/lib/python2.7/dist-packages/django/db/backends/__init__.py", line 308, in cursor
    cursor = util.CursorWrapper(self._cursor(), self)
  File "/usr/lib/python2.7/dist-packages/django/db/backends/postgresql_psycopg2/base.py", line 177, in _cursor
    self.connection = Database.connect(**conn_params)
  File "/usr/lib/python2.7/dist-packages/psycopg2/__init__.py", line 179, in connect
    connection_factory=connection_factory, async=async)
psycopg2.OperationalError: FATAL: password authentication failed for user "maas"
FATAL: password authentication failed for user "maas"

Related bugs:
 * bug 969462: [postgres] fails to start after install if invalid locale is set

[Raphaël Badin (rvb)]

Can you tell us with which package you're getting this?

I installed successfully the daily packages on quantal and raring canonistack instances (http://paste.ubuntu.com/5570250/ / http://paste.ubuntu.com/5570252/).

[Julian Edwards (julian-edwards)]

Your LC_ALL must be set then.

[Raphaël Badin (rvb)]

No it's not, see the two links I pasted.

[Julian Edwards (julian-edwards)]

The links aren't that useful because it's showing the post-facto state. If you could deliberately unset LC_ALL and then install the packages from scratch it would be closer to my own experience.

If it ends up installing ok for you, then there's some deep voodoo at work here! It's very repeatable on my quantal server installation.

[Dave Walker (davewalker)]

We have seen this previously on a bunch of packages, where the locale
isn't configured. We particularly saw this on cloud images, which
wouldn't have an understanding of the required locale until first
login (& it might not be installed.)

Personally, i feel the bug is the locale not being configured properly
on te platform.. but I appreciate the packaging should handle this
with more grace.

[Julian Edwards (julian-edwards)]

On 01/03/13 09:47, Dave Walker wrote:
> We have seen this previously on a bunch of packages, where the locale
> isn't configured. We particularly saw this on cloud images, which
> wouldn't have an understanding of the required locale until first
> login (& it might not be installed.)
>
> Personally, i feel the bug is the locale not being configured properly
> on te platform.. but I appreciate the packaging should handle this
> with more grace.
>

Completely agree on all counts.

[Raphaël Badin (rvb)]

> The links aren't that useful because it's showing the post-facto state. If you could deliberately unset LC_ALL and then install the
> packages from scratch it would be closer to my own experience.

That's what I did and it seems fine (easy to recreate on a canonistack instance): http://paste.ubuntu.com/5575795/

Am I missing something?

[Julian Edwards (julian-edwards)]

On 01/03/13 19:12, Raphaël Badin wrote:
>> The links aren't that useful because it's showing the post-facto state. If you could deliberately unset LC_ALL and then install the
>> packages from scratch it would be closer to my own experience.
>
> That's what I did and it seems fine (easy to recreate on a canonistack
> instance): http://paste.ubuntu.com/5575795/
>
> Am I missing something?
>

No. It's a mystery. :/

[Jeroen T. Vermeulen (jtv)]

Looks like this was caused by a missing en_US entry in Julian's /usr/share/locale.

I'm seeing another, but possibly related failure where the error is that the database is not running; filed as bug 1171696.

[Robie Basak (racb)]

> Personally, i feel the bug is the locale not being configured properly
> on te platform.. but I appreciate the packaging should handle this
> with more grace.

If LC_* isn't set, then AIUI locale "C" is implied; this should not need any further configuration and not break anything. I may be wrong though.

[Julian Edwards (julian-edwards)]

On 16/05/13 01:57, Robie Basak wrote:
>> Personally, i feel the bug is the locale not being configured properly
>> on te platform.. but I appreciate the packaging should handle this
>> with more grace.
>
> If LC_* isn't set, then AIUI locale "C" is implied; this should not need
> any further configuration and not break anything. I may be wrong though.
>

I know what is going wrong now.

ssh sets my LC_ to my client-side locale which happens to not exist on
the server side.

That is what screws up packaging - it fails when it gets to the postgres
parts.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Robie Basak (racb)]

When I ssh in to a Precise or Saucy server using:

LC_ALL=en_DK.UTF-8 ssh <server>

I get the message:

WARNING! Your environment specifies an invalid locale.
 This can affect your user experience significantly, including the
 ability to manage packages. You may install the locales by running:

   sudo apt-get install language-pack-en
     or
   sudo locale-gen en_DK.UTF-8

To see all available language packs, run:
   apt-cache search "^language-pack-[a-z][a-z]$"
To disable this message for all users, run:
   sudo touch /var/lib/cloud/instance/locale-check.skip

Do you guys see this?

I think it's a general problem that doesn't just apply to maas here, and so perhaps an appropriate fix should be applied more generally?

If the server doesn't have a particular locale configured, it cannot be expected to work. I think the solutions are:
    1) Tell the user to generate the locale.
    2) Switch the locale to C or C.UTF-8 (not sure which) so that at least things will work until the system locale is configured as required.

I was going to suggest (1) but then found that there's already something in place for this. Is this working for you?

[Julian Edwards (julian-edwards)]

No, I never saw this sort of message at all. I concur with falling back to a C locale though, because as it stands package installation of Postgres just *hangs*.

[Robie Basak (racb)]

@Julian

Do you have a ~/.cloud-locale-test.skip file present? If not, do you see a /etc/profile.d/Z99-cloud-locale-test.sh file? The warning is provided by cloud-init; I wonder what version provides it, and if we need to SRU it? What version of cloud-init do you have?

I think the root cause is:

1. A user remote to the system has a different locale set that isn't configured on the system. This seems reasonable to me, and perhaps is going to become even more common that it is now.

2. This user uses ssh to install maas.

3. The postgresql-9.1 postinst fails with:

    Error: The locale requested by the environment is invalid.
    Error: could not create default cluster. Please create it manually with

      pg_createcluster 9.1 main --start

    or a similar command (see 'man pg_createcluster').

4. The subsequent DB initialisation needed by maas fails.

Solutions (one more added):

1) Tell the user to generate the locale, and do nothing else.

2) Switch the locale to C or C.UTF-8 (not sure which) so that at least things will work until the system locale is configured as required.

3) Require that everything functions correctly with an invalid locale set, including the postgresql-9.1 postinst.

The second option would require a modification to cloud-init, so adding a task for that and for feedback.

[Scott Moser (smoser)]

imo, this is either a bug in postgres because it fails to install when locale is set to invalid, or its user error.
I'm not sure if whether or not it will become more or less common.

More generally, there is a question of:
 is installing packages on ubuntu with an invalid locale a supported operation?

If so, then postgres has a bug (as do some other packages).

If not, then I'm not really sure what to do.

Martin Pitt (postgres maintainer for ubuntu) clearly stated it is not when closing bug 969462 as 'Invalid'

[Julian Edwards (julian-edwards)]

It's clear that PG has a packaging bug, it should not hang the postinst script because of an invalid locale. That bug ought to be triaged and this can be a dupe of it.

@Robie
Yes I see a /etc/profile.d/Z99-cloud-locale-test.sh but not the skip file. I can recreate this trivially with new canonistack instances:

ed@beast:~$ echo $LC_TIME
en_GB.utf8
ed@beast:~$ ssh 10.55.60.195
The authenticity of host '10.55.60.195 (<no hostip for proxy command>)' can't be established.
ECDSA key fingerprint is 42:7b:4f:79:b2:2c:95:d7:ce:32:90:d4:c5:6c:f3:3f.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.55.60.195' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 12.04.2 LTS (GNU/Linux 3.2.0-43-virtual x86_64)

 * Documentation: https://help.ubuntu.com/
...
[snip]
...
ubuntu@server-72566baa-6d58-40c5-a382-0fef613e7256:~$ echo $LC_TIME
en_GB.utf8
ubuntu@server-72566baa-6d58-40c5-a382-0fef613e7256:~$ locale
locale: Cannot set LC_ALL to default locale: No such file or directory
LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME=en_GB.utf8
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=
ubuntu@server-72566baa-6d58-40c5-a382-0fef613e7256:~$ cat ~/.cloud-locale-test.skip
cat: /home/ubuntu/.cloud-locale-test.skip: No such file or directory
ubuntu@server-72566baa-6d58-40c5-a382-0fef613e7256:~$ ls /etc/profile.d/Z99-cloud-locale-test.sh
/etc/profile.d/Z99-cloud-locale-test.sh

[Robie Basak (racb)]

I've started a discussion thread at: https://lists.ubuntu.com/archives/ubuntu-devel/2013-June/037195.html

[Robie Basak (racb)]

Following discussion in that thread and on IRC, we have a compromised, explained in bug 969462. The postgres package will ignore the environmental locale and use /etc/default/locale or /etc/environment or fall back to C. This should fix the issue.

In case the workaround isn't clear, it is:

LC_ALL=C apt-get install maas

[Steve Langasek (vorlon)]

There was a lot of discussion of this on IRC today between myself, cjwatson, and smoser.

The common cause for wrong locales being set is that OpenSSH is passing them through as environment variables from client to server, without consideration of whether they are valid on the server.

Colin and I are agreed that OpenSSH should avoid setting invalid locales in the target environment. An upstream discussion of how to handle this can be found in https://bugzilla.mindrot.org/show_bug.cgi?id=1346 (and in particular, https://bugzilla.mindrot.org/show_bug.cgi?id=1346#c38).

In the near-term absence of an upstream solution for OpenSSH, I believe that cloud-init behavior should be changed to approximate such a design.

In descending order of priority:

- We must never leave the user with a shell with a broken locale setting. This causes a variety of problems; it makes the perl interpreter particularly noisy and annoying, but it also means users will have the wrong character set (ASCII vs UTF-8) than if they had set no locale at all in their environment (in which case they would get C.UTF-8 by default).
- We must never show a loud (16-line-long!) banner message telling users how to configure locales if the user's configured invalid locale is English. Nobody who is going to be reading English anyway needs this to be in their face. They should get a working locale and be able to get on with it, and if differences in LC_TIME between C and en_UK get in their way, they can use the normal techniques to resolve this.
- We should not show *this* loud banner message to users on a minimal image. We already have a motd informing users of minimal images that their system is minimized and not in a state meant for human interactive use. Our minimized image should be less verbose and do less handholding than a regular cloud image, and that means we should not include this banner from cloud-init on minimized images.
- I think the message is of limited value overall to users of non-English locales. First, while most of the upstream software we ship is quite well localized, I believe we do a mediocre job of localizing the Ubuntu-specific messages in the Ubuntu server; even where i18n support exists, I don't believe we are championing l10n for them or picking up the translations. So installing the langpacks isn't going to give a great experience in the user's target language. Second, because of the preceding, I believe the vast majority of actual Ubuntu Server users are comfortable navigating a CLI in English even if it is not their first language. And third, if a user does absolutely need their CLI to be in their native language, I believe the value provided to this user by having a banner message in English, even with embedded apt commands, is marginal relative to letting the user google documentation in their own language. For these reasons, I think the banner message should be dropped entirely.

I am happy to provide a patch to address either 1-3 or 1-4, whichever achieves consensus. If we don't do 4), I am inclined to also patch the message to condense it down from its current full-terminal size.

[Dimitri John Ledkov (xnox)]

Commandline usage is often quite broken in non-english, for example, sometimes manpages --help output are over translated, with flag names/options being translated, whilst the command in question doesn't accept non-english flags. Or worse only accepts non-english once in a different locale. Leaving users extremely confused, when flags don't work - or it is impossible to discover the english name of a flag.

Huge banners of gibberish are very pointless. If you want to be nice, at least that banner text should be preshipped localised, into all languages. And it should be short, probably a URL.

non-UTF8 locale fallback to C, instead of C.UTF-8 is very bad.

I have disabled forwarding of my client locale to the server by default, and I long time ago advocated that that should be the default on Ubuntu. I very much suffer from this, because many minimal systems do not have en_GB.UTF-8 generated/configured, or explicitly have it removed. I believe things got better, but not nearly so, especially when testing localised installs and trying to ssh into them from an en_GB.UTF-8 client.

I might be weird, but I expect to get the default behaviour of the remote systems; when i login into the remote machine. Not the one of the local one. And I see it as a privacy / information leak, to pass my local locale when I ssh into remote systems. This goes against the whole thin-client/multi-user mainframe pagadim, which i think is a bit historic in its usage - these days we have vastly more servers & computers, than humans. Thus on average there is only a fraction of a human logged into an average server.

RE: privacy - when i ssh push code to a public codehosting site, I do not want them to collect my local machine locale.

The people who i typically see use localised terminals are those that happen to have a latin based keyboard, which has all the keys of the english alphabet, and they are missordered. So if things are changed w.r.t. this, one should double check that e.g. german azerty client, can still azerty all the things across. Everyone else with more national layouts with missing english keys, have most likely a secondary en_US, and switch to that keyboard layout for terminal and ssh use - and would very much prefer and expect an english/C.UTF-8 locale. I know it sounds odd to derive locale from a keyboard layout; but for an interactive terminal, it is very intuitive.

ps Also screw $variables, and how dare countries to have money symbols to not be a variant of a dollar.

[Scott Moser (smoser)]

I've linked a merge proposal. Please review that if you are interested.

[Julian Edwards (julian-edwards)]

Thanks for addressing this bug at last. It's bitten me so many times over the years that I've got used to resetting the locale in remote servers in my bashrc. The Perl thing is particularly insidious as it outputs to stderr and confuses at least one API that I use.

[Steve Langasek (vorlon)]

On Fri, Jun 15, 2018 at 10:16:47PM -0000, Julian Edwards wrote:
> Thanks for addressing this bug at last. It's bitten me so many times
> over the years that I've got used to resetting the locale in remote
> servers in my bashrc. The Perl thing is particularly insidious as it
> outputs to stderr and confuses at least one API that I use.

Inconveniently, the perl case isn't reasonably (IMHO) fixable in the context
of the cloud-init profile.d script. Ubuntu client systems set a healthy
collection of LC_* variables which are then passed over ssh, and 'locale'
will only tell us about a select few of them being invalid - whereas perl
balks if any of them are invalid.

So without either writing a lot of shell code to decipher which variables
are invalid without the help of locale, or Scott's approach of overriding
/all/ environment variables by setting LC_ALL, a complete solution to this
requires support in ssh.

[Michael Hudson-Doyle (mwhudson)]

I'm somewhat inclined to think that doing this in shell is a bit silly. How about something like https://paste.ubuntu.com/p/NX7yqDVGyf/? It should probably be in libc-bin (maybe even an flag to locale(1)) eventually but for now we could stuff into any Essential package (base-files or util-linux or ...)

[Michael Hudson-Doyle (mwhudson)]

Here is my proposed patch.

[Michael Hudson-Doyle (mwhudson)]

I've tested this in a minimal image and it appears to work as designed.

[Scott Moser (smoser)]

@mwhudson,
your suggested change seems reasonable to me.
I don't love the use of 'eval', but it seems reasonably safe here.

Instead of quoting you could just reject argv[1] input if it
had characters other than [a-zA-z_.-] . Perhaps that makes
this more difficult.

Also, maybe you should try to 'setlocale(argv[1])' to check that
it is valid ? Ie, as it is right now if input is bogus then
the program will exit success and write bogus output.

$ LC_ALL=asdf /tmp/my-test bogus ; echo $?
LC_ALL='bogus'
0

[Michael Hudson-Doyle (mwhudson)]

On Wed, 8 Aug 2018 at 02:46, Scott Moser <email address hidden> wrote:

> @mwhudson,
> your suggested change seems reasonable to me.
> I don't love the use of 'eval', but it seems reasonably safe here.
>

I don't either, but I don't think there is any alternative here is there?

> Instead of quoting you could just reject argv[1] input if it
> had characters other than [a-zA-z_.-] . Perhaps that makes
> this more difficult.
>

If I'd thought of this before writing the quote function I'd probably have
gone for it...

> Also, maybe you should try to 'setlocale(argv[1])' to check that
> it is valid ? Ie, as it is right now if input is bogus then
> the program will exit success and write bogus output.
>

Hmm maybe that would make sense. OTOH given the circumstances it's called
in I'm not sure this is very important.

I could ameliorate both your points by just hardcoding the default locale
to C.UTF-8...

[Robie Basak (racb)]

Can I suggest that profile.d/01-locale-check.sh is a misnomer and it should be called something like profile.d/01-locale-fix.sh instead, because it (potentially) actively changes things? OTOH /usr/bin/locale-check is fine because it doesn't change anything in itself.

[Launchpad Janitor (janitor)]

This bug was fixed in the package base-files - 10.1ubuntu5

---------------
base-files (10.1ubuntu5) cosmic; urgency=medium

  * Install locale-check command to /usr/bin and invoke it from
    /etc/profile.d/01-locale-fix.sh to ensure locale related environment
    variables are set to valid values. (LP: #1134036)

 -- Michael Hudson-Doyle <email address hidden> Tue, 07 Aug 2018 11:53:28 +1200

[Łukasz Zemczak (sil2100)]

Can we get a Test Case and Regression Potential in the description following SRU guidelines? Thank you!

[Michael Hudson-Doyle (mwhudson)]

Ah sorry, now done.

[Łukasz Zemczak (sil2100)]

I'm not a big fan of this approach but I certainly don't have any better idea myself. At least the new executable should be relatively fast. Anyway, let's 'just do it' in this case. I guess the additional libc dep is a no-op.

[Łukasz Zemczak (sil2100)]

Hello Julian, or anyone else affected,

Accepted base-files into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/base-files/10.1ubuntu2.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Michael Hudson-Doyle (mwhudson)]

I'm not incredibly happy about this approach (doing it in pam_env.so would be best I think) but it does the job for now. I verified this in a minimal bionic image with and without base-files from proposed and the behaviour was as expected.

[Launchpad Janitor (janitor)]

This bug was fixed in the package base-files - 10.1ubuntu2.3

---------------
base-files (10.1ubuntu2.3) bionic; urgency=medium

  * Install locale-check command to /usr/bin and invoke it from
    /etc/profile.d/01-locale-fix.sh to ensure locale related environment
    variables are set to valid values. (LP: #1134036)

 -- Michael Hudson-Doyle <email address hidden> Mon, 20 Aug 2018 11:44:58 +1200

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for base-files has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.