Postgresql installation for MAAS fails on locales missing language packs

[Ante Karamatić]

Ubuntu 14.04 with all updates as of time of the writing.

MAAS - 1.5.4+bzr2294-0ubuntu1.1

# apt-get install maas
...
Setting up maas-dns (1.5.4+bzr2294-0ubuntu1.1) ...
 * Stopping domain name service... bind9 waiting for pid 29580 to die
                                                                                                                                                                                                              [ OK ]
 * Starting domain name service... bind9 [ OK ]
Setting up maas-region-controller (1.5.4+bzr2294-0ubuntu1.1) ...
locale: Cannot set LC_ALL to default locale: No such file or directory
Considering dependency proxy for proxy_http:
Module proxy already enabled
Module proxy_http already enabled
Module expires already enabled
Module wsgi already enabled
rsyslog stop/waiting
rsyslog start/running, process 32763
squid-deb-proxy stop/waiting
squid-deb-proxy start/running, process 349
 * Restarting message broker rabbitmq-server [ OK ]
Creating user "maas_longpoll" ...
...done.
Creating vhost "/maas_longpoll" ...
...done.
Setting permissions for user "maas_longpoll" in vhost "/maas_longpoll" ...
...done.
Creating user "maas_workers" ...
...done.
Creating vhost "/maas_workers" ...
...done.
Setting permissions for user "maas_workers" in vhost "/maas_workers" ...
...done.
 * No PostgreSQL clusters exist; see "man pg_createcluster"
dbconfig-common: writing config to /etc/dbconfig-common/maas-region-controller.conf

Creating config file /etc/dbconfig-common/maas-region-controller.conf with new version
unable to connect to postgresql server.
error encountered creating user:
psql: could not connect to server: No such file or directory Is the server running locally and accepting connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
/usr/bin/locale: Cannot set LC_ALL to default locale: No such file or directory

Here's the environment:
# export
declare -x HOME="/root"
declare -x LANG="en_US.UTF-8"
declare -x LANGUAGE="en_US:en"
declare -x LC_ADDRESS="hr_HR.UTF-8"
declare -x LC_IDENTIFICATION="hr_HR.UTF-8"
declare -x LC_MEASUREMENT="hr_HR.UTF-8"
declare -x LC_MONETARY="hr_HR.UTF-8"
declare -x LC_NAME="hr_HR.UTF-8"
declare -x LC_NUMERIC="hr_HR.UTF-8"
declare -x LC_PAPER="hr_HR.UTF-8"
declare -x LC_TELEPHONE="hr_HR.UTF-8"
declare -x LC_TIME="hr_HR.UTF-8"
declare -x LESSCLOSE="/usr/bin/lesspipe %s %s"
declare -x LESSOPEN="| /usr/bin/lesspipe %s"
declare -x LOGNAME="root"
declare -x LS_COLORS="rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36:"
declare -x MAIL="/var/mail/root"
declare -x OLDPWD
declare -x PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
declare -x PWD="/root"
declare -x SHELL="/bin/bash"
declare -x SHLVL="1"
declare -x SUDO_COMMAND="/bin/bash"
declare -x SUDO_GID="1000"
declare -x SUDO_UID="1000"
declare -x SUDO_USER="ubuntu"
declare -x TERM="xterm"
declare -x USER="root"
declare -x USERNAME="root"

After installing language-pack-hr package, I was successful at installing MAAS. To fix it, I had to purge maas.* and postgresql.* packages and then install maas package again.

This could easily be postgresql packaging bug, but it also renders MAAS uninstallable.

[Christian Reis (kiko)]

I wonder if we should be installing with LANG=C -- do you think you want localized collation, float separators, currencies, etc for your MAAS installation?

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Jeroen T. Vermeulen (jtv)]

There doesn't seem to be any need to install with the C locale; en_US.UTF-8 works just as well. As does installing the language pack(s) to match the session's locale settings. The real problem is that we have no hook for doing either before apt installs our dependencies: it installs the dependencies first, and only then do we get a say.

As Julian pointed out on the bug that was merged into this one, it's arguably a bug in the postgres packaging for neither failing nor falling back on unicode when the system does not support the session's locale, or openssh for keeping a locally unsupported locale in the session. Maybe sudo is to blame even for neglicence in sanitising the environment. Addressing the problem there would have wider consequences. Personally I think we should create our own postgres cluster during installation, with settings we dictate, instead of relying on the default one.

[Ante Karamatić (ivoks)]

Kiko I don't really care about currencies and all. I use my desktop with my locale and I use it to connect to a server. For some odd reason, SSH chooses to enforce my desktop locale on the server too. Then I have to install locale on the server too, before installing postgresql or at least generate a fake one.

It's pretty serious issue with posgresql packaging, taking into account that by default Ubuntu's sshd happily uses locale that is defined on user's desktop :)

[Martin Pitt (pitti)]

> It's pretty serious issue with posgresql packaging, taking into account that by default Ubuntu's sshd happily uses locale that is defined on user's desktop :)

That misfeature of ssh was worked around in postgresql-common around the raring timeframe:

postgresql-common (145) unstable; urgency=low
[...]
  * debian/maintscripts-functions, configure_cluster(): Do not trust the
    locale from the environment, as programs like ssh and sudo propagate
    remote and user locale by default. Instead, only use the locale settings
    from /etc/environment and /etc/default/locale, to prevent trying to
    configure the default cluster with a nonexisting or hard to predict
    locale. (LP: #969462, also see Debian #700271)

 -- Christoph Berg <email address hidden> Mon, 10 Jun 2013 17:01:01 +0200

So if you ssh into a trusty system it should ignore the broken locale that comes from ssh, and instead just look at /etc/default/locale or /etc/environment. So as you said this affects trusty: What's the content of these files there on the system you ssh'ed to? If that actually says hr_HR.UTF-8 without the locales generated, then this is merely a misconfigured system. If OTOH this configures a different locale that is available (locale -a), then apparently the fix above is incomplete. For that I really need the contents of the above files.

Thanks!

[Ante Karamatić (ivoks)]

I sshed from trusty to trusty. On client, locales are generated, default and functioning normally. 'Server' Trusty was installed with en_US locale and hr_HR locale was not installed or configured.

Requested information:

root@maas:~# cat /etc/environment
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games"
root@maas:~# cat /etc/default/locale
LANG="en_US.UTF-8"
LANGUAGE="en_US:en"
root@maas:~# locale -a
C
C.UTF-8
en_US.utf8
POSIX
root@maas:~#

[Martin Pitt (pitti)]

I just tried this scenario (de_DE.UTF-8 locally, ssh into machine with just en_US.UTF-8 available), and apt install postgresql succeeds and creates a cluster successfully which runs with en_US.UTF-8. But the more I look at the bug description (which isn't particularly clear) I have the feeling that we aren't talking about the postgresql package installation, but instead maas directly calls pg_createcluster? If so, how exactly does it do that?

The "disregard environment and only look at /etc/default/locale" workaround is done in postgresql-X.Y's postinst, i. e. it only applies to package installation time. If you call pg_createcluster directly as root in a broken environment it won't help you. Now, in theory that hackish workaround from postinst time could be moved direclty into pg_createcluster, but I veto that as this is established behaviour and I don't want to break cases where people call "LANG=xx_YY pg_createcluster" (see the help of --locale in help output and manpage).

So if in MAAS you also can't rely on the environment, a similar hack like https://alioth.debian.org/scm/loggerhead/pkg-postgresql/postgresql-common/trunk/view/head:/debian/maintscripts-functions#L40 might be required?

[Christian Reis (kiko)]

Perfect Martin. That makes things much clearer. Thanks!

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Adam Stokes (adam-stokes)]

We are experiencing this issue with the cloud installer. Specifically when someone is ssh'ed into a remote machine to perform the installation.

For reference: https://github.com/Ubuntu-Solutions-Engineering/openstack-installer/issues/236

Our workaround is to make sure language-pack-en is installed.

[Andres Rodriguez (andreserl)]

Martin,

MAAS does not call pg_createcluster directly. We use dbconfing-common for db creation.

I think this would mean that there might be a bug in dbconfig-common?

[Martin Pitt (pitti)]

Yeah, very likely (I don't know this at all, and it's rather undermaintained in Debian, and completely unmaintained in Ubuntu).

[Adam Collard (adam-collard)]

I doubt this is a dbconfig-common issue - note that the issue is that the PostgreSQL *cluster* is missing, not the databases.

[Adam Collard (adam-collard)]

Seems like the workaround described in #7 is not working, and it's nothing to do with dbconfig-common:

ubuntu@darby:~$ locale
locale: Cannot set LC_ALL to default locale: No such file or directory
LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC=en_GB.UTF-8
LC_TIME=en_GB.UTF-8
LC_COLLATE="en_US.UTF-8"
LC_MONETARY=en_GB.UTF-8
LC_MESSAGES="en_US.UTF-8"
LC_PAPER=en_GB.UTF-8
LC_NAME=en_GB.UTF-8
LC_ADDRESS=en_GB.UTF-8
LC_TELEPHONE=en_GB.UTF-8
LC_MEASUREMENT=en_GB.UTF-8
LC_IDENTIFICATION=en_GB.UTF-8
LC_ALL=
ubuntu@darby:~$ cat /etc/environment
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games"
ubuntu@darby:~$ locale -a
C
C.UTF-8
en_US.utf8
POSIX
ubuntu@darby:~$ sudo apt-get install postgresql-9.3
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
  libpq5 postgresql-client-9.3 postgresql-client-common postgresql-common
  ssl-cert
Suggested packages:
  oidentd ident-server locales-all postgresql-doc-9.3 openssl-blacklist
The following NEW packages will be installed:
  libpq5 postgresql-9.3 postgresql-client-9.3 postgresql-client-common
  postgresql-common ssl-cert
0 upgraded, 6 newly installed, 0 to remove and 0 not upgraded.
Need to get 3677 kB of archives.
After this operation, 15.4 MB of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://us.archive.ubuntu.com//ubuntu/ trusty-updates/main libpq5 amd64 9.3.5-0ubuntu0.14.04.1 [80.6 kB]
Get:2 http://us.archive.ubuntu.com//ubuntu/ trusty/main postgresql-client-common all 154 [25.4 kB]
Get:3 http://us.archive.ubuntu.com//ubuntu/ trusty-updates/main postgresql-client-9.3 amd64 9.3.5-0ubuntu0.14.04.1 [782 kB]
Get:4 http://us.archive.ubuntu.com//ubuntu/ trusty/main ssl-cert all 1.0.33 [16.6 kB]
Get:5 http://us.archive.ubuntu.com//ubuntu/ trusty/main postgresql-common all 154 [103 kB]
Get:6 http://us.archive.ubuntu.com//ubuntu/ trusty-updates/main postgresql-9.3 amd64 9.3.5-0ubuntu0.14.04.1 [2670 kB]
Fetched 3677 kB in 0s (22.6 MB/s)
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
        LANGUAGE = (unset),
        LC_ALL = (unset),
        LC_TIME = "en_GB.UTF-8",
        LC_MONETARY = "en_GB.UTF-8",
        LC_ADDRESS = "en_GB.UTF-8",
        LC_TELEPHONE = "en_GB.UTF-8",
        LC_NAME = "en_GB.UTF-8",
        LC_MEASUREMENT = "en_GB.UTF-8",
        LC_IDENTIFICATION = "en_GB.UTF-8",
        LC_NUMERIC = "en_GB.UTF-8",
        LC_PAPER = "en_GB.UTF-8",
        LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
locale: Cannot set LC_ALL to default locale: No such file or directory
Preconfiguring packages ...
Selecting previously unselected package libpq5.
(Reading database ... 56021 files and directories currently installed.)
Preparing to unpack .../libpq5_9.3.5-0ubuntu0.14.04.1_amd64.deb ...
Unpacking libpq5 (9.3.5-0ubuntu0.14.04.1) ...
Selecting previously unselected package postgresql-client-common.
Preparing to unpack .../postgresql-client-common_154_all.deb ...
Unpacking postgresql-client-comm...

[Martin Pitt (pitti)]

Adam, what are your settings in /etc/default/locale ?

[Adam Collard (adam-collard)]

On 12 November 2014 14:00, Martin Pitt <email address hidden> wrote:

> Adam, what are your settings in /etc/default/locale ?
>

$ cat /etc/default/locale
# Created by cloud-init v. 0.7.5 on Wed, 12 Nov 2014 13:51:27 +0000
LANG="en_US.UTF-8"

I wonder if the issue here is that it's /just/ LANG being set there and not
LC_ALL?

[Adam Collard (adam-collard)]

On 12 November 2014 14:17, Adam Collard <email address hidden> wrote:

> On 12 November 2014 14:00, Martin Pitt <email address hidden> wrote:
>
>> Adam, what are your settings in /etc/default/locale ?
>>
>
> $ cat /etc/default/locale
> # Created by cloud-init v. 0.7.5 on Wed, 12 Nov 2014 13:51:27 +0000
> LANG="en_US.UTF-8"
>
> I wonder if the issue here is that it's /just/ LANG being set there and
> not LC_ALL?
>

Confirmed my own theory, if I do this again with having LC_ALL in
/etc/default/locale set to en_US.UTF-8 then I get a cluster as expected.

[Dan Poler (l-dan)]

This behaviour continues to exist into the 1.8 experimental releases. We're seeing it with locale settings for far east and for Spanish.

[Stuart Bishop (stub)]

Consider destroying the cluster created by package installation (or inhibiting its creation), and calling pg_createcluster yourself with the '--data-checksums' option available in PG 9.3. This will allow PostgreSQL to detect corrupted files and fail appropriately.

[Peter Sabaini (peter-sabaini)]

Still getting this behaviour with
# cat /etc/default/locale
LANG="en_US.UTF-8"

Versions:
maas 1.9.0+bzr4533-0ubuntu1~trusty1
postgresql 9.3+154ubuntu1

[Mario Splivalo (mariosplivalo)]

I'm only able to replicate this when ssh sends over my LC_*, so I end up on a server not having the locale I'm using on my client.

Weather this is postgres package issue or openssh-client package issue I do not know.

But this is easily circumvented by commenting out "SendEnv LANG LC_*" in /etc/ssh/ssh_config on your local workstation.

[Andres Rodriguez (andreserl)]

Based on the comments above, this doesn't really seem to be an issue with MAAS, but rather an issue with SSH and/or postgres. I've marked this bug as incomplete for MAAS until we (or someone else) can confirm this is an issue with MAAS.

[Andres Rodriguez (andreserl)]

Hi!

**This is an automated message**

We believe this is may no longer be an issue in the latest MAAS release. Due to the original date of the bug report, we are currently marking it as Invalid. If you believe this bug report still valid against the latest release of MAAS, or if you are still interested in this, please re-open this bug report.

Thanks