Support using translations in devstack

[Monty Taylor]

We go to the trouble to install translations catalogs, but then we don't properly use them. We really need to fix that. There is a great example of how to use in-tree messages can be found in the sphinx source:

https://bitbucket.org/birkenfeld/sphinx/src/5d4cd2cca317/sphinx/locale/__init__.py

[Clark Boylan (cboylan)]

There appear to be two issues with the current Nova translation setup. First the .pot file is generated from trunk branch on LP which is quite out of date. This means the translation files in LP are generated with old data. Need to sync to LP properly or start looking at transifex and make sure transifex has an up to date .pot.

Second, at runtime Nova installs the translations but appears to only look in the default location for the translation files. You can probably only expect these to be there when installing from a package that installs to normal system locations. In many cases these files may be in a python egg or non system locations. Need to make the translation installation a little smarter about where it looks for things. The sphinx example above appears to deal with this issue.

[Thierry Carrez (ttx)]

Removing nova task until we're sure something actually needs to land into Nova code to fix that.

[Mark McLoughlin (markmc)]

Came up here again: http://lists.openstack.org/pipermail/openstack-dev/2013-January/004289.html

In Nova, we do:

  gettext.install('nova', unicode=1)

but we install our .po message catalogs in $sitelib/nova/locale

However, by default gettext looks for the binary .mo message catalogs in /usr/share

So, we need (1) to ensure we compile the message catalogs using babel at install time and (2) tell gettext to look for wherever we install them if we don't install them in /usr/share/local

[Monty Taylor (mordred)]

https://bitbucket.org/birkenfeld/sphinx/src/7c437d4e4f10b14889585d5b41d5d9c634ea1503/sphinx/locale/__init__.py?at=default

The example of how to properly use stuff is ^^^

Adding to nova, as it's actually not really a infra bug.

[Russell Bryant (russellb)]

This bug has been around a while and nobody is working on it. We have lived with it this long so I don't think it should block grizzly-rc1. I'm moving it to the grizzly-rc-potential list, instead.

[Davanum Srinivas (DIMS) (dims-v)]

fyi, there is at least one oslo bug that's related - https://bugs.launchpad.net/oslo/+bug/1076285 - Net, i don't think we will make any progress by Grizzly IMHO.

[Mark McLoughlin (markmc)]

Summary of what the sphinx code Monty points to actually does:

  1) it provides a l_() function which delays the actual translation of the string until the string's value is needed. This appears to be for the case where a module wants to define a translatable string at import time, but not have the translation happen until runtime. With C, you'd similarly mark a string using N_() and translate it using gettext() at runtime

  2) it has a locale.init() function which allows you to pass in multiple localedirs and it will use translations from any .mo files found in those dirs by merging them all into a single catalog. This is basically an improved gettext.install().

I don't know any cases where we need (1) yet - but we might in future.

I also don't think fixing this bug requires (2) - for any given translation domain, there's only ever a single directory in any installation where we need to look up translations.

[Mark McLoughlin (markmc)]

The issue here is that, for each translation domain, we need to support configuring the localedir.

At first thought, it might seem like this should be configurable via the CLI and config files (i.e. using cfg) but this needs to be configured before we ever translate a single message ... and we need to support translating messages before cfg is ever called.

In C programs, you'd typically configure this at build-time and the localedir would be baked into the binaries. We don't really have an equivalent of that with Python.

The default localedir in Fedora is /usr/share/locale - so if we install nova.mo under /usr/share/locale/$lang/LC_MESSAGES then what we do now actually works fine. I assume this is true of all distros.

The case where we want to configure an alternative localedir is devstack - e.g. if you run 'python setup.py compile_catatlog', then nova.mo will be under /opt/stack/nova/nova/locale/$lang/LC_MESSAGES/

I think we should simply add support for this via env variables e.g.

  gettext.install('nova', unicode=1, localedir=os.environ.get('NOVA_LOCALEDIR'))

and you'd run e.g. nova-api in devstack with:

  $> NOVA_LOCALEDIR=/opt/stack/nova/nova/locale /opt/stack/nova/bin/nova-api

[Mark McLoughlin (markmc)]

We don't yet actually translate any of the messages in our libraries but, if/when we do that, we'll also need to support configuring the localedir they use.

e.g.

  gettext.translation('quantumclient', fallback=True, localedir=os.environ.get('QUANTUMCLIENT_LOCALEDIR'))

We need to have a env variable for each translation domain because, again, for devstack we want them picked up from directories like /opt/stack/python-quantumclient/quantumclient/locale

[Monty Taylor (mordred)]

Excellent summary Mark, and I like the optional override through env var approach.

[Mark McLoughlin (markmc)]

The Oslo task here is about adding gettextutils helpers for the LOCALEDIR env vars

[Mark McLoughlin (markmc)]

For Nova, we want to support the NOVA_LOCALEDIR env var

[Mark McLoughlin (markmc)]

In devstack, we need to do two things - a) run 'setup.py compile_catalog' and b) set NOVA_LOCALEDIR

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/25820

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/25821

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/25824

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/25820
Committed: http://github.com/openstack/oslo-incubator/commit/d8b66c7c1bb28672402f2bf1e410a1ee8fc4bfb6
Submitter: Jenkins
Branch: master

commit d8b66c7c1bb28672402f2bf1e410a1ee8fc4bfb6
Author: Mark McLoughlin <email address hidden>
Date: Mon Apr 1 02:01:00 2013 +0100

    Add a gettextutils.install() helper function

    Part of fixing bug #995287

    Every top-level script in every project needs to call gettext.install()
    so that a _() builtin function is installed and strings are translated
    using the correct translation domain.

    However, translations are always installed in the default localedir
    (which is commonly '/usr/share/locale') and, in cases like devstack, may
    be found in a project-specific localedir. To support such a use case we
    should have project-specific environment variables for overriding the
    default value of localedir - e.g. NOVA_LOCALEDIR.

    Add a new gettextutils.install() helper method to make this as easy as
    possible for projects to get right.

    Change-Id: I6c8549c8ff00797c96f2dd4b0b5266d18d77aa19

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/25821
Committed: http://github.com/openstack/oslo-incubator/commit/c8f676bb54158d1a03aef231af920eeda9ebe1ba
Submitter: Jenkins
Branch: master

commit c8f676bb54158d1a03aef231af920eeda9ebe1ba
Author: Mark McLoughlin <email address hidden>
Date: Mon Apr 1 02:08:43 2013 +0100

    Support overriding oslo localedir too

    Part of fixing bug #995287

    Libraries need to translate strings according to their translation
    domain but they should not override the default _() builtin (which
    should only be installed by the top-level script) and instead should
    gettextutils._().

    To support the case where message catalogs are installed in a
    non-default (and perhaps project-specific) location, we allow the
    default localedir to be override with a project-specific environment
    variable e.g. QUANTUMCLIENT_LOCALEDIR.

    The code makes it seem like OSLO_LOCALEDIR is the env variable we're
    adding but, in fact, update.py magically replaces 'oslo' with the
    project name.

    Change-Id: I62b66892a4e27892ee488d2884ffd2f40ab467ee

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/26606

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/25824
Committed: http://github.com/openstack/nova/commit/5e7ef210c01d3db8c79b969da3aeda50d57c4923
Submitter: Jenkins
Branch: master

commit 5e7ef210c01d3db8c79b969da3aeda50d57c4923
Author: Mark McLoughlin <email address hidden>
Date: Mon Apr 1 02:15:32 2013 +0100

    Add NOVA_LOCALEDIR env variable

    Part of fixing bug #995287

    Syncs these two commits from oslo-incubator:

      Support overriding oslo localedir too
      Add a gettextutils.install() helper function

    to get a new gettextutils.install() function which allows the default
    localedir to be overwritten via an environment variable.

    Note that gettextutils.install() must be called before any other nova
    modules are imported since some modules attempt to translate strings
    at import time (e.g. the 'message' attributes on classes in
    nova.exception). This is broken and inefficient, but fixing it involves
    adding something like spinx's l_() function and would be very invaisve.

    Also, note that calling gettextutils.install() in nova.cmd.__init__
    means that no program which uses a different translation domain should
    ever import any of the modules under nova.cmd.

    Change-Id: I86562b3a65d371673bb21f7179eecc7602bc0775

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/28531

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/28544

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/28544
Committed: http://github.com/openstack/cinder/commit/68f28d257c6e167bd819ae4ce385ae8c3602c8de
Submitter: Jenkins
Branch: master

commit 68f28d257c6e167bd819ae4ce385ae8c3602c8de
Author: Mark McLoughlin <email address hidden>
Date: Wed May 8 13:52:51 2013 +0100

    Add CINDER_LOCALEDIR env variable

    Part of fixing bug #995287

    Syncs these two commits from oslo-incubator:

      Support overriding oslo localedir too
      Add a gettextutils.install() helper function

    to get a new gettextutils.install() function which allows the default
    localedir to be overwritten via an environment variable.

    Note that gettextutils.install() must be called before any other cinder
    modules are imported since some modules attempt to translate strings
    at import time (e.g. in cinder.flags). This is broken and inefficient,
    but fixing it involves adding something like sphinx's l_() function and
    would be very invaisve.

    Change-Id: I86562b3a65d371673bb21f7179eecc7602bc0775

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/26606
Committed: http://github.com/openstack/glance/commit/a0209e8757ae744202f94598fc788196e7fbf37c
Submitter: Jenkins
Branch: master

commit a0209e8757ae744202f94598fc788196e7fbf37c
Author: Mark McLoughlin <email address hidden>
Date: Wed Apr 10 12:12:03 2013 +0100

    Add GLANCE_LOCALEDIR env variable

    Part of fixing bug #995287

    Syncs these two commits from oslo-incubator:

      Support overriding oslo localedir too
      Add a gettextutils.install() helper function

    to get a new gettextutils.install() function which allows the default
    localedir to be overwritten via an environment variable.

    Note that gettextutils.install() must be called before any other glance
    modules are imported since some modules attempt to translate strings
    at import time (e.g. the help strings for config options in config.py).
    This is broken and inefficient, but fixing it involves adding something
    like spinx's l_() function and would be very invaisve.

    Also, note that calling gettextutils.install() in glance.cmd.__init__
    means that no program which uses a different translation domain should
    ever import any of the modules under glance.cmd.

    Change-Id: I86562b3a65d371673bb21f7179eecc7602bc0775

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/28531
Committed: http://github.com/openstack/keystone/commit/7ce56d5f45f7112f0b25061473e64145fc840c98
Submitter: Jenkins
Branch: master

commit 7ce56d5f45f7112f0b25061473e64145fc840c98
Author: Mark McLoughlin <email address hidden>
Date: Wed May 8 12:07:07 2013 +0100

    Add KEYSTONE_LOCALEDIR env variable

    Part of fixing bug #995287

    Syncs these two commits from oslo-incubator:

      Support overriding oslo localedir too
      Add a gettextutils.install() helper function

    to get a new gettextutils.install() function which allows the default
    localedir to be overwritten via an environment variable.

    A few things to note:

     - the gettext.install() call is moved from common.config to the
       toplevel scripts to fix cases (e.g. the legacy auth_token middleware)
       where keystone code might be imported by a program who's default
       translation domain is not 'keystone'.

     - we add a gettext.install() call in keystone.test so that tests have
       the _() builtin installed.

    Change-Id: I86562b3a65d371673bb21f7179eecc7602bc0775

[Sean Dague (sdague)]

This bug was last updated over 180 days ago, as devstack is a fast moving project
and we'd like to get the tracker down to currently actionable bugs, this is getting
marked as Invalid. If the issue still exists, please feel free to reopen it.