Support using translations in devstack

Bug #995287 reported by Monty Taylor on 2012-05-05
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
High
Huang Zhiteng
Glance
High
Mark McLoughlin
OpenStack Compute (nova)
High
Mark McLoughlin
OpenStack Identity (keystone)
High
Mark McLoughlin
devstack
High
Unassigned
oslo-incubator
High
Mark McLoughlin

Bug Description

We go to the trouble to install translations catalogs, but then we don't properly use them. We really need to fix that. There is a great example of how to use in-tree messages can be found in the sphinx source:

https://bitbucket.org/birkenfeld/sphinx/src/5d4cd2cca317/sphinx/locale/__init__.py

Monty Taylor (mordred) on 2012-05-05
Changed in openstack-ci:
assignee: wpc (clarkb) → Clark Boylan (cboylan)
Clark Boylan (cboylan) wrote :

There appear to be two issues with the current Nova translation setup. First the .pot file is generated from trunk branch on LP which is quite out of date. This means the translation files in LP are generated with old data. Need to sync to LP properly or start looking at transifex and make sure transifex has an up to date .pot.

Second, at runtime Nova installs the translations but appears to only look in the default location for the translation files. You can probably only expect these to be there when installing from a package that installs to normal system locations. In many cases these files may be in a python egg or non system locations. Need to make the translation installation a little smarter about where it looks for things. The sphinx example above appears to deal with this issue.

Thierry Carrez (ttx) wrote :

Removing nova task until we're sure something actually needs to land into Nova code to fix that.

tags: added: i18n
no longer affects: nova
Monty Taylor (mordred) on 2012-09-24
Changed in openstack-ci:
milestone: folsom → none
assignee: Clark Boylan (cboylan) → nobody
status: Triaged → Won't Fix
Monty Taylor (mordred) on 2012-11-25
Changed in openstack-ci:
status: Won't Fix → Triaged
assignee: nobody → matthew wagoner (matthew-wagoner)
milestone: none → grizzly
Mark McLoughlin (markmc) wrote :

Came up here again: http://lists.openstack.org/pipermail/openstack-dev/2013-January/004289.html

In Nova, we do:

  gettext.install('nova', unicode=1)

but we install our .po message catalogs in $sitelib/nova/locale

However, by default gettext looks for the binary .mo message catalogs in /usr/share

So, we need (1) to ensure we compile the message catalogs using babel at install time and (2) tell gettext to look for wherever we install them if we don't install them in /usr/share/local

Monty Taylor (mordred) wrote :

https://bitbucket.org/birkenfeld/sphinx/src/7c437d4e4f10b14889585d5b41d5d9c634ea1503/sphinx/locale/__init__.py?at=default

The example of how to properly use stuff is ^^^

Adding to nova, as it's actually not really a infra bug.

no longer affects: openstack-ci
Sean Dague (sdague) on 2013-02-19
Changed in nova:
importance: Undecided → High
status: New → Confirmed
milestone: none → grizzly-rc1
Changed in nova:
status: Confirmed → Triaged
Russell Bryant (russellb) wrote :

This bug has been around a while and nobody is working on it. We have lived with it this long so I don't think it should block grizzly-rc1. I'm moving it to the grizzly-rc-potential list, instead.

Changed in nova:
milestone: grizzly-rc1 → none
tags: added: grizzly-rc-potential

fyi, there is at least one oslo bug that's related - https://bugs.launchpad.net/oslo/+bug/1076285 - Net, i don't think we will make any progress by Grizzly IMHO.

Thierry Carrez (ttx) on 2013-03-25
tags: removed: grizzly-rc-potential
Mark McLoughlin (markmc) wrote :

Summary of what the sphinx code Monty points to actually does:

  1) it provides a l_() function which delays the actual translation of the string until the string's value is needed. This appears to be for the case where a module wants to define a translatable string at import time, but not have the translation happen until runtime. With C, you'd similarly mark a string using N_() and translate it using gettext() at runtime

  2) it has a locale.init() function which allows you to pass in multiple localedirs and it will use translations from any .mo files found in those dirs by merging them all into a single catalog. This is basically an improved gettext.install().

I don't know any cases where we need (1) yet - but we might in future.

I also don't think fixing this bug requires (2) - for any given translation domain, there's only ever a single directory in any installation where we need to look up translations.

Mark McLoughlin (markmc) wrote :

The issue here is that, for each translation domain, we need to support configuring the localedir.

At first thought, it might seem like this should be configurable via the CLI and config files (i.e. using cfg) but this needs to be configured before we ever translate a single message ... and we need to support translating messages before cfg is ever called.

In C programs, you'd typically configure this at build-time and the localedir would be baked into the binaries. We don't really have an equivalent of that with Python.

The default localedir in Fedora is /usr/share/locale - so if we install nova.mo under /usr/share/locale/$lang/LC_MESSAGES then what we do now actually works fine. I assume this is true of all distros.

The case where we want to configure an alternative localedir is devstack - e.g. if you run 'python setup.py compile_catatlog', then nova.mo will be under /opt/stack/nova/nova/locale/$lang/LC_MESSAGES/

I think we should simply add support for this via env variables e.g.

  gettext.install('nova', unicode=1, localedir=os.environ.get('NOVA_LOCALEDIR'))

and you'd run e.g. nova-api in devstack with:

  $> NOVA_LOCALEDIR=/opt/stack/nova/nova/locale /opt/stack/nova/bin/nova-api

Mark McLoughlin (markmc) wrote :

We don't yet actually translate any of the messages in our libraries but, if/when we do that, we'll also need to support configuring the localedir they use.

e.g.

  gettext.translation('quantumclient', fallback=True, localedir=os.environ.get('QUANTUMCLIENT_LOCALEDIR'))

We need to have a env variable for each translation domain because, again, for devstack we want them picked up from directories like /opt/stack/python-quantumclient/quantumclient/locale

summary: - actually use translated strings
+ Support using translations in devstack
Monty Taylor (mordred) wrote :

Excellent summary Mark, and I like the optional override through env var approach.

Mark McLoughlin (markmc) wrote :

The Oslo task here is about adding gettextutils helpers for the LOCALEDIR env vars

Changed in oslo:
status: New → Triaged
importance: Undecided → High
assignee: nobody → Mark McLoughlin (markmc)
Changed in nova:
assignee: nobody → Mark McLoughlin (markmc)
Mark McLoughlin (markmc) wrote :

For Nova, we want to support the NOVA_LOCALEDIR env var

Mark McLoughlin (markmc) wrote :

In devstack, we need to do two things - a) run 'setup.py compile_catalog' and b) set NOVA_LOCALEDIR

Changed in devstack:
assignee: nobody → Mark McLoughlin (markmc)
status: New → Confirmed

Fix proposed to branch: master
Review: https://review.openstack.org/25820

Changed in oslo:
status: Triaged → In Progress

Fix proposed to branch: master
Review: https://review.openstack.org/25821

Changed in nova:
status: Triaged → In Progress

Reviewed: https://review.openstack.org/25820
Committed: http://github.com/openstack/oslo-incubator/commit/d8b66c7c1bb28672402f2bf1e410a1ee8fc4bfb6
Submitter: Jenkins
Branch: master

commit d8b66c7c1bb28672402f2bf1e410a1ee8fc4bfb6
Author: Mark McLoughlin <email address hidden>
Date: Mon Apr 1 02:01:00 2013 +0100

    Add a gettextutils.install() helper function

    Part of fixing bug #995287

    Every top-level script in every project needs to call gettext.install()
    so that a _() builtin function is installed and strings are translated
    using the correct translation domain.

    However, translations are always installed in the default localedir
    (which is commonly '/usr/share/locale') and, in cases like devstack, may
    be found in a project-specific localedir. To support such a use case we
    should have project-specific environment variables for overriding the
    default value of localedir - e.g. NOVA_LOCALEDIR.

    Add a new gettextutils.install() helper method to make this as easy as
    possible for projects to get right.

    Change-Id: I6c8549c8ff00797c96f2dd4b0b5266d18d77aa19

Changed in oslo:
status: In Progress → Fix Committed

Reviewed: https://review.openstack.org/25821
Committed: http://github.com/openstack/oslo-incubator/commit/c8f676bb54158d1a03aef231af920eeda9ebe1ba
Submitter: Jenkins
Branch: master

commit c8f676bb54158d1a03aef231af920eeda9ebe1ba
Author: Mark McLoughlin <email address hidden>
Date: Mon Apr 1 02:08:43 2013 +0100

    Support overriding oslo localedir too

    Part of fixing bug #995287

    Libraries need to translate strings according to their translation
    domain but they should not override the default _() builtin (which
    should only be installed by the top-level script) and instead should
    gettextutils._().

    To support the case where message catalogs are installed in a
    non-default (and perhaps project-specific) location, we allow the
    default localedir to be override with a project-specific environment
    variable e.g. QUANTUMCLIENT_LOCALEDIR.

    The code makes it seem like OSLO_LOCALEDIR is the env variable we're
    adding but, in fact, update.py magically replaces 'oslo' with the
    project name.

    Change-Id: I62b66892a4e27892ee488d2884ffd2f40ab467ee

Mark McLoughlin (markmc) on 2013-04-10
Changed in glance:
status: New → Triaged
importance: Undecided → High
assignee: nobody → Mark McLoughlin (markmc)

Fix proposed to branch: master
Review: https://review.openstack.org/26606

Changed in glance:
status: Triaged → In Progress
Dean Troyer (dtroyer) on 2013-04-14
Changed in devstack:
importance: Undecided → High
status: Confirmed → Triaged

Reviewed: https://review.openstack.org/25824
Committed: http://github.com/openstack/nova/commit/5e7ef210c01d3db8c79b969da3aeda50d57c4923
Submitter: Jenkins
Branch: master

commit 5e7ef210c01d3db8c79b969da3aeda50d57c4923
Author: Mark McLoughlin <email address hidden>
Date: Mon Apr 1 02:15:32 2013 +0100

    Add NOVA_LOCALEDIR env variable

    Part of fixing bug #995287

    Syncs these two commits from oslo-incubator:

      Support overriding oslo localedir too
      Add a gettextutils.install() helper function

    to get a new gettextutils.install() function which allows the default
    localedir to be overwritten via an environment variable.

    Note that gettextutils.install() must be called before any other nova
    modules are imported since some modules attempt to translate strings
    at import time (e.g. the 'message' attributes on classes in
    nova.exception). This is broken and inefficient, but fixing it involves
    adding something like spinx's l_() function and would be very invaisve.

    Also, note that calling gettextutils.install() in nova.cmd.__init__
    means that no program which uses a different translation domain should
    ever import any of the modules under nova.cmd.

    Change-Id: I86562b3a65d371673bb21f7179eecc7602bc0775

Changed in nova:
status: In Progress → Fix Committed
Mark McLoughlin (markmc) on 2013-05-08
Changed in keystone:
status: New → In Progress
importance: Undecided → High
assignee: nobody → Mark McLoughlin (markmc)
Mark McLoughlin (markmc) on 2013-05-08
Changed in cinder:
status: New → In Progress
importance: Undecided → High
assignee: nobody → Mark McLoughlin (markmc)
Changed in cinder:
assignee: Mark McLoughlin (markmc) → Huang Zhiteng (zhiteng-huang)

Reviewed: https://review.openstack.org/28544
Committed: http://github.com/openstack/cinder/commit/68f28d257c6e167bd819ae4ce385ae8c3602c8de
Submitter: Jenkins
Branch: master

commit 68f28d257c6e167bd819ae4ce385ae8c3602c8de
Author: Mark McLoughlin <email address hidden>
Date: Wed May 8 13:52:51 2013 +0100

    Add CINDER_LOCALEDIR env variable

    Part of fixing bug #995287

    Syncs these two commits from oslo-incubator:

      Support overriding oslo localedir too
      Add a gettextutils.install() helper function

    to get a new gettextutils.install() function which allows the default
    localedir to be overwritten via an environment variable.

    Note that gettextutils.install() must be called before any other cinder
    modules are imported since some modules attempt to translate strings
    at import time (e.g. in cinder.flags). This is broken and inefficient,
    but fixing it involves adding something like sphinx's l_() function and
    would be very invaisve.

    Change-Id: I86562b3a65d371673bb21f7179eecc7602bc0775

Changed in cinder:
status: In Progress → Fix Committed

Reviewed: https://review.openstack.org/26606
Committed: http://github.com/openstack/glance/commit/a0209e8757ae744202f94598fc788196e7fbf37c
Submitter: Jenkins
Branch: master

commit a0209e8757ae744202f94598fc788196e7fbf37c
Author: Mark McLoughlin <email address hidden>
Date: Wed Apr 10 12:12:03 2013 +0100

    Add GLANCE_LOCALEDIR env variable

    Part of fixing bug #995287

    Syncs these two commits from oslo-incubator:

      Support overriding oslo localedir too
      Add a gettextutils.install() helper function

    to get a new gettextutils.install() function which allows the default
    localedir to be overwritten via an environment variable.

    Note that gettextutils.install() must be called before any other glance
    modules are imported since some modules attempt to translate strings
    at import time (e.g. the help strings for config options in config.py).
    This is broken and inefficient, but fixing it involves adding something
    like spinx's l_() function and would be very invaisve.

    Also, note that calling gettextutils.install() in glance.cmd.__init__
    means that no program which uses a different translation domain should
    ever import any of the modules under glance.cmd.

    Change-Id: I86562b3a65d371673bb21f7179eecc7602bc0775

Changed in glance:
status: In Progress → Fix Committed
Thierry Carrez (ttx) on 2013-05-29
Changed in glance:
milestone: none → havana-1
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2013-05-29
Changed in cinder:
milestone: none → havana-1
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2013-05-29
Changed in nova:
milestone: none → havana-1
status: Fix Committed → Fix Released

Reviewed: https://review.openstack.org/28531
Committed: http://github.com/openstack/keystone/commit/7ce56d5f45f7112f0b25061473e64145fc840c98
Submitter: Jenkins
Branch: master

commit 7ce56d5f45f7112f0b25061473e64145fc840c98
Author: Mark McLoughlin <email address hidden>
Date: Wed May 8 12:07:07 2013 +0100

    Add KEYSTONE_LOCALEDIR env variable

    Part of fixing bug #995287

    Syncs these two commits from oslo-incubator:

      Support overriding oslo localedir too
      Add a gettextutils.install() helper function

    to get a new gettextutils.install() function which allows the default
    localedir to be overwritten via an environment variable.

    A few things to note:

     - the gettext.install() call is moved from common.config to the
       toplevel scripts to fix cases (e.g. the legacy auth_token middleware)
       where keystone code might be imported by a program who's default
       translation domain is not 'keystone'.

     - we add a gettext.install() call in keystone.test so that tests have
       the _() builtin installed.

    Change-Id: I86562b3a65d371673bb21f7179eecc7602bc0775

Changed in keystone:
status: In Progress → Fix Committed
Thierry Carrez (ttx) on 2013-05-30
Changed in oslo:
milestone: none → havana-1
status: Fix Committed → Fix Released
Mark McLoughlin (markmc) on 2013-07-02
Changed in devstack:
assignee: Mark McLoughlin (markmc) → nobody
Thierry Carrez (ttx) on 2013-07-17
Changed in keystone:
milestone: none → havana-2
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2013-10-17
Changed in oslo:
milestone: havana-1 → 2013.2
Thierry Carrez (ttx) on 2013-10-17
Changed in glance:
milestone: havana-1 → 2013.2
Changed in cinder:
milestone: havana-1 → 2013.2
Thierry Carrez (ttx) on 2013-10-17
Changed in nova:
milestone: havana-1 → 2013.2
Thierry Carrez (ttx) on 2013-10-17
Changed in keystone:
milestone: havana-2 → 2013.2
Sean Dague (sdague) wrote :

This bug was last updated over 180 days ago, as devstack is a fast moving project
and we'd like to get the tracker down to currently actionable bugs, this is getting
marked as Invalid. If the issue still exists, please feel free to reopen it.

Changed in devstack:
status: Triaged → Invalid
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers