contextual-markers under-usage leads to babble looking translations

[Yves-Gwenael Bourhis]

Contextual markers: https://docs.djangoproject.com/en/dev/topics/i18n/translation/#contextual-markers
are used only in horizon.tables.actions.BatchAction._get_action_name in the whole code, and even there, the translation lacks of context.

Indeed, we have::

            if action_type == "past":
                msgstr = pgettext_lazy("past", "%(action)s %(data_type)s")
            else:
                msgstr = pgettext_lazy("present", "%(action)s %(data_type)s")

But "%(action)s and %(data_type)s" use "action" and "data_types" are already translations coming from a context agnostic translation.
This leads to buttons in French like "Redémarrage à Chaud Instances" (which is incorrect and should be "Redémarrage à chaud des instances") and "Terminer Instances" (which should be "Terminer les instances") (both the missing "les" (plural) and "l'" (singular) and the Camel-Case in the sentence, looks more like babbling rather than speaking in French, but I'm sure many other languages are impacted).
Currently The translators can do nothing about it and can not be blamed at all, simply because because the context is not handled, so they have to make a compromise of the "less babbling looking translation".

So BatchAction sublcasses should also use contextual-markers for their action_* and data_type_* attributes, because even there, translations are correct in a context, and wrong in an other (depending where it will be rendered) since "Instance" needs to be translated either "Instance" or "L'instance" depending on the context (and only Proper nouns have a capital letter inside a sentence in many languages like French so "L'instance" will also have to be "l'instance", etc...).

Many Translation errors are due to lack of contextual-markers in the translatable strings.

We should therefore add more contextual-markers in the translation strings and handle words which will be isolated or inside a sentence (for the capital letter on non-Nouns).

[Julie Pichon (jpichon)]

Thanks for looking into this. There is some context on the particular context marker you mention in bug 1253678 in that helps, more work in that area is definitely welcome.

[Julie Pichon (jpichon)]

Bug 1253174 and the related patch are probably of interest in relation to this patch.

[Yves-Gwenael Bourhis (yves-gwenael-bourhis)]

I reopened a django bug which was fixed with gettext_lazy, but not fixed with pgettext_lazy: https://code.djangoproject.com/ticket/19272#comment:7

So curently, you need to type::

    pgettext_lazy("context string", u"string to translate") # unicode format for string to translate

To avoid raising a "TypeError: Lazy object returned unexpected type." Exception.

[Yves-Gwenael Bourhis (yves-gwenael-bourhis)]

Fix proposed to branch: master
Review: https://review.openstack.org/91338

[Yves-Gwenael Bourhis (yves-gwenael-bourhis)]

reported django bug:
https://code.djangoproject.com/ticket/22565

[Yves-Gwenael Bourhis (yves-gwenael-bourhis)]

other django bug:
https://code.djangoproject.com/ticket/22566

[Yves-Gwenael Bourhis (yves-gwenael-bourhis)]

We could also use "Comments for translators" to avoid contextual markers in the code as per documented here:
https://docs.djangoproject.com/en/dev/topics/i18n/translation/#comments-for-translators
But this is subject to the following django ticket:
https://code.djangoproject.com/ticket/22578#ticket

[Yves-Gwenael Bourhis (yves-gwenael-bourhis)]

Note:
I don't know yet how Transifex will handle this, but the good news is that via the django_admin "makemessages" command, when a contextual marker is added, if a translation existed it is taken and mrked as "fuzzy" (so the translation is not lost).

[Yves-Gwenael Bourhis (yves-gwenael-bourhis)]

I noticed that we can't define singular and plural strings, and then decide ourselves when it is plural or singular because plural forms depend on the language as can be seen here:
http://www.gnu.org/savannah-checkouts/gnu/gettext/manual/html_node/Plural-forms.html
and here:
http://www.gnu.org/software/gettext/manual/html_node/Translating-plural-forms.html
and here:
http://localization-guide.readthedocs.org/en/latest/l10n/pluralforms.html

Some languages have 5 plural forms.
in English "0" is plural, while it's singular in French.
etc...
So we have no other solution than relying on ungettext(_lazy) for pluralization to work with internationalised strings.

We really need a method (or static method) receiving the number of items and returning an ungettext_lazy method call, otherwise the strings are not marked for multiple pluralization in pot files.

[Yves-Gwenael Bourhis (yves-gwenael-bourhis)]

New django (or GNU gettext?) issue:
https://code.djangoproject.com/ticket/22671#ticket

[Yves-Gwenael Bourhis (yves-gwenael-bourhis)]

For info, the following bug: https://code.djangoproject.com/ticket/22578#ticket is not a django bug but a GNU gettext bug:
https://savannah.gnu.org/bugs/index.php?42376

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/91338
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=f4594e81998f5d4418926d11cd36a680dd7735e1
Submitter: Jenkins
Branch: master

commit f4594e81998f5d4418926d11cd36a680dd7735e1
Author: Yves-Gwenael Bourhis <email address hidden>
Date: Wed Apr 30 13:35:49 2014 +0200

    Allow translators to translate BatchActions

    Currently BatchAction proposes only an action_present and action_past verb
    to be concatenated with a data_type_singular or data_type_plural string.

    The issue is that many languages need to conjugate the verb with the
    plural/singular subject, and some translations where terribly awfull and
    impossible to do.
    Indeed, some languages have up to 5 plural forms depending on the number of
    items.

    This patch introduces the fact that action_present is a method instead of an
    attribute, which in conjonction with https://review.openstack.org/57701 allows
    better translation to be performed.

    This patch also adds contextual mrkers to STATUS_DISPLAY_CHOICES,
    TASK_DISPLAY_CHOICES and POWER_DISPLAY_CHOICES because it was impossible to
    translate the stand-alone words out of context for a translator (i.e. the word
    "active" will be translated differently if the active object is feminine or
    masculine, since feminine and masculine depends also on the language, it is
    mission impossible for the translator to translate "active" if he does not know
    "what" is "active").

    It is important to use unicode strings with contextual markers because of the
    following django bug:
    https://code.djangoproject.com/ticket/22565
    The django community was very quick and responsive and fixed the bug very fast:
    https://code.djangoproject.com/changeset/142c27218a0bb07bb197faab7b1d8eb3a084f5b9
    and back-ported their fix in the 1.6 and 1.7 django branches:
    https://code.djangoproject.com/changeset/61fd00d4fd65f44d0a05be4c7e95124e4102ec27
    https://code.djangoproject.com/changeset/034866204b39f797c73997b03cd90443aa03aa91
    but since in horizon we support back to django 1.4, we will have to live with
    it by providing unicode strings until we drop support of old django versions.

    We can't introduce contextual markers and ungettext (pluralization) verywhere
    at once because Translators would totally loose their previous work. So we have
    to progressively introduce contextual markers and pluralization, when a string
    is changed, or for new strings. It is also the reason why this patch changes
    only the strings of the instance view actions.

    Since we can't use contextual markers and pluralization for all the code at
    once, this patch partially:

    partial-bug: 1307476

    Change-Id: I4cef42dc4a12f55cf750d9a18083254c6a96e716

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/121842

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/121879

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/121913

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/121957

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/121962

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/121972

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/121957
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=de81c575731a6d237fa6cc024bb6a206e1f36501
Submitter: Jenkins
Branch: master

commit de81c575731a6d237fa6cc024bb6a206e1f36501
Author: Doug Fish <email address hidden>
Date: Tue Sep 16 14:11:42 2014 -0500

    Fix concatenation in Delete Image action

    Remove concatenation and pluralization issues from Delete Image
    action

    Change-Id: I27d9c9f4c38d68183c16b5d23a6aba6f206d392c
    partial-bug: 1307476

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/121972
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=383366978ca5ae35c6e98cdd389278e39ef0f056
Submitter: Jenkins
Branch: master

commit 383366978ca5ae35c6e98cdd389278e39ef0f056
Author: Doug Fish <email address hidden>
Date: Tue Sep 16 15:09:50 2014 -0500

    Fix concatenation in Delete Rule action

    Remove concatenation and pluralization issues from Delete Rule
    action

    Change-Id: I1d98038339b31f81c27a2118d6a96fc4448f14e6
    partial-bug: 1307476

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/121962
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=a40f52f0974237398fff1ef14fb249df25af4e43
Submitter: Jenkins
Branch: master

commit a40f52f0974237398fff1ef14fb249df25af4e43
Author: Doug Fish <email address hidden>
Date: Tue Sep 16 14:30:42 2014 -0500

    Fix concatenation in network tables

    Remove concatenation and pluralization issues in network, subnet,
    port, interface,wq and router tables.

    Change-Id: I9ab759703505b188df0ec16b92f08d6f741ba944
    partial-bug: 1307476

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/121913
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=d40f9a91f7656fb259bd751f2c569e32ff1e85ce
Submitter: Jenkins
Branch: master

commit d40f9a91f7656fb259bd751f2c569e32ff1e85ce
Author: Doug Fish <email address hidden>
Date: Tue Sep 16 10:40:25 2014 -0500

    Remove concatenation from Delete Flavors

    Fix concatenation and pluralization problems with the Delete
    Flavors action

    Change-Id: I15f0b6cf0e12112687dd70fae48e9074202c83be
    partial-bug: 1307476

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/121879
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=2d4c249fe8f929e4efc168a766719f53e4d199db
Submitter: Jenkins
Branch: master

commit 2d4c249fe8f929e4efc168a766719f53e4d199db
Author: Doug Fish <email address hidden>
Date: Tue Sep 16 09:34:52 2014 -0500

    Update Users Table to Avoid Concatenation

    Updated DeleteUsersAction and ToggleEnable to avoid concatenation
    and handle pluralizing correctly.

    Change-Id: Ia834a4f46ed3bf725ac7d7b6fbed849694c1916d
    partial-bug: 1307476

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/121842
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=5803d802e5919d0fea30484efd08b79a885a9f9f
Submitter: Jenkins
Branch: master

commit 5803d802e5919d0fea30484efd08b79a885a9f9f
Author: Doug Fish <email address hidden>
Date: Tue Sep 16 09:04:52 2014 -0500

    Update Usage Comments for DeleteAction

    Update the attributes for DeleteAction to explain updated usage
    and note problem attributes which will be deprecated. Add note
    to BatchAction action_past.

    Change-Id: Ie50efae19711618f2a8caea84d45a648b31f7703
    partial-bug: 1307476

[Doug Fish (drfish)]

I've wrapped up the work I intend for the Juno time frame. After discussion with ygbo I'm leaving this open so that additional improvements can be completed.

[Doug Fish (drfish)]

Taking ownership again. I have a new pile of concatenations I'm working:
Delete Rule
Delete Rules
Confirm Delete Rule
Confirm Delete Rules
Delete Key Pair
Delete Key Pairs
Confirm Delete Key Pair
Confirm Delete Key Pairs
Release Floating IP
Release Floating Ips
Confirm Release Floating IP
Confirm Release Floating IPs
Delete Image
Delete Host Aggregate
Delete Host Aggregates
Confirm Delete Host Aggregate
Confirm Delete Host Aggregates
Delete Flavor
Confirm Delete Flavor
Confirm Delete Flavors
Delete ExtraSpec
Delete ExtraSpecs
Confirm Delete ExtraSpec
Confirm Delete ExtraSpecs
Delete Propertise
Delete Propertises
Confirm Delete Propertise
Confirm Delete Propertises
Confirm Delete Subnet
Confirm Delete Subnets
Confirm Delete Port
Delete DHCP Agent
Delete DHCP Agents
Confirm Delete DHCP Agent
Confirm Delete DHCP Agents
Delete Volume
Delete Volumes
Confirm Delete Volume
Confirm Delete Volumes
Delete Volume Snapshot
Delete Volume Snapshots
Confirm Delete Volume Snapshot
Confirm Delete Volume Snapshot

[Doug Fish (drfish)]

Oops! that overstates it a bit. The "Confirm" ones get fixed for free.

Delete Rule
Delete Rules
Delete Key Pair
Delete Key Pairs
Release Floating IP
Release Floating Ips
Delete Image
Delete Host Aggregate
Delete Host Aggregates
Delete Flavor
Delete ExtraSpec
Delete ExtraSpecs
Delete Propertise
Delete Propertises
Delete DHCP Agent
Delete DHCP Agents
Delete Volume
Delete Volumes
Delete Volume Snapshot
Delete Volume Snapshots

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/123820

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/123834

[Doug Fish (drfish)]

I updated segments for these actions
Delete Volume
Delete Volumes
Delete Volume Snapshot
Delete Volume Snapshots
Delete Rule
Delete Rules
Delete Host Aggregate
Delete Host Aggregates
Delete DHCP Agent
Delete DHCP Agents
-----------------------
Also I discovered these actions and updated them
Delete Volume Type
Delete Volume Types
Delete Qos Spec
Delete Qos Specs
Delete Policy
Delete Policies
Delete Firewall
Delete Firewalls
---------
other items in the list appeared to be either already translated, or couldn't be found

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/123843

[Doug Fish (drfish)]

Pro tip:
grep -r "data_type_pl" .

will generate a list of improper actions that remain. There are more than I had hoped.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/123878

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/124041

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/124053

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/123820
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=60b376882d53d29355a73a875a1762f7cb4c3d57
Submitter: Jenkins
Branch: master

commit 60b376882d53d29355a73a875a1762f7cb4c3d57
Author: Doug Fish <email address hidden>
Date: Wed Sep 24 14:00:47 2014 -0500

    Fix concatenation in Volume related actions

    Fix concatenation and pluralization issues for Volume,
    Volume Type, Volume Snapshot, Extra Spec and QoS Specs
    related actions.

    Change-Id: I9ec9caeb0103663f34fc24a5bddd6276ff8b37e7
    partial-bug: 1307476

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/123834
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=4061516fb9cd7ce34146f67a17ae67231ca46ea8
Submitter: Jenkins
Branch: master

commit 4061516fb9cd7ce34146f67a17ae67231ca46ea8
Author: Doug Fish <email address hidden>
Date: Wed Sep 24 14:49:48 2014 -0500

    Fix concatenation in firewall related actions

    Remove concatenation and pluralization issues from firewall
    Rules, Policies, and Link actions.

    Change-Id: I7e00d2d56d22eddcc973ea325dc6d49ba1beb797
    partial-bug: 1307476

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/123843
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=151ce127e98e7c440d5b145931517597fdfcb30e
Submitter: Jenkins
Branch: master

commit 151ce127e98e7c440d5b145931517597fdfcb30e
Author: Doug Fish <email address hidden>
Date: Wed Sep 24 15:25:51 2014 -0500

    Fix concatenation is misc delete actions

    Remove concatenation and pluralization issues from Delete
    Host Aggregates, and Delete DHCP Agents actions

    Change-Id: I1565d84002c785f5c5fb99b87dbb0877a4f577f3
    partial-bug: 1307476

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/124124

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/124129

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/124041
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=6a7791f6366abb574ed5b673cb926c4811f7270e
Submitter: Jenkins
Branch: master

commit 6a7791f6366abb574ed5b673cb926c4811f7270e
Author: Doug Fish <email address hidden>
Date: Thu Sep 25 07:42:58 2014 -0500

    Fix concatenation in data processing actions

    Remove concatenation and pluralization issues from Delete Job,
    Launch Job (incl Launch Job, Launch New Cluster, Choose Plugin,
    Relaunch, Relaunch new cluster), Delete Job binary,
    Delete Data source, Delete Template, Delete Job execution,
    Delete Cluster, Unregister Image, and Delete Template actions.

    Change-Id: Idb615f4e2830ea5d9ddd0f871bfb3d90854f45f5
    partial-bug: 1307476

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/124129
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=41d16ce4caba47b6436847bc3578855924d6ed41
Submitter: Jenkins
Branch: master

commit 41d16ce4caba47b6436847bc3578855924d6ed41
Author: Doug Fish <email address hidden>
Date: Thu Sep 25 13:44:46 2014 -0500

    Fix concatenation in more misc actions

    Remove concatenation and pluralization issues from Evacuate
    Host, Sell Puppies (renamed from Delete Puppies. I cannot
    condone the deleting of virtual puppies.), Delete Continer,
    Delete Object, Delete Stack

    Change-Id: Ia6eb39cb2d736345afbdb3bfce8cb61bb2586871
    partial-bug: 1307476

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/124124
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=827e5a36fabee65bba739da66dacb7388d604f5d
Submitter: Jenkins
Branch: master

commit 827e5a36fabee65bba739da66dacb7388d604f5d
Author: Doug Fish <email address hidden>
Date: Thu Sep 25 13:08:01 2014 -0500

    Fix concatenation in Network and Router actions

    Remove concatenation and pluralization issues from Delete
    Network, Delete Subnet, Delete Router Rule, Clear Gateway,
    and Delete Network Profile

    Change-Id: I0898b446abd79672b43b55be62dba6250a4fdd58
    partial-bug: 1307476

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/123878
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=3e97f885f80b6847d910dab73c254519409cb7bc
Submitter: Jenkins
Branch: master

commit 3e97f885f80b6847d910dab73c254519409cb7bc
Author: Doug Fish <email address hidden>
Date: Wed Sep 24 17:38:34 2014 -0500

    Fix concatenation in identity related actions

    Fix concatenation and pluralization issues from
    Delete Groups, Remove Users, Add Users, Delete
    Projects, and Delete Domains actions.

    Change-Id: I7ba06a1acd720165af23a6fb475d061d2cab0e8a
    partial-bug: 1307476

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/124053
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=718b8eed0e8f8bd32d50bd34f89cc2b5afb1754f
Submitter: Jenkins
Branch: master

commit 718b8eed0e8f8bd32d50bd34f89cc2b5afb1754f
Author: Doug Fish <email address hidden>
Date: Thu Sep 25 08:43:47 2014 -0500

    Fix concatenation in Database actions

    Remove concatenation and pluralization issues from Terminate
    Instance, Restart Instance, Delete User, Delete Database,
    and Delete Backup actions.

    Change-Id: Ic8f343e7ad03497591e154d92e78d637618bf5e9
    partial-bug: 1307476

[Akihiro Motoki (amotoki)]

According to grep result, the followings are remaining.

$ grep -l "action_present = " openstack_dashboard/**/*.py

openstack_dashboard/dashboards/admin/instances/tables.py
openstack_dashboard/dashboards/project/loadbalancers/tables.py
openstack_dashboard/dashboards/project/volumes/volumes/tables.py
openstack_dashboard/dashboards/project/vpn/tables.py

Once they are updated, we can remove attribute "action_present/past".

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/126153

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/126153
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=969ff4828b40a9be2283a9f20ba95d0668cb7e4b
Submitter: Jenkins
Branch: master

commit 969ff4828b40a9be2283a9f20ba95d0668cb7e4b
Author: Akihiro Motoki <email address hidden>
Date: Sat Oct 4 21:11:58 2014 +0900

    Use action_present/past methods in BatchAction

    action_present/past methods were introduced in BatchAction to allow
    translators to control word orders and plural forms. This patch
    replaces the remaining old attribute version of action_present/past
    with the method version.

    Change-Id: I9f1cc8ff862471f33a5cb40f9e210c5222b7fb94
    Partial-Bug: #1307476