upstream-translation-update Jenkins job failing

[Andreas Jaeger]

Various upstream-translation-update succeed apparently but looking into the file errors get ignored - and there are errors uploading the files to the translation site like:

Error uploading file: There is a syntax error in your file.
Line 1936: duplicate message definition...
Line 7: ...this is the location of the first definition

For keystone, see:
http://logs.openstack.org/78/7882359da114079e8411bd3f97c5628f2cd1c098/post/keystone-upstream-translation-update/27cbb22/

This has been fixed on ironic with:
https://review.openstack.org/#/c/83935
See also: http://lists.openstack.org/pipermail/openstack-i18n/2014-March/000479.html

[Andreas Jaeger (jaegerandi)]

Same problem on cinder: http://logs.openstack.org/55/55272d4f1e2eec617d3305603d5dca5f50572e30/post/cinder-upstream-translation-update/655775a/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/83951

[Andreas Jaeger (jaegerandi)]

Patch for keystone: https://review.openstack.org/#/c/81925/

[Andreas Jaeger (jaegerandi)]

nova:

  http://logs.openstack.org/8d/8df565d39cd3216bde8488e656957a49dd6949fa/post/nova-upstream-translation-update/d8f54e6/console.html#_2014-03-28_17_45_32_985

ceilometer:
http://logs.openstack.org/dc/dc72b333973eb47fc96df578bd01f59241f6a624/post/ceilometer-upstream-translation-update/14c7de2/

[Andreas Jaeger (jaegerandi)]

cinder patch: https://review.openstack.org/#/c/83951/

[Andreas Jaeger (jaegerandi)]

ceilometer patch: https://review.openstack.org/#/c/83954/

[Andreas Jaeger (jaegerandi)]

Also a problem for heat:
http://logs.openstack.org/17/17d8cca36d36954391c19bc073f15e011e1a7983/post/heat-upstream-translation-update/73dfdd3/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/83960

[Andreas Jaeger (jaegerandi)]

Patch for nova: https://review.openstack.org/#/c/83961/

[Łukasz Jernaś (deejay1)]

Hmm, IMHO that looks like a bug in the transifex client. Do we know which version is on the doc builders?

[Andreas Jaeger (jaegerandi)]

AFAIK it's version 1.2.1 of the transifex client - we always use the latest release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/81925
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=e8d8306b21cb59d600c7500d6cdf2effecc3a679
Submitter: Jenkins
Branch: master

commit e8d8306b21cb59d600c7500d6cdf2effecc3a679
Author: Andreas Jaeger <email address hidden>
Date: Thu Mar 20 22:34:47 2014 +0100

    Fix Jenkins translation jobs

    The jobs keystone-propose-translation-update and
    keystone-upstream-translation-update do not update from
    transifex since our po files contain duplicate entries where obsolete
    entries duplicate normal entries.

    Remove all obsolete entries to fix the jobs.

    Closes-Bug: #1299349

    Change-Id: Ibbf26bb8158735a5dc57bff7d77a49e00239857c

[Andreas Jaeger (jaegerandi)]

As discussed on #openstack-infra and #openstack-cinder, I'm proposing patches for the pep8 queue that check the locale files with "msgfmt --check-format" so that no new errors can get introduced.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/84211

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/84226

[Andreas Jaeger (jaegerandi)]

Glance also affected: https://jenkins.openstack.org/view/All/job/glance-propose-translation-update/lastBuild/consoleText

[Andreas Jaeger (jaegerandi)]

Neutron also contains this:
https://jenkins.openstack.org/view/All/job/neutron-propose-translation-update/lastBuild/consoleText

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/84234

[Dolph Mathews (dolph)]

Should this be backported to keystone's milestone-proposed for icehouse?

[Andreas Jaeger (jaegerandi)]

Dolph, yes I suggest to backport this since I assume to compilation of message catalogs will fail otherwise. Will you so?

[Dolph Mathews (dolph)]

Andreas: Happy to, although I just tried and patchset 3 of https://review.openstack.org/#/c/81925/ conflicts with milestone-proposed... I assume you automated the process of removing the duplicates - can you share that? Thanks!

[Andreas Jaeger (jaegerandi)]

Dolph, I used the script from comment #2 in:
https://bugs.launchpad.net/ironic/+bug/1298645

Just run it over all po files.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/83960
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=8ec3bbb7cf0400b6b0ae82f88aad45e3fb366ab6
Submitter: Jenkins
Branch: master

commit 8ec3bbb7cf0400b6b0ae82f88aad45e3fb366ab6
Author: Andreas Jaeger <email address hidden>
Date: Sat Mar 29 09:38:30 2014 +0100

    Fix Jenkins translation jobs

    The jobs heat-propose-translation-update and
    heat-upstream-translation-update do not update from
    transifex since our po files contain duplicate entries where
    obsolete entries duplicate normal entries.

    Remove all obsolete entries to fix the jobs.

    Change-Id: I2071e3a792d107840449bbfe39e3a540ea1b626f
    Closes-Bug: #1299349

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: milestone-proposed
Review: https://review.openstack.org/84457

[Dolph Mathews (dolph)]

Thanks Andreas, I added a bit more automation to that script so you can just point it at the entire locale/ directory:

  python clean_po.py keystone/locale/

See: https://gist.github.com/dolph/9915293

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/83951
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=c72cc4b2345e9e271205d1de00c9812d0a2684ec
Submitter: Jenkins
Branch: master

commit c72cc4b2345e9e271205d1de00c9812d0a2684ec
Author: Andreas Jaeger <email address hidden>
Date: Sat Mar 29 06:38:52 2014 +0100

    Fix Jenkins translation jobs

    The jobs cinder-propose-translation-update and
    cinder-upstream-translation-update do not update from
    transifex since our po files contain duplicate entries where
    obsolete entries duplicate normal entries.

    Remove all obsolete entries to fix the jobs.

    Change-Id: I6d41dbdcc41646fcbd1ee84ce48cb0c461cd454c
    Closes-Bug: #1299349

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: milestone-proposed
Review: https://review.openstack.org/84563

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/84241
Committed: https://git.openstack.org/cgit/openstack/openstack-manuals/commit/?id=9e9dc87d3feef7ed07d8416004d4a12839c3dbc7
Submitter: Jenkins
Branch: master

commit 9e9dc87d3feef7ed07d8416004d4a12839c3dbc7
Author: Andreas Jaeger <email address hidden>
Date: Mon Mar 31 22:39:42 2014 +0200

    Check that all po/pot files are valid

    Perform minimal checks on po and pot files so that no broken files can
    get imported.

    Change-Id: Ic39d68106259a225fa5e5399123063e7c35ba150
    Co-Authored-By: Clark Boylan <email address hidden>
    Related-Bug: #1299349

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/84243
Committed: https://git.openstack.org/cgit/openstack/operations-guide/commit/?id=6c24c72de4f2cacf298985a59bbbb6f7e6adb629
Submitter: Jenkins
Branch: master

commit 6c24c72de4f2cacf298985a59bbbb6f7e6adb629
Author: Andreas Jaeger <email address hidden>
Date: Mon Mar 31 22:45:07 2014 +0200

    Check that all po/pot files are valid

    Perform minimal checks on po and pot files so that no broken files can
    get imported.

    Change-Id: I9b5edbad9f4302ca24e4255f503ee2db45ebd5d7
    Co-Authored-By: Clark Boylan <email address hidden>
    Related-Bug: #1299349

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/84563
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=c773cf96a52924a7dc42f774c57463e8ed66c9ef
Submitter: Jenkins
Branch: milestone-proposed

commit c773cf96a52924a7dc42f774c57463e8ed66c9ef
Author: Andreas Jaeger <email address hidden>
Date: Sat Mar 29 06:38:52 2014 +0100

    Fix Jenkins translation jobs

    The jobs cinder-propose-translation-update and
    cinder-upstream-translation-update do not update from
    transifex since our po files contain duplicate entries where
    obsolete entries duplicate normal entries.

    Remove all obsolete entries to fix the jobs.

    Change-Id: I6d41dbdcc41646fcbd1ee84ce48cb0c461cd454c
    Closes-Bug: #1299349
    (cherry picked from commit c72cc4b2345e9e271205d1de00c9812d0a2684ec)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/85123

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/84234
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=085a35d657cf0fa41a402f2af66c4beaa0f60db2
Submitter: Jenkins
Branch: master

commit 085a35d657cf0fa41a402f2af66c4beaa0f60db2
Author: Andreas Jaeger <email address hidden>
Date: Mon Mar 31 22:17:18 2014 +0200

    Fix Jenkins translation jobs

    The job neutron-propose-translation-update
    does not update from
    transifex since our po files contain duplicate entries where
    obsolete entries duplicate normal entries.

    Remove all obsolete entries to fix the job.

    Add test to pep8 that checks that no new breakages get
    introduced.

    Change-Id: I9c680eeefecfa124c5b2900b7fa7a0a8f7e8ebd6
    Closes-Bug: #1299349

[John Griffith (john-griffith)]

Just a note for other projects... Did a comparison of Dolph's script versus what I did in Cinder manually and we achieved the same results.

HIGHLY recommend Dolph's script here and just to highlight it a bit:
https://bugs.launchpad.net/keystone/+bug/1299349/comments/25

[Dolph Mathews (dolph)]

Advertised the fix for this (especially for milestone-proposed branches) on openstack-dev:

  http://lists.openstack.org/pipermail/openstack-dev/2014-April/031918.html

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/84233
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=977766080993c22ed279889c904bcc19883ff8b9
Submitter: Jenkins
Branch: master

commit 977766080993c22ed279889c904bcc19883ff8b9
Author: Andreas Jaeger <email address hidden>
Date: Mon Mar 31 22:13:41 2014 +0200

    Fix Jenkins translation jobs

    The job glance-propose-translation-update
    does not update from
    transifex since our po files contain duplicate entries where
    obsolete entries duplicate normal entries.

    Remove all obsolete entries to fix the job.

    Add test to pep8 that checks that no new breakages get
    introduced.

    Change-Id: I1bae1c1bf27bf664b24cf75fc96efb0fcfddc392
    Closes-Bug: #1299349

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: milestone-proposed
Review: https://review.openstack.org/85676

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: milestone-proposed
Review: https://review.openstack.org/85679

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/83954
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=faada69444988c66a09ade812e10fa2ab5a0adae
Submitter: Jenkins
Branch: master

commit faada69444988c66a09ade812e10fa2ab5a0adae
Author: Andreas Jaeger <email address hidden>
Date: Sat Mar 29 06:52:48 2014 +0100

    Fix Jenkins translation jobs

    The jobs ceilometer-propose-translation-update and
    ceilometer-upstream-translation-update do not update from
    transifex since our po files contain duplicate entries where
    obsolete entries duplicate normal entries.

    Remove all obsolete entries to fix the jobs.

    Add test to pep8 that checks that no new breakages get
    introduced.

    Change-Id: I3d7d04a7d5d073d5123f1f7b1e16fd3ef167acfb
    Closes-Bug: #1299349

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/84207
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=8f5018eec7726112e378f090e119d9d3cbfb598e
Submitter: Jenkins
Branch: master

commit 8f5018eec7726112e378f090e119d9d3cbfb598e
Author: Andreas Jaeger <email address hidden>
Date: Mon Mar 31 20:37:27 2014 +0200

    Check that all po/pot files are valid

    Perform minimal checks on po and pot files so that no broken files can
    get imported.

    Change-Id: I5dd0ca872fa25f8ef9557f039904090c4a672ff3
    Co-Authored-By: Clark Boylan <email address hidden>
    Related-Bug: #1299349

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/83961
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=79fe7a509cd3bebf513fd24939954df53e7d8a52
Submitter: Jenkins
Branch: master

commit 79fe7a509cd3bebf513fd24939954df53e7d8a52
Author: Andreas Jaeger <email address hidden>
Date: Sat Mar 29 09:42:52 2014 +0100

    Fix Jenkins translation jobs

    The jobs nova-propose-translation-update and
    nova-upstream-translation-update do not update from
    transifex since our po files contain duplicate entries where
    obsolete entries duplicate normal entries.

    Remove all obsolete entries to fix the jobs.

    Change-Id: I7f51922472f12f3a13feb71a6fb344076781d7e9
    Closes-Bug: #1299349

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: milestone-proposed
Review: https://review.openstack.org/85897

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: milestone-proposed
Review: https://review.openstack.org/85955

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: milestone-proposed
Review: https://review.openstack.org/85958

[Openstack Gerrit (openstack-gerrit)]

Reviewed: https://review.openstack.org/84226
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=41870b6e553572ff29965efc417509468995090d
Submitter: Jenkins
Branch: master

commit 41870b6e553572ff29965efc417509468995090d
Author: Andreas Jaeger <email address hidden>
Date: Mon Mar 31 21:39:28 2014 +0200

    Check that all po/pot files are valid

    Perform minimal checks on po and pot files so that no broken files can
    get imported.

    Change-Id: Ib438faddb362eb8e61eaa64a870160196f16c594
    Related-Bug: #1299349
    Co-Authored-By: Clark Boylan <email address hidden>

[Openstack Gerrit (openstack-gerrit)]

Reviewed: https://review.openstack.org/84211
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=48394142371702715e7204f4ffb7fd32a2b50a6b
Submitter: Jenkins
Branch: master

commit 48394142371702715e7204f4ffb7fd32a2b50a6b
Author: Andreas Jaeger <email address hidden>
Date: Mon Mar 31 20:54:06 2014 +0200

    Check that all po/pot files are valid

    Perform minimal checks on po and pot files so that no broken files can
    get imported.

    Add msgfmt to list of requirements in doc/source/setup.rst.

    Change-Id: Ifa9fe2fc1ca9a41621ff3039c249700cdb1cd0ac
    Related-Bug: #1299349
    Co-Authored-By: Clark Boylan <email address hidden>