Automatically add translator credits to authors list

[GunChleoc]

Translator credits are now manages in the new gettext template "translator_credits".

Parse this file to add them to the authors list, both in Widelands and on the Homepage.

[kaputtnik (franku)]

Am i right that the new "authors.lua" file has to be parsed for the website?

Should the new authors list on the homepage contain links to the profile page of the homepage if the author/developer/translator do have an account on widelands.org? This would imho solve an issue, if someone want to inform some the developers of some things. See https://bugs.launchpad.net/widelands-website/+bug/1421807

[GunChleoc (gunchleoc)]

Yes, it will need to be parsed for the homepage as well.

I think the best solution for the other bug would be a members list function, available only to logged-in users.

[kaputtnik (franku)]

I am trying to parse the AUTHORS.lua file, but it is very difficult (for me).

What about the translators list? There is currently only a statement for this:

> h1_authors(_"Translators") ..
> "%s" .. -- Place holder to insert translators list, which is parsed in C++.

So where are the translator names stored?

For now i am for a few days in holiday :-)

[SirVer (sirver)]

It sounds to me like we should have a small c++ tool that writes out a developer file (probably directly an html file) that can easily be processed by the website and that we can update nightly.

[GunChleoc (gunchleoc)]

We could running add such a tool to the merge and push translations script - I am trying to run this about once a week, so the information will be current enough.

[SirVer (sirver)]

That seems even better. It can also write out a lua script that can be included in the developers file an that sets a variable.

> Am 23.03.2015 um 11:28 schrieb GunChleoc <email address hidden>:
>
> We could running add such a tool to the merge and push translations
> script - I am trying to run this about once a week, so the information
> will be current enough.
>
> --
> You received this bug notification because you are subscribed to
> Widelands Website.
> https://bugs.launchpad.net/bugs/1424072
>
> Title:
> Automatically add translator credits to authors list
>
> Status in Widelands:
> New
> Status in The Widelands Website:
> New
>
> Bug description:
> Translator credits are now manages in the new gettext template
> "translator_credits".
>
> Parse this file to add them to the authors list, both in Widelands and
> on the Homepage.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/widelands/+bug/1424072/+subscriptions

[kaputtnik (franku)]

Maybe i do not fully understand what you are saying, but wouldn't it be easier to have only some txt files which contains the develepoers lists? Those txt files could then easily translated into html/lua/something else. In my mind future changes could then also be better handled. F.e. switch to html5 or python3. Just a thought :-)

[GunChleoc (gunchleoc)]

Would a conf file be easier to parse for you than Lua tables? i.e. What format would you need?

[kaputtnik (franku)]

The main problem (for me) in parsing the AUTHORS.lua file is to get the subheaders with the corresponding developer names, but i am not really familiar with regular expressions so maybe the problem is about me and not about the file :-D
Another thing what comes in my mind are possible little changes to the authors.lua file wich could break the regexps.

A conf file was in my mind too, but i don't know much of those formats. I am looking at https://docs.python.org/2/library/configparser.html but i am unsure if this is a good way to go.

A suggestion for a conf file:

> [Headername]
> position: h1
> devnames: Comma, Separated, List, Of, Names, in Headername
>
> [Headername2]
> position: h2
> devnames: #no names here
>
> [Headername3]
> position: h3
> devnames: Comma, Separated, List, Of, Names, in Headername3

A short test of such a file works as aspected and could easily parsed. The Sections ([Headername]) should be in the same order as they should appear on the Website. The 'position' value is the header level. The 'devnames' value could be as long as it is needed. It is also possible to make this value mutilined if you put a tab on the newline:

> devnames: GunChleoc, SirVer (Holger Rapp),
>[tab] Franz Marque, Further, Names

[SirVer (sirver)]

Not a fan of the config files - the config file format is deprecated and we should not use it for new stuff. Lua is where we are moving because it is strictly superior in expressive qualities.

> Another thing what comes in my mind are possible little changes to the authors.lua file wich could break the regexps.

This is a big disadvantage of a common base data structure - if we change it in trunk, the derived code (i.e. the website) might break and we will not notice. Therefore I do not like having a flat file that can be parsed in Python and the engine, see below.

> Maybe i do not fully understand what you are saying, but wouldn't it be easier to have only some txt files which contains the develepoers lists? Those txt files could then easily translated into html/lua/something else. In my mind future changes could then also be better handled. F.e. switch to html5 or python3. Just a thought :-)

Sorry, my last comment was less then clear. No, while maybe (slightly) easier now, I believe it is more work in the long run and brittle. We already thought once that it would be convenient to just reparse the shipped data in another language. This was for the encyclopedia on the homepage. We parsed the tribes/**/conf files in python, but the python parser was not updated to understand ships and everything broke.

My opinion is that the Widelands engine is the thing that should be able to read Widelands data. And the engine should consist of various libraries that can be easily reused (in small binaries) to parse any kind of data the game consumes, be it AUTHORS.lua, tribes, worlds, maps, replays or whatever.

This thought drives my design: make AUTHORS.lua easy to consume for the engine, so that it can be adapted to cater to the engine's needs in future changes. For example like now, where we want to include translators in another way than all the other contributors, we have to change the file format for that. Then use this easy consumption to transform the data if it is needed in another format with a tiny binary.

In this case, we could have a small binary that writes out the data as a HTML page or a JSON blob that is easily parsed and used in python for the website.

[kaputtnik (franku)]

i agree.

Couldn't explain exactly, but JSON is easier to read and more app independent than html, so in my opinion JSON should be used.

[GunChleoc (gunchleoc)]

Transifex supports all kinds of file formats. So, how about storing each language's translators in a simple, separate text file? I know we want to get away from the config file format, but I guess such files would be easy to parse? Maybe we could parse them in Lua directly - if each locale has its own file. it will contain a string where we will only need to take care of line breaks.

I could also upload lua files to Transifex, but they would be treated as simple text files, so the coding part if it would appear.

[kaputtnik (franku)]

We need all of the developers, not only the translators. Following your suggestion would result in a bunch of text files, where each file contain the name(s) of one header in https://wl.widelands.org/developers/ . So each file represents a header (as i understand your suggestion). Don't know if it is a problem having so many text files. But it will result in many downloads/reads. How to determine what file represents a sub[sub]header? And the file naming schema? Hmmm...

[GunChleoc (gunchleoc)]

My suggestion was for the translators only, because the names will be difficult to manage otherwise.

The rest would still be in Lua tables. Or are Lua tables a problem in general for the website?

[kaputtnik (franku)]

> My suggestion was for the translators only, because the names will be difficult to manage otherwise.
Ah, ok.

As described in #9 parsing the the Lua tables is not easy with regular expressions. There some Python modules which could handle Lua tables (LUPA f.e.), but i don't want to blow up the website with more modules. And i don't know how changes to the Lua tables could be handled easily with such a module.

A widelands export function(s) to a (f.e.) json file is surely the best option, because it is independent to the code language. If something changes in the widelands code (f.e. adding ships) the export function(s) must handle this changes and writes the changes to the file. The code on the website is just a read from this file and could make the website content depending on the file. With a standard file format it is much easier than with Lua table i think. Correct me if i am wrong :-)

[GunChleoc (gunchleoc)]

I have updated the branch with a concept for JSON files. It's just the translators for now, but we can add a JSON file for the rest of the developers. What do you think, does this work for you?

http://bazaar.launchpad.net/~widelands-dev/widelands/authors/revision/7421

[kaputtnik (franku)]

I've done parsing the file developers.json for the website and its very easy. Just a few lines of code.

You've changed some headers, but i think it's ok.

Could you please explain how the translaters are handled? Are they added to the developers.json?

[GunChleoc (gunchleoc)]

The translators will have a separate file for each locale. I have added 2 example locales to txts/translators. You can have a look at utils/update_authors.py to see how I'm parsing them. The point to watch out for is if

translators["locale-translators"]["translator-list"] == 'translator-credits'

we don't want it, because there are no translator credits for the locale available.

Also, the directory contains the generated Lua file which you need to skip, so do check for the file extension.

[kaputtnik (franku)]

Ok, i am working on it, but have another question(s):

1. There are different formats, how the names are stored in the json files. In developers.json they are a comma separated list, in the translator files they are all together in one string, separated with "\n". Is this difference needed? It's no problem to get the names out of one string, so it's just a question of interest. On the other side: wouldn't it be better to have one format for all files?

2. Ordering the Translators list into the developers list: Actually they are shown between "Campaign Story" and "Packagers".
Because i need a simple txt object with all entries (header, subheader, names) in the right order for parsing into wl_markdown, i currently could not obtain a "position" where the translators has to be inserted. So i edited the developers.json and put in there a "marker" like:

  {
   "heading": "Translators",
   "entries":[]
  },

So the heading name could be retrieved from the json file and i could check against the heading where to put the translators. The edited file is attached. Is this ok?

If i am ready i push a branch for the website.

[kaputtnik (franku)]

Added a branch for the website.

[GunChleoc (gunchleoc)]

1. The format is different for the translators, because that is what we will get from Transifex. The other developers will be added to the JSON file by hand, and having them as separate entries is more human-readable than a long string with \n's in it IMHO.

2. Yes, this is OK - I can adjust my code accordingly.

[kaputtnik (franku)]

Ok, than all is fine and good :-)

Somewhere we should explain the keywords, or write a few sentences where the keywords are used to give a warning if someone want to change them in future... What do you think? Maybe a simple text file in the folders will do that thing?

[GunChleoc (gunchleoc)]

Good idea. I couldn't find any documentation on how to add comments to JSON files. The JSON files would be the best place to put the documentation. Did I overlook something?

I changed the language names to use their native form, could you please check if "Arabic" is displayed correctly? The forum can handle it, but I'd rather be sure. It should look like this: العربية

[kaputtnik (franku)]

Comments are not allowed in json files, unfortunally. See http://stackoverflow.com/questions/244777/can-i-comment-a-json-file Don't know how to manage informations about the keys.

The native arabic is shown correct :-) See attached file. Don't worry about the double entries, i've just added your changes as new json files to the old ones, so they appears twice.

[GunChleoc (gunchleoc)]

Excellent :)

I will create an extra readme file then.

[kaputtnik (franku)]

Just thinking: Maybe the native arabic has to be checked with a browser in windows? Windows has sometimes bad behavior...

But i think the worst thing is that a windows browser could show rectangles or other signs instead of the arabic signs. I didn't set up the website on a windows machine (even actually don't know who), so i couldn't check this.

[GunChleoc (gunchleoc)]

This actually depends more on the website code and the browser than the operating system.

Modern Windows versions can handle this as such, no problem - they have quite extensive unicode fontsets.

I think what would worry me more is smartphones and tablets, now that you mention it - they might not have enough fontsets installed in order to save some space.

[kaputtnik (franku)]

I am thinking about how much time it takes to open many files. Here on my local machine it will work fast, but i don't know where the developer and translator files are hosted, so opening theses files could take maybe a long time if they are stored somewhere else.

I put these files on my cloud storage and tried to fetch them with my local installation of the website to get a feeling what will happen then. But i get only the error "File not found". I think its related because i try to use a "http:" request instead of a local path. Don't know how to manage this :-(

I updated my branch and added some exception handling.

[GunChleoc (gunchleoc)]

I don't think people look at the developers list that often, so I guess opening many files shouldn't be a problem performance wise.

It is definitely not a problem on the Widelands side, because the JSON files will only be parsed when updating the translations or developers list in trunk.

[GunChleoc (gunchleoc)]

The merge request has now been greenlit. @kaputtnick, can I merge this into trunk without breaking the website now, or do I have to wait until somebody can update the webpage?

[kaputtnik (franku)]

Because the file name changed from txts/developers to txts/developers.json, the website would break and an IOError occurs if you want to see the developers-page on the website.

If you really want to merge there should be stored an additional file txts/developers with old content and if the webpage related branch is merged this file could be removed. Don't know if this a good way to go...

[GunChleoc (gunchleoc)]

It's better to wait until someone can deploy an update to the website then.

[SirVer (sirver)]

Support for this has been merged and deployed on the website in r391 and into widelands in r7482.

Thanks for your concerted hard work on this and sorry for taking so long!

[kaputtnik (franku)]

> Thanks for your concerted hard work on this and sorry for taking so long!

No problem... RL is much importand :-)

There is one language which isn't displayed correct on the homepage (firefox on linux). See attached image. All other languagenames are ok. Maybe it's a misconfigured language name in the source file?

[Venatrix (elisabeth)]

For me it is shown correctly. At least I suppose it, as I don’t know those symbols or which language that is. I guess, you’re just missing a corresponding font?

[wl-zocker (wl-zocker)]

I see the same symbols as kaputtnik.

The language is Burmese [1], and the symbol Venatrix sees seem to be correct.

[1] https://en.wikipedia.org/wiki/Burmese_language

[kaputtnik (franku)]

Maybe i miss a corresponding font... how could i find such? And what about tablets and smartphones as GunChleoc thinking of in #27?

In my opinion it is a nice idea to use the native language names :-) , but

1. for a translator who has to know english it should be no problem to read his native language name in english
2. for interested users who doesn't know any of the other languages it would be better to show the names in english to give them the ability to understand what language it is (without using a translator or searching for a specific font)

Thanks wl-zocker for figuring this out.

[Venatrix (elisabeth)]

Thanks, wl-zocker. I even had a look in the source code of the page to see whether the name of the language is shown there… (No, it isn’t, only ID numbers for languages using not-ASCII symbols.)

I think it’s cool to show the native names, too, but I would like to see the english name, too (for example in paranthesis). If I see such a list, I want to know, which languages that are.

And for the corresponding font: I’m not sure. I have quite some font packages installed, so I can only guess which of them provides the burmese letters. Maybe Unifont? I don’t know exactly. I can only say that it depends on the fonts your system is able to show.

[kaputtnik (franku)]

> And for the corresponding font: I’m not sure. I have quite some font packages installed, so I can only guess which of them provides the burmese letters. Maybe Unifont? I don’t know exactly.

My question about which font i should search for was a bit kidding, sorry. It should only show that there is no possibility for a normal user to find such a font (beside he would also not know that a font is missing).

[GunChleoc (gunchleoc)]

Yes, it's definitely a font issue. Burmese is tricky, the font we use in Widelands is called Myanmar.

I wouldn't be opposed to adding the English names to the end - e.g.:

العربية (Arabic)

I am thinking we should shift everything that's in i18n-lua into the new system as well, so we have everything for each language in one spot - Native name, English name, sort order name (e.g. Chinese gets sorted as "Fanti Zhongwhen", fontset, RTL/LTR.

In the long run, I would like to get rid of the fontset definitions there and use Unicode code points instead, so we can have mixed fonts - e.g. the best fontset for Arabic and that our localizer recommended to me we could use doesn't have Latin characters. Also, the native language names are now screwed up in the developers' list in Widelands itself - I did this on purpose as an incentive to fix stuff.

How about we tackle this again once we have the new font renderer merged?

[kaputtnik (franku)]

I think we have two use cases here:

1. Showing the translators in the game: In the game it would be ok to show the native language names, because normally it runs with localized texts.
2. Showing the translators on the Website: On the website, where all is english, the language names should also be in english (latin characters)

[GunChleoc (gunchleoc)]

I have created a new branch with the proposed changes.

https://code.launchpad.net/~widelands-dev/widelands/authors_and_languages

Note that I shifted the JSON files to i18n/locales and renamed the master file to locales_translators.json.

The new key for you to parse on the website is "your-language-name-in-english"

[kaputtnik (franku)]

Thanks. I am ready with the changes to the website code but have a question related to the json files. They all have the same entry:

> "locale-translators": { ... }

Is this needed by your code? It would make the code much convenient if we could omit this entry.

[GunChleoc (gunchleoc)]

I thought that JSON needs to name the main array

[kaputtnik (franku)]

As i understand all between two curly brackets is an "object" in json (object = associative array). Because each file begins with "{" and ends with "}" each file represents an object with key/value pairs. See the examples in wikipedia https://en.wikipedia.org/wiki/JSON

In my opinion there is no need to put our needed values in an extra object ("locale-translators": { ... }) because each file represents an object itself.

Wouldn't it be more convenient to the lua code if we omit the "locale-translotors" object? If not we could leave the files as they are. My thought was just to make the code easier (and put in there some checks on the keys).

[kaputtnik (franku)]

Attached file contains all locale files without "locale-translators" object. Just if you want to test it :-)

[GunChleoc (gunchleoc)]

Done - works like a charm :)

[kaputtnik (franku)]

Branch added.

As described in the merge proposal i've made some test's to the keywords of the translator files. The keys from developers.json are not tested.

[GunChleoc (gunchleoc)]

Fixed in build19-rc1.