Add tribes directory to developer scripting reference

[GunChleoc]

There are RST comments in the tribes/ subdirectories that aren't included in the Widelands Scripting reference

https://wl.widelands.org/docs/wl/reference/

[kaputtnik (franku)]

As i understand this is part of widelands source code, not directly related to the website. But because there is a server error related to the documentation, i will look into it. This sphinx stuff is quite complicated ...

[GunChleoc (gunchleoc)]

Yep, coding will probably need to be done in:

https://bazaar.launchpad.net/~widelands-dev/widelands/trunk/files/head:/doc/sphinx

and the results tested with the website...

[kaputtnik (franku)]

The culprit is the lambda in extract_rst.py:

http://bazaar.launchpad.net/~widelands-dev/widelands/trunk/view/head:/doc/sphinx/extract_rst.py#L87

which scans only the directory data/scripting and not data/tribes. I have an idea what lambdas are good for, but i don't know exactly how they work.

IMHO it would be better to define the directories where to look for RST comments and scan the files in the defined directory... hm...

Defining the directories is done, but i need to adjust the output regarding to the replace_auxilary_toc() function. Currently only the last created files are shown :-D

[kaputtnik (franku)]

Where has the additional documentation should show up? In "Core functionality" or in "Auxiliary scripts"? Or somewhere else?

[kaputtnik (franku)]

Thats how it looks like now.

[GunChleoc (gunchleoc)]

Semantically, it would belong in Core functionality -> wl.map, except for the help scripts, which are for the in-game encyclopedia.

Alternatively, we could also add two new main hooks "Tribes" and "World" - then we can have world documentation in Lua too.

I need to add some RST so you will have more test data.

[GunChleoc (gunchleoc)]

Can you link up your branch, or do you want me to create a new one?

[kaputtnik (franku)]

See https://code.launchpad.net/~widelands-dev/widelands/sphinx_changes

This branch contains changes to extract_rst.py to scan also the folder data/tribes/scripting/help and some modifications to lua files which causes warnings by sphinx-build.

Having a specific ordering is not possible with the current script, because sphinx does not know where to put specific help texts in the lists. The script just collect the rst comments, put them into 'auxiliary.rst.in' and then save it as 'auxiliary.rst'. All the entries in the "Auxiliary scripts" list are disordered, except the ordering of folders where the files (containing rst comments) are found. So all rst comments in files under data/scripting appears before the things in data/tribes/scripting/help.

I want also to change the coloring of the created formulas in geometry. So i want to play with this branch a bit further.

Are the last two entries (modindex and search) in index.rst needed? Do they work with a locally created documentation? F.e. in a html formatted documentation?

[kaputtnik (franku)]

Colored math symbols...

Top: Current (black)
Middle: "Green"
Bottom: "Cerulian"

I have also adjusted "vertical-align" of the formula images.

Reference:
http://www.nongnu.org/dvipng/dvipng_4.html#Command_002dline-options
http://mirrors.ctan.org/macros/latex/contrib/xcolor/xcolor.pdf (One of "dvispnames" or "svgnames")

Is someone building the documentation locally at home? In this case the coloring should fit with the background used locally.

The used images have to be adjusted also.

[kaputtnik (franku)]

I have updated the sphinx_changes branch with support for the coloring and removed the print statements in extract_rst.py

[GunChleoc (gunchleoc)]

Good idea about the coloring :)

[kaputtnik (franku)]

I will try to rework extract_rst.py to make it a bit better (from coding point of view).

And we need a decision for the formula coloring. Available colors could be found f.e. here: http://www.latextemplates.com/svgnames-colors (i did not figure out how to use RGB values for coloring)

I am in favor to have a different color than green, to have a distinction from green links. If no one complains i use the "Cerulian" color (bottom example in the provided image in #9 ).

[kaputtnik (franku)]

Rework of extract_rst.oy is done. If you could live with the actual state (see image in #5), i could propose for merging...

If we/you knew exactly which file contains a specific documentation and where to order it into the list, i believe we could do that...

[GunChleoc (gunchleoc)]

I like the Cerulian color.

#5 is missing a g in global_helptexts. I still need to do some of the documentation that I want to have show up.

[GunChleoc (gunchleoc)]

I have added some RST to the tribes so you get a general idea of what I'm aiming at. I will want more RST for all building types, for wares, workers, immovables, ships... one entry for each type. And then likewise for world objects.

[kaputtnik (franku)]

Slow down :-)

Thanks for the additional RSTs... i have to find out how to get them into the toctree... at the right places.

- The missing 'g' is fixed

[GunChleoc (gunchleoc)]

Yes, over to you now :)

[kaputtnik (franku)]

This is a big task... i am not sure to get a solution for build 19 right now ...

Just about the RST comments: This is RestructeredText markup and some things related to formatting are different... f.e. bold text are marked with **This is bold text**, not the html tags <b>...</b>.

See http://www.sphinx-doc.org/en/stable/rest.html#rst-primer

I found some missing/wrong function directives. A function is described as f.e.:

-- .. function:: new_tribe(table)

In some files the double colons are missing, or the function directive is missing at all. The function description will never show up if the colons are missing.

I will fix the things i find in the current branch, but ask you to pay attention on this when writing new RST comments :-)

If you, or someone else wants to set up a test environment here is what i did:
1. Set up widelands website as described in the README for the website
2. Got to the folder wlwebsite and run:
   bzr branch lp:~widelands-dev/widelands/sphinx_changes wl_source
3. cd to the folder ~/wlwebsite/wl_source/doc/sphinx (activate the virtual environment)
4. run: ./extract_rst.py
5. run: sphinx-build -b json -d build/doctree source build/json
6. New sub directories have been created ...
7. Start the local website and go to http://localhost:8000/admin/sphinxdoc/app/
8. Change the Path to the absolute path where the json directory was made in 5. This should be something like:
   /home/USERNAME/wlwebsite/wl_source/doc/sphinx/build/json/
9. Open localhost:8000/docs/wl/reference/ in your browser

Every time the source is changed (by running extract_rst.py) it is better to remove the 'build' directory (created in step 5) before running the command in 5. This will ensure that all files are created from scratch and there is no remaining file or old data in browsers cache.

[GunChleoc (gunchleoc)]

Thanks for the link and the instructions - I hope that I have fixed all the issues in the new files at least.

This doesn't need to be ready for Build 19 - this is new documentation that we haven't had before, so any time that it's done is fine.

[kaputtnik (franku)]

Thanks for fixing :-)

Looks like running clang-format brakes some indentations in the RST comments in cc-files. I get warnings like:

autogen_globals.rst:61: WARNING: Explicit markup ends without a blank line; unexpected unindent.

This is caused because clang-format creates linebreaks for too long lines i think. See the line break around lines 198..200 in http://bazaar.launchpad.net/~widelands-dev/widelands/trunk/revision/8048#src/scripting/lua_globals.cc

By searching around i found the clang-format config option which should prevent such auto editing for the RST comments:

CommentPragmas (std::string)
    A regular expression that describes comments with special meaning, which should not be split into lines or otherwise changed.

I am not much familiar with regular expressions ... something like this(?):

CommentPragmas ('^ RST.*')

Reference:
http://clang.llvm.org/docs/ClangFormatStyleOptions.html (scroll down to find the option)

Maybe we could add a modified clang-config file or a script which invokes clang-format with the option CommentPragmas to the utils directory?

I will fix this in this branch...

[GunChleoc (gunchleoc)]

Modifying the .clang-format would be best. Maybe we should do this in a separate branch together with the necessary fixes to the RST? Otherwise, the next time we update the Widelands version & documentation on the website, it will break things.

[kaputtnik (franku)]

The documentation on the website get automatically updated every hour. So there the current result could be seen yet, f.e. https://wl.widelands.org/docs/wl/autogen_globals/#pgettext or https://wl.widelands.org/docs/wl/autogen_wl_map/#productionsitedescription

Yes, making a separate branch would be better. Do you want to work on this? The files were sphinx build is complaining about are :

src/scripting/lua_maps.cc
src/scripting/lua_globals.cc

If you want me to work on this i would just replace the new RST comments with the old ones. I would leave the lines longer than 100 chars to have a test scenario for clang-format. You could then test and provide the config file for clang with the new created branch.

You or i?

[GunChleoc (gunchleoc)]

Long lines as a test case sounds good to me.

Could you please take care of it? I have been in serve moving hell all week for another site and my brain is dead.

[kaputtnik (franku)]

I've made a branch: https://code.launchpad.net/~widelands-dev/widelands/fix_RST_comments/+merge/302205

Don't know if i find time to create a config file, but this could also be done the next time one is running clang-format.SirVer told me that the wl/docs on the website aren't used much...

I will also create a bugreport for rereading and maybe adjusting the RST comments. This is a task for a rainy sunday :-)

Ah just seen that the branch willl get merged, thanks :-)

[GunChleoc (gunchleoc)]

> the wl/docs on the website aren't used much...

Define much... I use them ;)

[kaputtnik (franku)]

Current state...

You can define directorys to search for lua files (containing RST comments) and define a TOC where to put the found files (conatining RST comments) in. Defining is made in extract.py:

# (src_dir, toc_to_place_found_RSTs_in)
lua_dirs = (
    ('data/scripting', 'auxiliary'),
    ('data/scripting/editor', 'lua_world'),
    ('data/scripting/win_conditions', 'lua_world'),
    ('data/tribes', 'lua_tribes'),
    ('data/tribes/scripting/help', 'lua_tribes'),
    ('data/tribes/buildings/militarysites/atlanteans/castle', 'lua_tribes'),
)

The tocs defined here (second argument in each line) have to be present as a file. Currently there are three predefined TOC files available:

auxiliary.rst.org (renamed from auxiliary.rst.in)
lua_world.rst.org
lua_tribes.rst.org

I have to work more on extract.py to give the variables a better name.

Is this something you can work with? If we could say 'each RST in files living under doc/tribes and the subdirs are related to lua_tribes' we could maybe simplify defining lua_dirs to search for 'main folder including subfolders' so one doesn't have to define each directory in lua_dirs.

There is one remaining warning about a double defined function (text_line()):

/home/kaputtnik/Quellcode/widelands-repo/sphinx_changes/doc/sphinx/source/autogen_lua_world_format_editor.rst:14: WARNING: duplicate object description of text_line, other instance in /home/kaputtnik/Quellcode/widelands-repo/sphinx_changes/doc/sphinx/source/autogen_lua_tribes_building_help.rst, use :noindex: for one of them

[GunChleoc (gunchleoc)]

This is exactly what I was looking for, thanks!

Leave the error message in for now, I should have a look if that function is identical for both, then we can pull it out. If not, we should rename one of them.

[kaputtnik (franku)]

I've done code cleanuo and merged with trunk.

You may want to edit the descriptions and headers in:

lua_world.rst.org
lua_tribes.rst.org

I had no idea what to write there (beside my english knowledge) :-D

The branch contains also some more RST related fixes to lua files. So i am in favor to get this in the main trunk as soon the descriptions and headers are updated.

[GunChleoc (gunchleoc)]

I have added some info for world and tribes and hopefully fixed the warning in #26. Not tested though - I had to reinstall my system and haven't had time yet to set up the website project.

[kaputtnik (franku)]

Looks good to me :-)

No warnings.

My previous explanation of testing was a bit complicated. Of course you could also use the folder of the corresponding branch (sphinx_changes). Just apply the full path in the admin page for sphinxdoc. Before running the command in the folder of the branch (../sphinx_changes/doc/sphinx)

sphinx-build -b json -d build/doctree source build/json

you should activate the virtual environment, otherwise you have to provide the full path to the command (f.e. ~/wlwebsite/bin/sphinx-build-b json -d build/doctree source build/json)

[GunChleoc (gunchleoc)]

Is there a way to run the website without reCaptcha? I'd rather be able to test without having go get a Google account just so I can run it on localhost.

[kaputtnik (franku)]

reCaptcha isn't needed for running the website on localhost. reCaptcha-keys are only needed if you want to test registering.

What is needed are defined constants like they are in local_settings.py.sample:

NORECAPTCHA_SITE_KEY = 'dummy'
NORECAPTCHA_SECRET_KEY = 'dummy'

The values doesn't matter, until you do not try to register over localhost.

If this doesn't solve your problem, please give more information.

I just found that you have to use 'wl' for the slug field in http://localhost:8000/admin/sphinxdoc/app/1/

[GunChleoc (gunchleoc)]

Never mind, I created the local settings files in the wrong copy of the code.

[GunChleoc (gunchleoc)]

I managed to build the sphinx doc now, but the website doesn't find it:

Page not found (404)
Request Method: GET
Request URL: http://localhost:8000/docs/wl/
Raised by: sphinxdoc.views.documentation

I managed to create HTML files in widelands (rather than widelands-website) though, so I can use that for testing without running the webserver.

[kaputtnik (franku)]

Maybe you have overlooked this:

> I just found that you have to use 'wl' for the slug field in http://localhost:8000/admin/sphinxdoc/app/1/

[kaputtnik (franku)]

Screenshot of my settings. The string in 'Path' on my system is:

/home/kaputtnik/Quellcode/widelands-repo/sphinx_changes/doc/sphinx/build/json/

[GunChleoc (gunchleoc)]

That worked, thanks!

[GunChleoc (gunchleoc)]

Alternatively, we could also have a colored background for the parameters like standard sphinx output will give you.

I'd be fine with either, but less tabbing will also make for shorter pages.

[kaputtnik (franku)]

Damn, i could spend hours to get into this...

And in the end i think i haven't enough knowledge of this lua stuff to find a good solution. Attached is something i end up for now. I believe if we want people contributing we should explain a bit more in the documentation.

And i think we should have a consensus for a good markup before more work is done on the documentation.

The screenshot shows using a definition list instead of listing the table entries with strong emphasis and shows some little code snippets. I have done this only for the first two table entries (and i am not sure doing the code snippets right)

Are all table entries for "new_tribe" needed? E.g. "ship_names"? What is "ship: The internal name of the tribe’s ship."? A tribe could have a lot of ships, why is there a single "internal name" and what is the difference to "ship_names"?

Lost in questions :-D

[GunChleoc (gunchleoc)]

I think your markup idea looks good - can you stick that in a branch? I can take it from there.

Once I have done the markup, you could tell me which bits you don't understand so that I can rephrase - I am too close to this.

[kaputtnik (franku)]

I just added this to media/css/documentation.css of the website:

th {
 background-color: #595959;
}

And used definition lists explained here: http://www.sphinx-doc.org/en/stable/rest.html#lists-and-quote-like-blocks (third example).

We should also rethink the css of <code> <pre> tags in the code. The Black background color isn't good to see.

[kaputtnik (franku)]

I played a bit around with the css and linked the branch https://code.launchpad.net/~widelands-dev/widelands-website/dev_doc_css

It's not the best, but a beginning.

Feel free to change it to your needs. If we want more changes, i am pleased to work on this further.

[GunChleoc (gunchleoc)]

I have played with your css a bit, what do you think?

I have also added a new animation file and updated the militarysite documentation in the attached sphinx-formatting branch. Can you please take a look before I do the rest of them?

[kaputtnik (franku)]

Regarding CSS: I think using the same text color for Functions and Links is confusing. Maybe just increase the font size of functions to 1.5em?
Not related to your changes: There is a double background for pre-tags and tags with class highlite. The 'highlite' background uses the whole width, the pre tag only the width of text (see screenshot). The pre tag is defined in base.css and we may want to overwrite it in documentation.css like:

pre {
 display: block; /*show black background over the whole width*/
}

Alternative is to display only the black background of pre without additional borders: Disable 'border' and 'background-color' for .highlite in documentation.css.

The militarysite documentation is really good now :-)

Animation:
lgtm :-) Since there are 'Mandatory' and 'Optional' statements, i think having them at most places would lead into less confusion. E.g. in <tribename>.lua something like "All table entries are mandatory."

[GunChleoc (gunchleoc)]

Fixed in build20-rc1