Upgrade to django 1.8

[kaputtnik]

Just an information on the upgrade to django 1.8

The branch is imho ready to test. What is not tested:
- sphinx search
- sphinxdoc
- wlwebchat

Setting up a shpinx environment seems a bit complicated to me. So i hope doing the tests on the server may work...

A short description of the changes:
- included problematic third party apps to the project itself, namely: django-ratings (not maintained anymore), notification (absolutely incompatible with the old db-tables), threadedcomments, tracking and sphinxdoc (not longer maintained)
- Adjusted all apps (including formerly third party) to django 1.8
- Use a more up date package for pagination
- Registration uses now recaptcha v2
- Reworked the news section a bit
- included "NOCOMMS franku:" to some things which i noticed
- some old comments i leaved (mostly those where "franku" is missing)

The plan is now:
1. Load this branch to alpha.widelands.org
2. Set up a new database for testing purposes. After that test is ok:
3. Activate the old database of the alpha site and test. If that is ok:
4. Merge this branch to trunk
5. Test on the alpha site again
6. Deploy the changes to the productive website

Known failures:
- ./manage.py inspectdb does not work with sqlite3 because of a failure in the database backend. This will be fixed in Django 1.9. When using mysql it should work.
- If a new user get added through the admin page "Authentication and Authorization" a wl_profile isn't been created automatically.

Deprecation Warnings:
- wsgiref through also a warning: "RemovedInDjango19Warning: `MergeDict` is deprecated, use `dict.update()` instead."

[SirVer (sirver)]

Tremendous effort!!! Chapeau.

> Am 25.04.2016 um 17:45 schrieb kaputtnik <email address hidden>:
>
> ** Description changed:
>
> Just an information on the upgrade to django 1.8
>
> The linked branch is the first version which should work when installing
> from scratch, but don't be too optimistic that really everything works
> fine :-D There are huge changes from django 1.3.7 to django 1.8...
>
> For testing purposes i use a copy from the database from the alpha site,
> just to have more data. But some data is missing nevertheless (f.e.
> comments to maps or news). Problems i faced are described in the .txt
> files of the branch:
>
> - update_problems.txt describes general problems with packages (third party apps)
> - - Database_affected_changes.txt describes things which affects the migration from old data
> + - Database_affected_changes.txt describes things which affects the migration from old data
>
> I want to keep this txt files up to date instead of posting the issues
> here.
>
> All together the code is far away from being productive, but you could
> play around with this version and point me on some failures in an early
> state of development.
>
> Known failures:
> - - news: Currently it is not possible to get news derived from months. But you don't have such news i believe ;)
> - - Preview of content (posts) does not work
> - Feeds do not work
> - Like in the old version you have to create users over the admin page (except the superuser created by './manage.py createsuperuser')
> - ./manage.py inspectdb does not work with sqlite3 because of a failure in the database backend. This will be fixed in Django 1.9. When using mysql it should work.
>
> I am not sure if the changes should be part of current trunk. Because
> the changes are such huge we might want to have a new trunk for
> widelands-website?
>
> --
> You received this bug notification because you are subscribed to
> Widelands Website.
> https://bugs.launchpad.net/bugs/1552425
>
> Title:
> Upgrade to django 1.8
>
> Status in Widelands Website:
> Confirmed
>
> Bug description:
> Just an information on the upgrade to django 1.8
>
> The linked branch is the first version which should work when
> installing from scratch, but don't be too optimistic that really
> everything works fine :-D There are huge changes from django 1.3.7 to
> django 1.8...
>
> For testing purposes i use a copy from the database from the alpha
> site, just to have more data. But some data is missing nevertheless
> (f.e. comments to maps or news). Problems i faced are described in the
> .txt files of the branch:
>
> - update_problems.txt describes general problems with packages (third party apps)
> - Database_affected_changes.txt describes things which affects the migration from old data
>
> I want to keep this txt files up to date instead of posting the issues
> here.
>
> All together the code is far away from being productive, but you could
> play around with this version and point me on some failures in an
> early state of development.
>
> Known failures:
> - Feeds do not work
> - Like in the old version you have to create users over t...

[kaputtnik (franku)]

Thanks. In the last few month i needed a pause from this, because the work follows me into the night ;)

It's a big construction site....

[kaputtnik (franku)]

Some files seems not to be used. I rename them and put a 'delete' after the filename. So they are still there and could be renamed again if they are needed or easy be searched for final deletion/remove.

The feeds are activated for main page (news) and Wiki, but i am not sure about atom feeds. Here in my browser they look the same as on widelands.org. The file atomformat.py seems not to be used so i changed the name as described above.

[kaputtnik (franku)]

Some conclusions for now:

BeautifulSoup:
Version 3.2 is deprecated, but i couldn't get version 4 to work. The main problem is that BeautifulSoup 4 isn't capable of inserting html tags created through unicode strings (like we do for WikiWords or insert_smileys right now). In result we need to implement such things in a differen way. Currently the problems appear in three functions (in mainpage.templatetags.wl_markdown):

  * _insert_revision(): This is used for exchanging revision strings into links while displaying the changelog on the website. Since it is planned to show only sum summary of the changes in the changelog, i think we could omit this function
  * Exchanging of WikiWords to wikilinks: GunChleoc and i wanted to omit this too. It was a bad idea to replace everything that looks like WikiWord (some upper chars in one word) to a link in out wiki, because it also catches some words where no wiki article exists (f.e. "MingW" or "PlayerColor". Instead we should use the WikiLinks extension from python markdown
  * _insert_smileys(): This is IMHO the only useful function. But BeautifulSoup4 isn't really a good tool for replacing parts of strings with tags (like replacing "some text :-) more text" with "some text <img src="imageurl">imagename</img> more text").

All other functions using BeautifulSoup (like "_classify_links()") are working with BeautifulSoup4 (AFAIK). So the main problem is inserting smileys.

* Using "save_mode = ..." for markdown
  The usage of keyw "save_mode" is deprecated in favour of using "bleach" (additional dependency) for sanitizing input of untrusted users. This applies to posts on forum and wiki (edits).

My idea is now to get rid of "save_mode" and use "bleach". And to split the views into "trusted" and "untrusted" so that trusted contents (like news or mainpage) are only using markdown, whereas untrusted contents (like wiki and forum posts) are using markdown AND bleach.

Third Party apps that updated versions won't work anymore:
I am in favour of including this apps as wl specific apps, because new versions are incompatible with old data, or aren't maintained anymore. In result we may lost some old data if we update the third party apps. I believe adapting the third party apps to wl specific apps is more save than trying to update them and adjust the database.

[GunChleoc (gunchleoc)]

I think this sounds like a plan. We can always revisit the third party apps again later as we find replacements for them & the time for implementation.

[SirVer (sirver)]

> - I haven't the root password for mysql on the server. So i couldn't create a test database. I could use the default-mysql-root-user which is standard on debian based systems. Is that ok?

I put the root passwd into a file db.txt in your home directory on the server. Please delete the file once you have the password.

- I have no idea where or how to use the wlggz-stuff. It seems that there are some joints to a ggz-database which is available on the server. F.e. i couldn't delete a user on my local installation (with my copy the database of wl_django_alpha) because this ggz-database is missing on my local mysql. Don't know how to handle this.

We used to use ggz (gnu gaming zone) as our metaserver implementation. When the project went defunct, we decided not to use it anymore. Most of the databases are unused now and we should probably just delete wlggz completely. One thing is still used though: since the metaserver password is saved in cleartext on users machine and not well encrypted during transport is is separate from the website login. We still use wlggz_ggzauth.password for this. We also use .permissions to decide if a user is anonymous/normal/admin on the metaserver. Everything else is unused.

- Is there somewhere a redirection if widelands.org is offline? Some page that says "widelands.org is offline for maintenance stuff and will be back soon"?

No. Users just get a 505 when the homepage is down IMHO. Something like that would be nice though :)

[kaputtnik (franku)]

Thanks :-)

wlggz: I think removing/overhauling this could be done later on.

503 Maintenance site: This would be a nice job for janus :-) I am still confused about network related things...

I have forgotten to spend time on replacing "mark_save" with "bleach" as suggested in #4. I look into this before setting up the tests on alpha.widelands.org.

[SirVer (sirver)]

Not sure where to put this information, but something in the alpha database is snafu:
RuntimeError: Conflicting 'article' models in application 'wiki': <class 'widelands.wiki.models.Article'> and <class 'wiki.models.Article'>.

See /var/log/upstart/alpha.log for the full backtrace.

[kaputtnik (franku)]

I can reproduce the failure now locally. It appears only when starting with command:

gunicorn wlwebsite_wsgi.wsgi

not when invoking with

./manage.py runserver

So i have overlooked this.

[kaputtnik (franku)]

The alpha site is running now with a new database, so all content is missing. i would be happy if someone else took a look it and make some tests (f.e. registering, creating wikiarticles, uploading maps and so on). if you want i could give several acces to more actions.

Happy testing :-)

[GunChleoc (gunchleoc)]

How about giving the forum regulars access for testing? This way we have a better chance of finding any remaining bugs.

[kaputtnik (franku)]

What do you mean with "regular access"? Access without registration? This is imho not possible without modifying lots of code.

The previous plan was to delete the new database on alpha.widelands.org after this test has finished. But i think it would be better to keep the new database and delete the old alpha-database (after testing the migrations). SirVer told me once the old database used with alpha.widelands.org is somehow broken, so having a new database for testing purposes is maybe a good idea. SirVer should tell his opinion about this idea.

It would be nice if someone could give me "superuser" status on widelands.org to get full access to the admin page (currently i have only limited access). I could do so myself by modifying the database directly, but i don't want to do so ...

Currently i found the bug that feeds do not work with wiki. I look into that later on.

[GunChleoc (gunchleoc)]

We could invite everybody who wants to help with testing to register an account at the alpha site.

New database sounds good to me.

You should be superuser now on the live site unless I messed things up.

[kaputtnik (franku)]

Yes, a call for test is planned... and pre written ;) I will post it later on in th Homepage subforum.

I found the bug with the rss feed, but this isn't in the rss feed. Currently when a new wiki article is saved the author isn't saved :-( Doesn't noticed this until now. At least this bug should be solved when a call for testing is postet.

Thanks for superuser status :-)

[kaputtnik (franku)]

Saving Article user is solved, i've just commented out too many code :-D