LUAI_UINT32 no longer exists

[Dirk Stöcker]

Hello,

in src/scripting/pdep/llimits.h the type LUAI_UINT32 is used. This does no longer exists in lua 5.2, which prevents building widelands.

Could this please be fixed, so that building is possible agian.

[Nasenbaer (nasenbaer)]

I guess the big question is, why we provide our own version of pdep, as the README of the files in our source notes:

These files are directly copied from the Lua distribution, with the
exception of lzio.h, which is s/lua{ZM}/pdep/g and has an include removed.

SirVer, I assigned you, as you hopefully know, why we ship our own pdep and how to handle the problem.

[SirVer (sirver)]

Okay, here is the story: For saving lua data to files we need a much deeper inside into the implementation of Lua than Lua cares to give. Therefore we not only need access to the public headers, but also the private ones. Since they are not included in dev packages of lua, we had to copy them over. This basically bundles a modified version of the (now abandoned) pluto library[1].

That said, switching from Lua 5.1 to Lua 5.2 will be a non trivial undertaking for the moment, because the internal format has likely changed, there are new types that we need to learn how to persist and so on and so. It will also (likely) break save game compatibility.

We link Lua statically (for this very reason) which is custom in the Lua world, that means only people building widelands ever need a lua library on their system. Lua 5.2 is not a big step forward for Lua, I am unsure if it is wise to even work towards supporting it.

[1] http://luaforge.net/projects/pluto/

[Hans Joachim Desserud (hjd)]

After a recent upgrade of lua, this bug has now hit Arch. From what I can see, the 5.1 version should still be available

pacman -Ss lua
extra/lua 5.2.1-3 [installed]
    A powerful light-weight programming language designed for extending applications
extra/lua51 5.1.5-3 [installed]
    A powerful light-weight programming language designed for extending applications
...

However, if I uninstall lua (5.2) expecting lua51 to be used instead I get the following error message:
/src/scripting/pdep/llimits.h:15:19: fatal error: lua.hpp: No such file or directory.

[SirVer (sirver)]

I cannot help with the arch linux problem. But just for the record, pluto (the lua persisting library) is continued to be developed on github.
https://github.com/hoelzro/pluto/network

It has no support for Lua 5.2 though.

[SirVer (sirver)]

Someone else seems to have solved the persistence problem in a version independant manner:
https://code.google.com/p/corsix-th/wiki/Persistance

[Hans Joachim Desserud (hjd)]

Turns out the problem with Arch was less dramatic than I thought at first. It will obviously fail with lua (version 5.2), but it works fine with only lua51 installed. (Just make sure to build completely from scratch if you have attempted to build with 5.2). I guess when I tried to compile it earlier, something still pointed to the newer version, thus failing the build.

I have updated the package list for building on Arch in the wiki.

Note that while we can use the lua51 package for now, it has been orphaned in Arch (https://www.archlinux.org/packages/extra/i686/lua51/).

[SirVer (sirver)]

The corsix code is not able to persist Lua coroutines which is critical for us.

I reached out to the Luajit mailinglist[1] for advise and the consensus there is to bundle a version of Lua with the source code and generally to stay away from Lua 5.2 as it is not very widely adopted yet and has (apparently) some non backwards compatibility.

I feel that this is very much non-critical at this point in time. Lua 5.2 is not a drop in replacement for Lua 5.1 so will not supersede it in the packages as quickly. If it becomes an issue, we can bundle Lua 5.1 with our code(it is tiny) - this would be nice for other reasons as well, as we could add new features :).

I am reducing this in criticality and targetting and unassign myself.

[1] http://www.freelists.org/post/luajit/Persistence-in-luajit

[SirVer (sirver)]

Of interest is that Fedora seems to have decided to keep lua 51 around indefinitely: http://<email address hidden>/msg59897.html

[Peter de Ridder (cavalier)]

I want to ask you if you could include the Lua source with Widelands as you are really dependant on it and require the internals.
As mentioned before this brings other benefits too.

For Lunar-Linux we currently extract the Lua source in the build tree and use the following to make Widelands use it:
make -C lua-5.1.5 linux # configure and compile Lua
make -C lua-5.1.5 local # make an in-tree installation
export LUA_DIR=lua-5.1.5 # tell cmake to use this lua

[SirVer (sirver)]

I agree that this is the way to go in the future. Not for b18 though.

The way forward is likely https://github.com/fnuecke/eris which seems great. We would need to dump this into our repo and build Lua while building Widelands - this seems to be the preferred approach of embedding Lua in bigger projects anyways, so I think it is a suitable approach for us as well.

[Nasenbaer (nasenbaer)]

Maybe it would be good to alter the CMake file that way, that it does not accept LUA 5.2 or newer and instead tells the user about the incompatibility?

I am pretty sure there will be quite some problems reported with compiling once build18 is released.

[Jens Beyer (qcumber-some)]

This should be doable if we set the requirement of CMake to Version 2.8 (if we don't have done that already) - CMake 2.8 should ship a FindLua51.cmake file to search for that version explicitly.

I can do that thursday night and request a review/merge if no one is doing it earlier ;-)

[Jens Beyer (qcumber-some)]

Hmm... just checked the sources.

We declare:
find_package(Lua51 REQUIRED)

The FindLua51.cmake file from CMake 2.8 extracts the version information from lua.h and should fail via HANDLE_STANDARD_ARGS.

Doublechecked it on Debian Jessie with liblua5.1-dev uninstalled and liblua5.2-dev installed - failed with required package not found.

I guess the suggestion from #11 is already there.

[SirVer (sirver)]

Please do not commit a fix for this before b18 is out - it is touching the build infrastructure which we should probably not do after feature freeze.

[Jens Beyer (qcumber-some)]

I agree, however, as I said, in my eyes there is no change necessary. ;-)

[SirVer (sirver)]

I started working on this in lp:~widelands-dev/widelands/lua_eris

[Ryan Schmidt (z-launchpad-d)]

widelands has been unbuildable in MacPorts since its lua was updated to 5.2.x 6 months ago. Please work to make widelands compatible with lua 5.2. Thanks.

[SirVer (sirver)]

1) This is marked in Progress - that means it is in progress. No need to pressure us, it won't go faster.
2) Lua 5.2 is a very different beast than 5.1 - not even the Lua community suggest replacing your usage of 5.1 with 5.2, only new projects should use 5.2. The versions are not even compatible on a syntax level - even less on a binary level. MacPorts are wrong if they *replaced* 5.1 with 5.2. They should bundle both versions. Go, complain to them instead.
3) homebrew offers lua 5.1 - you could use that.
4) Or just do the trivial install yourself. wget http://www.lua.org/ftp/lua-5.1.4.tar.gz && tar xzvf lua-5.1.4.tar.gz && cd lua-5.1.4 && make macosx && sudo make install

We will eventually use Lua 5.2, but this will break safegame compatibility, so we step carefully.

[Ryan Schmidt (z-launchpad-d)]

I am a manager of the MacPorts project. I will investigate what would be involved in re-adding lua 5.1.

[SirVer (sirver)]

Ryan, after b18 we will very quickly move to shipping our own version of Lua - the code is already written. So if you want to wait for a few more months your trouble will go away soon. However, as mentioned superseeding 5.1 with 5.2 is just the wrong move for the macports project as a whole.

[Hans Joachim Desserud (hjd)]

I think the release notes for build18 should mention that lua5.2 is not supported yet, similar to how we mentioned the gcc 4.7 compilation issue last time (https://launchpad.net/widelands/build17/build-17). Shouldn't have any effect on Windows/OS X where we ship binaries, but it is useful information for packagers/users of other systems.

[SirVer (sirver)]

I think this is less of an issues than it was for build 17. Mainly because the lua community acknowledges that moving from 5.1 to 5.2 is very difficult and not feasible for many projects. If you feel strongly about it, feel free to write a paragraph into the release docs - mail me your google account and I give you write access to the document or just post it somewhere and I'll copy and paste from there.

[Hans Joachim Desserud (hjd)]

"Known issues / Notes to packagers:

Lua
Widelands currently depends on lua5.1, and will not build with lua5.2. This is being worked on (https://bugs.launchpad.net/widelands/+bug/1073932)."

Something like that should be sufficient I guess. I put a / in the title because I couldn't really decide. The release notes should contain only one though.

[SirVer (sirver)]

We no longer depend on any lua version - we ship Lua with the game now.

[Joseph O Morrow (josephomorrow-o)]

Building in Fedora 20:

Extract widelands build 18 into a source folder.

Make sure dependency packages are all installed.

Then install lua-5.1 compat packages:

#yum install compat-lua*

Then run "ccmake <path to unpacked widelands-build18-src folder>" in a build folder and "c" (configure).
Find (in "t" advanced mode) LUA_INCLUDE_DIR and set it to: /usr/include/lua-5.1
(All other settings should be OK for building to install into /usr/local)

Then "c" (configure) again, "g" (generate), "make", and "sudo make install".

[GunChleoc (gunchleoc)]

Fixed in build19-rc1.