gizmod in Ibex fails to run on amd64 systems due to bug in boost-1.34.1

[Tim Burrell]

Binary package hint: gizmod

AMD64 users see this upon executing gizmod in Ibex:

gizmod: symbol lookup error: /usr/lib/libboost_python-gcc42-1_34_1.so.1.34.1: undefined symbol: Py_InitModule4

Apparently this is due to python having renamed a function and boost not catching it... as noted by one gizmod user:

Found the following information on the problem by googling "Py_InitModule4". Looks like quite a few projects have been affected. Here's an interesting one: https://launchpad.net/ubuntu/+source/clearsilver/+bug/86685

"Py_InitModule4 in 64bit systems was renamed to Py_InitModule4_64 to allow addressing more than 4Gbytes"
https://launchpad.net/ubuntu/+source/clearsilver/+bug/86685/comments/1

Note I am the author of Gizmod and am fine with pushing out a new release version of gizmod for this if need be, but this is currently stopping amd64 users from using gizmod so I'd like to get a new version pushed out or backported if possible.

I believe the fix is as simple as recompiling gizmod with boost-1.35 linked in as opposed to 1.34.1 but I can't test as I don't have an amd64 system.

[Tim Burrell (tim-burrell)]

TEST CASE:

(only affects amd64 Ibex systems)
sudo apt-get install gizmod
gizmod

ACTUAL OUTPUT:

GizmoDaemon v3.4 -=- (c) 2007, Tim Burrell <email address hidden>
=---------=

gizmod: symbol lookup error: /usr/lib/libboost_python-gcc42-1_34_1.so.1.34.1: undefined symbol: Py_InitModule4

EXPECTED OUTPUT:

No symbol lookup error.

[James Westby (james-w)]

Hi,

Thanks for the report, do you have confirmation from someone that
recompiling gizmod fixes the problem? I don't have amd64, otherwise
I would confirm for you.

Thanks,

James

[Tim Burrell (tim-burrell)]

Hi James,

Unfortunately I don't have confirmation... no one has tried the recompile. It's a fairly intensive compile requiring a couple hundred megs of dev packages so understandable.

In the interim I've learned a bit more of what's going on. I guess the default in Hardy was python 2.4, and in Ibex it's 2.5? The actual problem is that boost 1.34.1 was only intended to be built with python 2.4. When built with python 2.5 (which has had the aforementioned ABI change), boost 1.34.1 breaks.

More details on this here:

http://lists.debian.org/debian-python/2008/02/msg00033.html
http://lists.debian.org/debian-python/2008/03/msg00033.html

I googled quite a bit to see if I could find confirmation that boost 1.35.0 fixes this issue, but I cannot. Basically I have no idea if it will fix it or not. I am sort of guessing it will, but it would be nice to know for sure :). Is there anyone who can help me test this theory?

[James Westby (james-w)]

On Mon, 2008-11-03 at 14:33 +0000, Tim. wrote:
> Hi James,
>
> Unfortunately I don't have confirmation... no one has tried the
> recompile. It's a fairly intensive compile requiring a couple hundred
> megs of dev packages so understandable.
>
> In the interim I've learned a bit more of what's going on. I guess the
> default in Hardy was python 2.4, and in Ibex it's 2.5? The actual
> problem is that boost 1.34.1 was only intended to be built with python
> 2.4. When built with python 2.5 (which has had the aforementioned ABI
> change), boost 1.34.1 breaks.
>
> More details on this here:
>
> http://lists.debian.org/debian-python/2008/02/msg00033.html
> http://lists.debian.org/debian-python/2008/03/msg00033.html
>
> I googled quite a bit to see if I could find confirmation that boost
> 1.35.0 fixes this issue, but I cannot. Basically I have no idea if it
> will fix it or not. I am sort of guessing it will, but it would be nice
> to know for sure :). Is there anyone who can help me test this theory?
>

Whoever reported the problem can presumably help you.

I suggest you push the package to a PPA, and then requesting testing
from amd64 users. That will rebuild the package with the newer boost
and as such confirm whether the problem is fixed.

Thanks,

James

[Bernardo Kuri (accounts-bernardokuri)]

Hello, I am the one who originally reported this problem to Tim.

I am more than willing to help recompiling gizmod to see if the issue goes away. However, I have been unable to successfully compile by using the official instructions (http://gizmod.wiki.sourceforge.net/Compile+from+Source).

Are there any considerations that I should take in order to compile successfully using an AMD64 system (i.e. compilation flags, system variables, etc)?

I am willing to provide compiled debs as well, but I am not familiar with that process as well. I am fairly new to Linux, so any help with this would be appreciated.

[Tim Burrell (tim-burrell)]

Hey Bernardo,

Could you either post a message on the gizmod forums or open a new bug against the documentation, and describe in detail what step(s) when wrong and where? I think I may know what went wrong already, and probably has to do with some source changes that were necessary for gcc 4.3 that comes with Ibex.

If it was a compile error, and it's not too much trouble, could you try installing from SVN? http://gizmod.wiki.sourceforge.net/Compile+from+SVN

Thanks for all your help on this!

Tim.

[Christopher Peplin (chris.peplin)]

I have run into this same bug, and tried to compile from source as reccomended. I followed up to Bernardo's bug report here: https://sourceforge.net/tracker/index.php?func=detail&aid=2221145&group_id=139428&atid=743549 .

Chris

[Christopher Peplin (chris.peplin)]

I had some luck after upgrading to boost 1.36 from this ppa: http://ppa.launchpad.net/rjmyst3/ubuntu and making a few modifications to the Gizmod source to compile with the newer library. I've attached a package build with checkinstall here, if anyone else wants to try it out.

[Tim Burrell (tim-burrell)]

Thanks for that Peplin!

I've updated the gizmod svn repo to compile clean with boost 1.36. Looks like this should be ticket for AMD64 users. Unfortunately I'm not sure how this affects gizmod's currently broken state in Ibex, being that 1.36 is a requirement and 1.36 isn't in Ibex.

Are any MOTUs following along? Anyone know what the right thing to do here is? I'm hoping for the answer not to be to leave it broken, but if that's the only option I can provide debs on the Gizmod home page.

[Christopher Peplin (chris.peplin)]

No problem, Time - I'm happy to have gizmod working again, it's a great program.

I can't offer any advice as to how to get this fixed in Intrepid. I doubt boost 1.36 will make it to backports, but it's possible a smaller patch could be pushed. This is almost certainly effecting other packages as well, no?

Also, on another AMD64 machine with a clean Intrepid install I did try upgrading to the boost 1.35 packages, but that didn't help - compilation still errored out in serialization/vector.hpp.

For anyone trying that deb, you will need to create a symlink for libGizmod like this:

$ sudo ln -s /usr/lib/libGizmod.so.3.4.0 /usr/lib/libGizmod.so.3

Not sure what I did wrong there, but this will fix it.

[James Westby (james-w)]

Tim,

We can fix this in Intrepid somehow. We need to find out what the fix is
though.

You suggested that a rebuild was fine, but Chistopher is now suggesting that
it fails to build.

I've uploaded gizmod to my PPA at

  https://edge.launchpad.net/~james-w/+archive

which will force a rebuild. If an amd64 Intrepid user can test the debs from
there and report back the results that will help.

Thanks,

James

[Christopher Peplin (chris.peplin)]

Let me clarify what I did:

Clean Intrepid install (libboost1.34 installed) - gizmod package fails to run, recompile from SVN stops with boost errors

Intrepid install with libboost1.35 installed from repositories (proposed?) - same thing as above

Intrepid with libboost1.36 from PPA (http://ppa.launchpad.net/rjmyst3/ubuntu) - gizmod package fails to run, but compilation from SVN works and runs fine

[James Westby (james-w)]

Hi,

The package does fail to build

[ 1%] Building CXX object libH/CMakeFiles/H.dir/Average.o
cd /build/buildd/gizmod-3.4/obj-i486-linux-gnu/libH && /usr/bin/g++ -DHAVE_CONFIG_H -g -O2 -g -Wall -O2 -O3 -DNDEBUG -I/build/buildd/gizmod-3.4/obj-i486-linux-gnu/libH -I/build/buildd/gizmod-3.4/libH -I/build/buildd/gizmod-3.4/obj-i486-linux-gnu -fPIC -Wall -Werror -o CMakeFiles/H.dir/Average.o -c /build/buildd/gizmod-3.4/libH/Average.cpp
/usr/bin/cmake -E cmake_progress_report /build/buildd/gizmod-3.4/obj-i486-linux-gnu/CMakeFiles 2
[ 3%] Building CXX object libH/CMakeFiles/H.dir/Debug.o
cd /build/buildd/gizmod-3.4/obj-i486-linux-gnu/libH && /usr/bin/g++ -DHAVE_CONFIG_H -g -O2 -g -Wall -O2 -O3 -DNDEBUG -I/build/buildd/gizmod-3.4/obj-i486-linux-gnu/libH -I/build/buildd/gizmod-3.4/libH -I/build/buildd/gizmod-3.4/obj-i486-linux-gnu -fPIC -Wall -Werror -o CMakeFiles/H.dir/Debug.o -c /build/buildd/gizmod-3.4/libH/Debug.cpp
/usr/bin/cmake -E cmake_progress_report /build/buildd/gizmod-3.4/obj-i486-linux-gnu/CMakeFiles 3
[ 5%] Building CXX object libH/CMakeFiles/H.dir/Exception.o
cd /build/buildd/gizmod-3.4/obj-i486-linux-gnu/libH && /usr/bin/g++ -DHAVE_CONFIG_H -g -O2 -g -Wall -O2 -O3 -DNDEBUG -I/build/buildd/gizmod-3.4/obj-i486-linux-gnu/libH -I/build/buildd/gizmod-3.4/libH -I/build/buildd/gizmod-3.4/obj-i486-linux-gnu -fPIC -Wall -Werror -o CMakeFiles/H.dir/Exception.o -c /build/buildd/gizmod-3.4/libH/Exception.cpp
/usr/bin/cmake -E cmake_progress_report /build/buildd/gizmod-3.4/obj-i486-linux-gnu/CMakeFiles 4
[ 7%] Building CXX object libH/CMakeFiles/H.dir/FileEventWatcher.o
cd /build/buildd/gizmod-3.4/obj-i486-linux-gnu/libH && /usr/bin/g++ -DHAVE_CONFIG_H -g -O2 -g -Wall -O2 -O3 -DNDEBUG -I/build/buildd/gizmod-3.4/obj-i486-linux-gnu/libH -I/build/buildd/gizmod-3.4/libH -I/build/buildd/gizmod-3.4/obj-i486-linux-gnu -fPIC -Wall -Werror -o CMakeFiles/H.dir/FileEventWatcher.o -c /build/buildd/gizmod-3.4/libH/FileEventWatcher.cpp
cc1plus: warnings being treated as errors
/build/buildd/gizmod-3.4/libH/FileEventWatcher.cpp: In member function 'void H::FileEventWatcher::handleEventsOnFile(pollfd&)':
/build/buildd/gizmod-3.4/libH/FileEventWatcher.cpp:456: error: suggest explicit braces to avoid ambiguous 'else'

Full build log at

http://launchpadlibrarian.net/19520688/buildlog_ubuntu-intrepid-i386.gizmod_3.4-0ubuntu1%2B0jw1_FAILEDTOBUILD.txt.gz

this is pretty critical to make any changes in Intrepid.

Thanks,

James

[Tim Burrell (tim-burrell)]

Hi James,

Yes I had to make some source changes to accommodate boost 1.36.

If it's helpful I could push out a new version (3.4.1 perhaps)? Or provide a patch set to you? Whatever will be easiest / best for you guys is fine by me.

Thanks,

Tim.

[James Westby (james-w)]

On Sun, 2008-11-09 at 21:44 +0000, Tim Burrell wrote:
> Hi James,
>
> Yes I had to make some source changes to accommodate boost 1.36.
>
> If it's helpful I could push out a new version (3.4.1 perhaps)? Or
> provide a patch set to you? Whatever will be easiest / best for you
> guys is fine by me.

This is with boost 1.34 though. Ubuntu doesn't have 1.36 yet as far
as I can see.

     boost | 1.34.1-11ubuntu1 | intrepid | source
     boost | 1.34.1-11ubuntu1 | jaunty | source

Thanks,

James

[Tim Burrell (tim-burrell)]

Hey James,

Yeah unfortunately the bug is with boost and there is no way to fix it unless the move is made to 1.36. This is why I was wondering if there was a way forward with this at all?

If not, understandable, given the odd set of circumstances. I don't mind providing AMD64 debs on the gizmod homepage until the next Ubuntu release comes out. That being said, I can't help but wonder if this isn't affecting any other packages too? Any Ibex packages that use boost.python must be broken on AMD64. Does anyone know of any other packages using boost.python, and if so what are they doing to get around this?

Thanks,

Tim.

[Tim Burrell (tim-burrell)]

I suppose another possibility might be to provide python 2.4 in the repo, not sure if that's possible either? Technically linking boost <= 1.35 against python > 2.4 is wrong (according to the boost people), so I guess it's not so much a bug in boost as it is an Ubuntu packaging issue.

[James Westby (james-w)]

Hi Tim,

The error from my PPA build appears to be unrelated to boost.

The code it is complaining about is the ambiguous else clause in

if (BytesRead < 0)
   if (errno == EINTR)
     return;
   else
     throw H::Exception("Failed to Read from Inotify Device!", __FILE__, __FUNCTION__, __LINE__)

and the build is being done with -Werror.

I just pushed a new package to my PPA adding the explicit else, so we can
see if that builds.

Thanks,

James

[Tim Burrell (tim-burrell)]

Oh right... yes that's fixed in SVN as well. The -Werror by default is also fixed in SVN :).

There's a couple other places where that happens with gcc 4.3 I think, so it might fail again. Was there not a patch for this? How did it end up building in the first place? Interesting :).

At any rate, once those are taken care of it should build fine, but if you're building it with boost <= 1.35 and python > 2.4, although the compile will go fine, it will not run on AMD64.

Boost 1.36 is required for python 2.5 and higher on AMD64.

Tim.

[James Westby (james-w)]

Hi,

Now I get

cd /tmp/buildd/gizmod-3.4/obj-i486-linux-gnu/libH && /usr/bin/g++ -DHAVE_CONFIG_H -g -O2 -g -Wall -O2 -O3 -DNDEBUG -I/tmp/buildd/gizmod-3.4/obj-i486-linux-gnu/libH -I/tmp/buildd/gizmod-3.4/libH -I/tmp/buildd/gizmod-3.4/obj-i486-linux-gnu -fPIC -Wall -Werror -o CMakeFiles/H.dir/SocketServer.o -c /tmp/buildd/gizmod-3.4/libH/SocketServer.cpp
/usr/bin/cmake -E cmake_progress_report /tmp/buildd/gizmod-3.4/obj-i486-linux-gnu/CMakeFiles 12
[ 22%] Building CXX object libH/CMakeFiles/H.dir/Util.o
cd /tmp/buildd/gizmod-3.4/obj-i486-linux-gnu/libH && /usr/bin/g++ -DHAVE_CONFIG_H -g -O2 -g -Wall -O2 -O3 -DNDEBUG -I/tmp/buildd/gizmod-3.4/obj-i486-linux-gnu/libH -I/tmp/buildd/gizmod-3.4/libH -I/tmp/buildd/gizmod-3.4/obj-i486-linux-gnu -fPIC -Wall -Werror -o CMakeFiles/H.dir/Util.o -c /tmp/buildd/gizmod-3.4/libH/Util.cpp
In file included from /tmp/buildd/gizmod-3.4/libH/Util.cpp:29:
/tmp/buildd/gizmod-3.4/libH/Util.hpp: In function 'Functor H::for_all(Object&, Functor)':
/tmp/buildd/gizmod-3.4/libH/Util.hpp:66: error: 'for_each' is not a member of 'std'
make[3]: *** [libH/CMakeFiles/H.dir/Util.o] Error 1

You originally said that you thought rebuilding would fix it, is that incorrect?

If so then the best thing would be to find the patch in boost that fixed it, and
apply that to boost in Intrepid and rebuild gizmod.

Thanks,

James

[Tim Burrell (tim-burrell)]

Hi James,

Yes sorry for the confusion. I did originally say that, but that was just my guess without being able to test.

Since then I've learned that either boost 1.36 is required, or python 2.4 is. There's simply no way to make this work with the python 2.5 that's in Ibex's repos, without either patching boost, or upgrading to boost 1.36.

If you're attempting to build with boost 1.35 and python 2.5 it's not going to work even if it does finish compiling.

So basically, I'm not sure what the right thing to do is here. It seems unlikely that Ibex will push a new version of boost in, or an old version of python correct? In that case Is it possible to remove gizmod from the amd64 repos? I don't know how Ubuntu usually deals with these situations.

Thanks for all your effort on this btw! And again sorry for the confusion.

Tim.

[James Westby (james-w)]

On Mon, 2008-11-10 at 15:30 +0000, Tim Burrell wrote:
> So basically, I'm not sure what the right thing to do is here. It seems
> unlikely that Ibex will push a new version of boost in, or an old
> version of python correct? In that case Is it possible to remove gizmod
> from the amd64 repos? I don't know how Ubuntu usually deals with these
> situations.

The preferred thing to do would be to patch boost to fix whatever
problem was causing this bug.

Do you know which change in boost fixed this?

Thanks,

James

[Tim Burrell (tim-burrell)]

That sounds preferable to me too.

I don't have time to look into it just now, but I can take a look later. I don't think it should be a significant change. I keep wondering if any other packages are dealing with this right now as well?

From the threads here: http://lists.debian.org/debian-python/2008/03/msg00034.html

and here: http://lists.debian.org/debian-python/2008/03/msg00033.html

The python defect relating to this is here: http://www.python.org/dev/peps/pep-0353/

This may be relevant as well: https://launchpad.net/+search?field.text=Py_InitModule4_64+undefined

It seems like it might just be a build thing, with an additional flag being required by bjam. I wonder, does Ubuntu's build call bjam with python 2.5 specified?

Googling for this produces quite a few results actually, but unfortunately most of the resolutions I see involve linking against python 2.4 instead of 2.5 :(.

So far I'm not sure what the fix is, but I ran out of time for the moment. Will have to come back to this later.

Tim.

[Ilya Barygin (randomaction)]

I uploaded a version of gizmod which should have FTBFS bugs fixed and will build against boost 1.40. It would be nice if someone with a 64-bit Lucid system (e.g. LiveCD) could check if this bug has been fixed in gizmod 3.4-0ubuntu2.