generated atom types written within source tree, and failing to block later build steps

[David Hart]

Apart from just being bad form (nothing in the build process should write to files in the source tree; writes should happen only in the build tree), something in the process is dependency-unaware, causing parallel make on virgin source trees to fail.

The problem can be easily replicated with these steps:

bzr branch lp:opencog
mkdir opencog/bin; cd opencog/bin
cmake ..
make -j8

FAILS

make
make clean
make -j8

WORKS

The problem can also be recreated where you have read access to but do not own the files in the source tree with these steps:

mkdir ~/build_opencog; cd ~/build_opencog
cmake /usr/local/src/opencog
make

FAILS

[David Hart (dhart)]

FAIL 1

Scanning dependencies of target cogita
[ 50%] Building CXX object opencog/nlp/chatbot/CMakeFiles/cogita.dir/CogitaConfig.cc.o
[ 50%] Building CXX object opencog/nlp/chatbot/CMakeFiles/cogita.dir/IRC.cc.o
[ 52%] Building CXX object opencog/nlp/chatbot/CMakeFiles/cogita.dir/go-irc.cc.o
[ 52%] Building CXX object opencog/nlp/chatbot/CMakeFiles/cogita.dir/whirr-sockets.cc.o
Linking CXX executable cogita
[ 52%] Built target cogita
[ 52%] Generating opencog types
CMake Error at /usr/local/src/opencog/lib/OpenCogAtomTypes.cmake:28 (FILE):
  file Internal CMake error when trying to open file:
  /usr/local/src/opencog/opencog/nlp/types/atom_types.h for writing.

make[2]: *** [/usr/local/src/opencog/opencog/nlp/types/atom_types.h] Error 1
make[1]: *** [opencog/nlp/types/CMakeFiles/nlp-types.dir/all] Error 2
make: *** [all] Error 2

FAIL 2

[ 54%] Generating opencog types
CMake Error at /usr/local/src/opencog/lib/OpenCogAtomTypes.cmake:28 (FILE):
  file Internal CMake error when trying to open file:
  /usr/local/src/opencog/opencog/embodiment/AtomSpaceExtensions/atom_types.h
  for writing.

[ 54%] make[2]: *** [/usr/local/src/opencog/opencog/embodiment/AtomSpaceExtensions/atom_types.h] Error 1
make[1]: *** [opencog/embodiment/AtomSpaceExtensions/CMakeFiles/embodiment_atom_types.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
Built target Control
make: *** [all] Error 2

[linas (linasvepstas)]

Well, the error message does say "Internal CMake error" so maybe this is actually a cmake bug.

Perhaps multiple threads are all trying to write this file, and this confuses cmake?

[linas (linasvepstas)]

Ahh .. I see .. perhaps the problem is that the autogenerated files should not be written to the source tree, but should be written to the build tree instead ...

[linas (linasvepstas)]

the script that generates these is located in lib/OpenCogAtomTypes.cmake

[linas (linasvepstas)]

This is fixed, I beleive, in bzr rev 7793.

[xoicy (xoicy)]

i have the same problem.

[ 84%] Building CXX object opencog/spacetime/CMakeFiles/space_server.dir/HandleToTemporalEntryMap.cc.o
In file included from /home/xoicy/opencog/opencog/atomspace/TLB.h:29:0,
                 from /home/xoicy/opencog/opencog/spacetime/HandleToTemporalEntryMap.cc:28:
/home/xoicy/opencog/opencog/atomspace/Atom.h:33:42: fatal error: opencog/atomspace/atom_types.h: No such file or directory
compilation terminated.
make[2]: *** [opencog/spacetime/CMakeFiles/space_server.dir/HandleToTemporalEntryMap.cc.o] Error 1
make[1]: *** [opencog/spacetime/CMakeFiles/space_server.dir/all] Error 2
make: *** [all] Error 2

[linas (linasvepstas)]

Hi xoicy,

Can you verify that this is still occurring on the latest bzr rev? You may need to cmake .. to get it all to come out correctly.

These files are now located in the build directory, and the makefiles should be looking for them there.

[linas (linasvepstas)]

I am seeing one issue: when building with multiple threads, it looks like the file autogenerator will be invoked from multiple threads, writing the same file, and occasionally corrupting it s a result.

[David Hart (dhart)]

linas: Yes, I've seen this too. I'm not a big fan of the complexity of atom_type autogeneration, and adding a regestration/queuing system with a single generation step at the end would add yet more complexity. Is it possible, do you think to write an overall less complex but more robust and thread-safe atom_types generation system? Or perhaps better yet, do this at runtime initialization?

[linas (linasvepstas)]

On 15 October 2012 22:13, David Hart <email address hidden> wrote:

Or perhaps better yet, do
>
> this at runtime initialization?
>

Yes, what we really need is a way of declaring types at run-time. (and not
just initialization) This requires more work, and as I am not an active
user of the atomspace just right now, am fain to "fix it" (viz, I can't fix
it in a short afternoon)

After a move to git, you could try making "dynamic runtime atom types" a
new rquested feature, in github. This is something that a coder with no AI
experience, could implement ...

--linas

[xoicy (xoicy)]

My revision number is 7806. So, i created "/home/../opencog/bin" directory and try to execute this "sudo cmake .." and i get this error atom_types.h: No such file or directory.

[David Hart (dhart)]

xoicy: while in your bin directory, do 'rm -rf *', then 'cmake ..' with no sudo.

[xoicy (xoicy)]

I clear my "bin" directory with `rm -rf *`, then execute:

cmake .. -DLIB_DIR_SUFFIX="/usr/local/lib/"
make

and i've got this error.

[ 84%] Building CXX object opencog/cython/CMakeFiles/cogserver_cython.dir/opencog/cogserver.cpp.o
In file included from /home/xoicy/opencog/bin/opencog/cython/opencog/cogserver.cpp:242:0:
/home/xoicy/opencog/opencog/atomspace/ClassServer.h:35:42: opencog/atomspace/atom_types.h: Нет такого файла или каталога
компиляция прервана.
make[2]: *** [opencog/cython/CMakeFiles/cogserver_cython.dir/opencog/cogserver.cpp.o]
make[1]: *** [opencog/cython/CMakeFiles/cogserver_cython.dir/all]
make: *** [all]

[linas (linasvepstas)]

This should fix it:

=== modified file 'opencog/cython/CMakeLists.txt'
--- opencog/cython/CMakeLists.txt 2012-10-21 22:43:19 +0000
+++ opencog/cython/CMakeLists.txt 2012-10-21 23:01:04 +0000
@@ -65,6 +65,8 @@
     opencog/cogserver.cpp
 )

+ADD_DEPENDENCIES(atomspace_cython opencog_atom_types)
+
 # cogserver just has the MindAgent template class at the moment
 #TARGET_LINK_LIBRARIES(cogserver_cython
 #)

[xoicy (xoicy)]

thank you, linas. now is everything all right.