New way of classpath and JVM Singleton handling

[Cezar Andrei]

This bug is to track work related to makeing Zorba modules implemented in Java be robust and working all together.

It was decided that a few things need to happen:
Critical:
1. (chris) add cmake macros to let build know a module is using java
2. (chris) cmake macros that define it's own jar implementation
3. (chris) cmake macros that define external dependencies (like xmlbeans.jar, xsl-fo.jar)
4. (cezar) add zorba classpath option, and CLASSPATH environment
5. (chris) add core module that can resolve a known (by URI) external module
6. (chris) add external java-helper module that provides concatenation of classpathes
  - the order on calsspath is: ModuleOwnPath, ModuleDependecy, zorba -cp, env CLASSPATH
7. (chris) make sure binary packages set up classpathes correctly
8. (cezar) add code to external java-helper module to create a singleton JVM
9. (cezar) refactor xmlbeans and xsl-fo modules to make use of external java-helper module

Nice to have, not critical:
10. Have a generic way to implement modules directly in Java, Python etc, using swig extension.

[Chris Hillery (ceejatec)]

I don't think #1 is necessary. It may be necessary when we consolidate the JVM startup (#8) so Zorba can know whether a particular module import requires the JVM, but that depends on exactly how we do that refactoring.

I don't understand #5; what does that mean? Is that the "given a URI, return the ExternalModule object for it" feature? If so, that's not necessary yet either (and we're not sure it's even desirable or safe). Also, that would only be implemented at the C++ API level, not in a module, I don't think.

[Cezar Andrei (cezar-andrei)]

#1 may not be required like you said, I wanted to have it on the list

About the second part, you're correct the #5 feature would be implemented at the C++ API level not in a module.

[Cezar Andrei (cezar-andrei)]

Related to modules: another feature is code sharing between modules: utility functions and even more functionality.

[Chris Hillery (ceejatec)]

OK, I've completed #2 and #3.

There's a new macro, which needs some better documentation once we understand exactly what the whole Java story is:

# Inform Zorba of a .jar file that should be made available on the CLASSPATH
# of the JVM, should the JVM be started. QQQ more doc needed
#
# Args: FILE - path to file (must be absolute)
#       EXTERNAL - (optional) FILE specifies a path that should be added
#              to CLASSPATH as-is
#       TEST_ONLY - (optional) Jar file is for testcases only and should not
#              be installed

MACRO (DECLARE_ZORBA_JAR)
  ....

There's also a test case in test/rbkt/modules/java. For the time being this doesn't do anything module-ish; it just creates a .jar file and uses DECLARE_ZORBA_JAR() on it. Also note that the test case depends on CMake 2.8.6, which has significantly improved support for Java.

DECLARE_ZORBA_JAR(FILE ..../foo.jar) will put foo.jar into the directory "jars/" on Zorba's library path (ie, LIB_PATH/jars in your build directory, or lib/zorba/core/2.1.0/jars (Zorba core module) or lib/zorba/jars (non-core module) in your installed Zorba).

DECLARE_ZORBA_JAR(FILE ..../foo.jar EXTERNAL) will cause a file called <project>-jars.txt to be created in the same "jars/" directories as above, with the full path to foo.jar listed in it. (Multiple external jars for the same project will be listed on separate lines in that file.)

The upshot is that when Zorba needs to create a classpath for the JVM, all it needs to do is:

1. Step through each directory on the Library Path until it finds a "jars" directory; call this JARS_DIR.
2. Add each .jar file in JARS_DIR to the classpath.
3. For each -jars.txt file in JARS_DIR, add each line of the file to the classpath.

[Chris Hillery (ceejatec)]

One problem I thought of: The Library Path I mentioned above is actually a feature of the static context. When Zorba needs to launch the JVM, the only static context it would be reasonable to look at would be the root static context. However, this is not the static context which gets populated by zorbacmd's --lib-path argument; zorbacmd creates a child static context for executing the query, and populates that static context's library path.

So, a user could not use --lib-path (or --module-path) to point to an alternate module library installation directory and then invoke a query using a Java-based module. I'm not sure yet what the right answer here is. Perhaps zorbacmd could just directly modify the root static context rather than the child.

Also, we will need to document for C++ API users that only the root static context's library path will be used for the JVM classpath.

[Cezar Andrei (cezar-andrei)]

Along side of DECLARE_ZORBA_JAR(FILE ..../foo.jar EXTERNAL), can you also make a macro that adds a PATH list not only one jar?
Maybe: DECLARE_ZORBA_JAR(PATH ..../foo.jar:../bar.jar:etc EXTERNAL)

[Chris Hillery (ceejatec)]

I guess, although you can also just call DECLARE_ZORBA_JAR multiple times
for the same effect. I didn't want to directly accept a path because I
wasn't sure how to handle : vs ; path separators.

[Chris Hillery (ceejatec)]

I've made two tweaks by request:

1. You can pass multiple filenames to DECLARE_ZORBA_JAR(), eg.

   DECLARE_ZORBA_JAR(FILE /path/to/myjar.jar /path/to/myjar2.jar)

You can't pass a :-separated path, however. Also, the EXTERNAL flag, if specified, applies to all named jars.

2. I've added the names of installed jars to the -classpath.txt files in addition to the absolute paths to external jars. So you only have to scan the jars/ directory for the .txt files, not the .jar files. Then, for each line in the text file: If the path is absolute, put it on the classpath; if it's relative, resolve it to the same directory as the .txt file itself is in, and put it on the classpath.

(Note that I've changed the name of the .txt file to <project_name>-classpath.txt.)

[Cezar Andrei (cezar-andrei)]

4. 8. Done.
9. xmlbeans module done, xsl-fo remaining.