flex header incompatibility

[Chris Hillery]

For an unknown reason, on Mountain Lion, the generated files from Flex are incompatible with the clang compiler in Xcode, resulting in a compilation failure.

[Paul J. Lucas (paul-lucas)]

More specifically, the errors are:

[ 49%] Building CXX object src/CMakeFiles/zorba_simplestore.dir/compiler/parser/xquery_scanner.cpp.o
xquery_scanner.yy.cpp:6108:21: error: out-of-line definition of 'LexerInput' does not match any declaration in 'ZorbaFlexLexer'size_t yyFlexLexer::LexerInput( char* buf, size_t max_size )
                    ^~~~~~~~~~
xquery_scanner.yy.cpp:6135:19: error: out-of-line definition of 'LexerOutput' does not match any declaration in 'ZorbaFlexLexer'
void yyFlexLexer::LexerOutput( const char* buf, size_t size )
                  ^~~~~~~~~~~2 errors generated.

[Paul J. Lucas (paul-lucas)]

Even though the bzr repository contains a pre-build scanner and parser, for some reason, the build process regenerates the scanner:

[ 27%] Generating compiler/parser/xquery_parser.cpp, compiler/parser/xquery_parser.hpp, compiler/parser/location.hh, compiler/parser/position.hh, compiler/parser/stack.hh
[ 27%] Generating compiler/parser/xquery_scanner.cpp

Did somebody change how the build system "knows" whether the files ought to be generated?

[Paul J. Lucas (paul-lucas)]

A temporary work-around is to install flex from MacPorts, then:

$ cd /usr/bin
$ sudo mv flex flex.orig

The build then succeeds.

[Paul J. Lucas (paul-lucas)]

Part of the problem is that the file FlexLexer.h is copied by the copyparser.sh script form /usr/include to src/compiler/parser. However, if for whatever reason the xquery_scanner.cpp is regenerated, it's of course using the currently installed flex. In Mountain Lion's case, that flex generates a .cpp file with LexerInput() and LexerOutput() taking size_t arguments that do NOT match the declarations in src/compiler/parser/FlexLexer.h.

Hence, I think the fix is that if the xquery_scanner.cpp is regenerated, the FlexLexer.h file should be copied from /usr/include to build/src/compiler/parser so it will be used instead of the one that's bundled in src/compiler/parser.

Since this is more of a cmake fix, I'm reassigning.

[Paul J. Lucas (paul-lucas)]

BTW: I don't think this bug has anything to do with bison. It's not a bison/flex incompatibility. It's a flex .h/.cpp incompatibility due to differing versions of flex for each.

[Paul J. Lucas (paul-lucas)]

BTW2: I don't think this has anything to do with clang, either.

[Chris Hillery (ceejatec)]

If your diagnosis is correct, then it only has anything to do with clang inasmuch as only clang seems to complain about the problem.

[Paul J. Lucas (paul-lucas)]

I think any compiler would complain about the problem since the declaration uses int and the definition uses size_t (or vice versa).

[Chris Hillery (ceejatec)]

s/complain/throw an error/, then. The build completes with gcc.

[Chris Hillery (ceejatec)]

Ok, based on Paul's diagnosis, I've added a step to the flex generation CMake macro that copies the flex header to CMAKE_BINARY_DIR/src. I chose that directory for two reasons:

1. It's already the first -I/dir listed on the compile line.
2. This file will be shared by all flex'd input files (currently only one, but maybe in future...)

I also updated the copyparser.sh script to copy it from there, rather than hard-coded /usr/include.

This SHOULD ensure that the generated lexer .cpp file is always in sync with the FlexLexer.h it includes.

FYI, I couldn't easily reproduce the problem on Linux, because even after installing flex 2.5.36 and 2.5.37, FlexLexer.h always had int arguments to LexerInput and LexerOutput, rather than size_t. I can only conclude that the MacPorts version of flex either modifies flex, or sets some option, to make it generate those files with size_t. So I want Paul to try out my fix on a Mac by regenerating the scanner, and confirming that the .cpp files have size_t arguments AND that the copied/included FlexLexer.h also has size_t arguments (and that copyparser.sh copies the correct one into place in src/compiler/parser).

[Ghislain Fourny (gislenius)]

Hi Chris,

I am afraid the build is now broken on my machine. Flex seems to generate all files using "int" on my side, as you describe on Linux, whereas the one coming from /usr/include as well as the one in CMAKE_BINARY_DIR/src have "size_t", which causes the exact opposite error:

xquery_scanner.yy.cpp:6109:18: error: out-of-line definition of 'LexerInput' does not match any declaration in 'ZorbaFlexLexer'
int yyFlexLexer::LexerInput( char* buf, int max_size )

xquery_scanner.yy.cpp:6136:19: error: out-of-line definition of 'LexerOutput' does not match any declaration in 'ZorbaFlexLexer'
void yyFlexLexer::LexerOutput( const char* buf, int size )

[Ghislain Fourny (gislenius)]

Uninstall MacPort's flex (2.5.36) (using the system's flex instead, 2.5.35) seems to have fixed the issue. size_t is now generated.

[Ghislain Fourny (gislenius)]

s/Uninstall/Uninstalling/