zorba has severe issues when compiled with GCC 4.9.0

[Martin Gieseking]

I currently try to prepare Zorba for Fedora 21 which requires to build all packages with GCC 4.9.0. The code compiles without errors but the resulting binary has severe issues when one of the optimization options -O1, -O2, or -O3 are applied. It segfaults on any query due to multiple deallocation of the same memory. Initially, I thought that there must be a regression in GCC's optimizer but after analyzing the issue, I guess that some of Zorba's utility classes are the origin.

I tracked down the segfault for the simple query "1" (see attached patch) but there are still a lot more issues in the code that lead to similar problems. Since I don't know the implementation details of Zorba, it would be great if one of the Zorba experts could dig into code and help to fix it.

Here's the output of GCC's address sanitizer written when Zorba executes the bundled test query test/rbkt/Queries/zorba/schemas/val-inplace2.xq:

==17767==ERROR: AddressSanitizer: heap-use-after-free on address 0x61400000aa50 at pc 0x7f3caa205da9 bp 0x7fff38c445e0 sp 0x7fff38c445d0
READ of size 8 at 0x61400000aa50 thread T0
    #0 0x7f3caa205da8 in zorba::store::Item::getKind() const /builddir/build/BUILD/zorba-3.0/src/store/api/item.h:125
    #1 0x7f3caa205da8 in zorba::store::Item::removeReference() /builddir/build/BUILD/zorba-3.0/src/store/naive/item.cpp:124
    #2 0x7f3ca97d6e50 in ~ItemHandle /builddir/build/BUILD/zorba-3.0/src/store/api/item_handle.h:62
    #3 0x7f3ca97d6e50 in ~ForVarState /builddir/build/BUILD/zorba-3.0/src/runtime/core/var_iterators.h:315
    #4 0x7f3ca97d6e50 in zorba::StateTraitsImpl<zorba::ForVarState>::destroyState(zorba::PlanState&, unsigned int) /builddir/build/BUILD/zorba-3.0/src/runtime/base/plan_iterator.h:265
    #5 0x7f3ca97d6e50 in zorba::NoaryBaseIterator<zorba::ForVarIterator, zorba::ForVarState>::closeImpl(zorba::PlanState&) /builddir/build/BUILD/zorba-3.0/src/runtime/base/noarybase.h:109
    #6 0x7f3ca95f091e in zorba::PlanIterator::close(zorba::PlanState&) /builddir/build/BUILD/zorba-3.0/src/runtime/base/plan_iterator.h:390
    #7 0x7f3ca95f091e in zorba::UnaryBaseIterator<zorba::ApplyIterator, zorba::ApplyIteratorState>::closeImpl(zorba::PlanState&) /builddir/build/BUILD/zorba-3.0/src/runtime/base/unarybase.h:125
    #8 0x7f3ca9a045c5 in zorba::PlanIterator::close(zorba::PlanState&) /builddir/build/BUILD/zorba-3.0/src/runtime/base/plan_iterator.h:390
    #9 0x7f3ca9a045c5 in zorba::TransformIterator::closeImpl(zorba::PlanState&) /builddir/build/BUILD/zorba-3.0/src/runtime/update/update.cpp:956
    #10 0x7f3ca95a8c0b in zorba::PlanIterator::close(zorba::PlanState&) /builddir/build/BUILD/zorba-3.0/src/runtime/base/plan_iterator.h:390
    #11 0x7f3ca95a8c0b in zorba::PlanWrapper::close() /builddir/build/BUILD/zorba-3.0/src/runtime/api/plan_wrapper.cpp:192
    #12 0x7f3ca7be3904 in zorba::XQueryImpl::execute(std::ostream&, Zorba_SerializerOptions const*) /builddir/build/BUILD/zorba-3.0/src/api/xqueryimpl.cpp:1159
    #13 0x417ed9 in compileAndExecute(zorba::Zorba*, zorba::XmlDataManager*, ZorbaCMDProperties const&, zorba::SmartPtr<zorba::StaticContext>&, std::string const&, std::istream&, std::ostream&, TimingInfo&) /builddir/build/BUILD/zorba-3.0/bin/zorbacmd.cpp:871
    #14 0x4088fb in main /builddir/build/BUILD/zorba-3.0/bin/zorbacmd.cpp:1204
    #15 0x7f3c9e1960bf in __libc_start_main (/lib64/libc.so.6+0x200bf)
    #16 0x40cdf6 (/builddir/build/BUILD/zorba-3.0/build/bin/zorba+0x40cdf6)

0x61400000aa50 is located 16 bytes inside of 432-byte region [0x61400000aa40,0x61400000abf0)
freed by thread T0 here:
    #0 0x7f3cb0e45a1f in operator delete(void*) (/lib64/libasan.so.1+0x58a1f)
    #1 0x7f3ca97d6e50 in ~ItemHandle /builddir/build/BUILD/zorba-3.0/src/store/api/item_handle.h:62
    #2 0x7f3ca97d6e50 in ~ForVarState /builddir/build/BUILD/zorba-3.0/src/runtime/core/var_iterators.h:315
    #3 0x7f3ca97d6e50 in zorba::StateTraitsImpl<zorba::ForVarState>::destroyState(zorba::PlanState&, unsigned int) /builddir/build/BUILD/zorba-3.0/src/runtime/base/plan_iterator.h:265
    #4 0x7f3ca97d6e50 in zorba::NoaryBaseIterator<zorba::ForVarIterator, zorba::ForVarState>::closeImpl(zorba::PlanState&) /builddir/build/BUILD/zorba-3.0/src/runtime/base/noarybase.h:109

previously allocated by thread T0 here:
    #0 0x7f3cb0e4551f in operator new(unsigned long) (/lib64/libasan.so.1+0x5851f)
    #1 0x7f3caa4fabcf in zorba::simplestore::BasicItemFactory::createPendingUpdateList() /builddir/build/BUILD/zorba-3.0/src/store/naive/simple_item_factory.cpp:2151

SUMMARY: AddressSanitizer: heap-use-after-free /builddir/build/BUILD/zorba-3.0/src/store/api/item.h:125 zorba::store::Item::getKind() const

[Paul J. Lucas (paul-lucas)]

Thanks for your good patch. We've applied it. As for the rest, I am able to reproduce a similar problem using Mac OS X and a MacPorts prerelease version of gcc 4.9. However, the core dump it produces isn't useful: gdb complains that it can't unwind the stack past a certain point. It doesn't actually matter whether any -O option is given: it fails regardless.

I also tried gcc 4.8 and it runs fine under Linux (Ubuntu), but also fails on Mac. Ubuntu currently doesn't have a gcc 4.9 available. We could try compiling it from source, but that will take some effort.

[Martin Gieseking (martin-gieseking)]

Paul, thanks for your confirmation and for applying the patch.
If there are issues with the core dump, you could try to build Zorba with the gcc options -fsanitize=address and -fsanitize=undefined. That leads to a detailed crash report if the program terminates unexpectedly. Alternatively, valgrind could also be an option to get more information about the context of the segfault.

I currently don't have a Ubuntu machine available but it seems you can get pre-built packages of gcc 4.9.0 here:
https://launchpad.net/~ubuntu-toolchain-r/+archive/test
http://ppa.launchpad.net/ubuntu-toolchain-r/test/ubuntu/pool/main/g/gcc-4.9

[Martin Gieseking (martin-gieseking)]

Paul, did you already have the time to look into this bug? I tried to locate the issue but don't quite understand the implementation concepts of the zstring/rstring classes. Their memory management and those of the derived/specialized classes seems to cause the segfaults. Unfortunately, I don't have enough time to dig deeper into the code.

Here is the crash report of the query file test/rbkt/Queries/zorba/schemas/val-bad-attr.xq.

==29261==ERROR: AddressSanitizer: heap-use-after-free on address 0x60b00004dbf0 at pc 0x7fbc111aab8f bp 0x7fffe3b466b0 sp 0x7fffe3b466a0
READ of size 4 at 0x60b00004dbf0 thread T0
    #0 0x7fbc111aab8e in zorba::atomic_int::load_impl() const /builddir/build/BUILD/zorba-3.0/src/util/atomic_int.h:173
    #1 0x7fbc111aab8e in zorba::atomic_int::load() const /builddir/build/BUILD/zorba-3.0/src/util/atomic_int.h:100
    #2 0x7fbc111aab8e in zorba::rstring_classes::rep_base<zorba::atomic_int, std::char_traits<char>, std::allocator<char> >::is_sharable() /builddir/build/BUILD/zorba-3.0/src/util/string/rep_base.h:216
    #3 0x7fbc111aab8e in zorba::rstring_classes::rep_proxy<zorba::rstring_classes::rep<zorba::atomic_int, std::char_traits<char>, std::allocator<char> > >::share(zorba::rstring_classes::rep_proxy<zorba::rstring_classes::rep<zorba::atomic_int, std::char_traits<char>, std::allocator<char> > > const&, std::allocator<char> const&, std::allocator<char> const&) /builddir/build/BUILD/zorba-3.0/src/util/string/rep_proxy.h:120
    #4 0x7fbc111aab8e in zorba::rstring<zorba::rstring_classes::rep<zorba::atomic_int, std::char_traits<char>, std::allocator<char> > >::assign(zorba::rstring<zorba::rstring_classes::rep<zorba::atomic_int, std::char_traits<char>, std::allocator<char> > > const&) /builddir/build/BUILD/zorba-3.0/src/util/string/rstring.tcc:121
    #5 0x7fbc1242da8d in zorba::simplestore::XmlNode::getBaseURI(zorba::rstring<zorba::rstring_classes::rep<zorba::atomic_int, std::char_traits<char>, std::allocator<char> > >&) const /builddir/build/BUILD/zorba-3.0/src/store/naive/node_items.h:496
    #6 0x7fbc12193ea1 in zorba::Validator::realValidationValue(zorba::store::ItemHandle<zorba::store::Item>&, zorba::store::ItemHandle<zorba::store::Item> const&, zorba::store::ItemHandle<zorba::store::Item> const&, zorba::TypeManager*, zorba::ParseConstants::validation_mode_t, zorba::static_context const*, zorba::QueryLoc const&) /builddir/build/BUILD/zorba-3.0/src/types/schema/validate.cpp:230
    #7 0x7fbc121958a8 in zorba::Validator::effectiveValidationValue(zorba::store::ItemHandle<zorba::store::Item>&, zorba::store::ItemHandle<zorba::store::Item> const&, zorba::store::ItemHandle<zorba::store::Item> const&, zorba::TypeManager*, zorba::ParseConstants::validation_mode_t, zorba::static_context const*, zorba::QueryLoc const&) /builddir/build/BUILD/zorba-3.0/src/types/schema/validate.cpp:77
    #8 0x7fbc11c7b87f in zorba::ValidateIterator::nextImpl(zorba::store::ItemHandle<zorba::store::Item>&, zorba::PlanState&) const /builddir/build/BUILD/zorba-3.0/src/runtime/schema/schema_impl.cpp:63
    #9 0x7fbc11ea813b in zorba::PlanIterator::produceNext(zorba::store::ItemHandle<zorba::store::Item>&, zorba::PlanState&) const /builddir/build...

[Martin Gieseking (martin-gieseking)]

Just a short addition: I was able to reproduce the bug on Ubuntu 14.04 using the GCC 4.9 packages from the PPA mentioned in comment #3.

[Paul J. Lucas (paul-lucas)]

We don't have a Fedora machine. We have an Ubuntu machine and gcc 4.9 is not available on it. I've tried compiling it from source without success.

- Paul

[Martin Gieseking (martin-gieseking)]

As mentioned above, there are packages of gcc 4.9 available for recent Ubuntu releases. They can be installed pretty easily and work fine as far as I can tell from my initial tests.
It would be great if anyone of you Zorba experts could help to fix the issue. Otherwise, I probably have to drop the Zorba package from Fedora since I'm unable to track down the buggy code in a timely manner (Fedora 21 is to be released soon).

[Paul J. Lucas (paul-lucas)]

OK, I've updated our machine so we can get gcc 4.9... Working on it now.

- Paul

[Paul J. Lucas (paul-lucas)]

> Initially, I thought that there must be a regression in GCC's optimizer
> but after analyzing the issue, I guess that some of Zorba's utility classes are the origin.

For the remaining issue, can you elaborate your analysis to arrive at your conclusion that the issue is not with gcc's optimizer?

[Martin Gieseking (martin-gieseking)]

Thank you for taking the time to investigate the bug.

> For the remaining issue, can you elaborate your analysis to
> arrive at your conclusion that the issue is not with gcc's optimizer?

Actually, I'm not sure if the code of the utility classes or gcc causes the issue. After recompiling Zorba with gcc 4.9 a few weeks ago, I thought there must be a bug in gcc's optimizer because the binary generated with gcc 4.8.2 works correctly. But then I found the reason for the initial segfault in Zorba's code and thus thought the remaining memory issue(s) might also be caused by buggy code. However, after some hours of debugging I'm unable to find the link between the deallocation of and the following access to the concerning memory block of the heap. They seem to be quite unrelated. But since I don't know all the implementation details of the utility classes, I can't exclude that there might be some memory to be shared by several instances of a class.
If it's really a bug in gcc's optimizer, I guess it's necessary to derive a minimal code example that demonstrates the issue so that the gcc folks can investigate it further.

[Paul J. Lucas (paul-lucas)]

I looked through the code as well and it's not obvious to me about the use of the memory after the deallocation either.

How did you discover that it's the val-inplace2.xq test that manifests this problem? Are there other tests that manifest this problem? Whittling down the code to a minimal example would be a non-trivial task.

[Martin Gieseking (martin-gieseking)]

> How did you discover that it's the val-inplace2.xq test that manifests
> this problem? Are there other tests that manifest this problem?

I simply ran zorba on all .xq files located somewhere inside or below the test folder:
  for f in $(find ../../test -name *.xq); do
    ./zorba $f >/dev/null 2>&1
    echo "$?: $f"
  done

The following test files make zorba crash. The leading numbers are the corresponding error codes (134=abortion with core dump, 139=segmentation fault):

139: test/update/Queries/zorba/store/sc1_ex3.xq
134: test/update/Queries/zorba/transform1.xq
134: test/update/Queries/zorba/jimbug.xq
134: test/rbkt/Queries/zorba/copytransform/copytransform01.xq
134: test/rbkt/Queries/zorba/updates/transform01.xq
134: test/rbkt/Queries/zorba/updates/upd4.xq
139: test/rbkt/Queries/zorba/misc/misc5.xq
134: test/rbkt/Queries/zorba/misc/baseuri.xq
134: test/rbkt/Queries/zorba/parsing_and_serializing/parse-fragment-skip-root-17.xq
134: test/rbkt/Queries/zorba/parsing_and_serializing/parse-fragment-skip-root-38.xq
134: test/rbkt/Queries/zorba/parsing_and_serializing/parse-fragment-doctype-01.xq
134: test/rbkt/Queries/zorba/parsing_and_serializing/parse-fragment-skip-root-56.xq
134: test/rbkt/Queries/zorba/parsing_and_serializing/parse-fragment-skip-root-23.xq
134: test/rbkt/Queries/zorba/parsing_and_serializing/parse-xml-fragment-20.xq
134: test/rbkt/Queries/zorba/parsing_and_serializing/parse-fragment-skip-root-29.xq
134: test/rbkt/Queries/zorba/parsing_and_serializing/parse-fragment-skip-root-19.xq
134: test/rbkt/Queries/zorba/parsing_and_serializing/parse-fragment-skip-root-08.xq
134: test/rbkt/Queries/zorba/parsing_and_serializing/parse-fragment-skip-root-52.xq
134: test/rbkt/Queries/zorba/parsing_and_serializing/parse-xml-fragment-03.xq
134: test/rbkt/Queries/zorba/parsing_and_serializing/parse-xml-fragment-26.xq
134: test/rbkt/Queries/zorba/parsing_and_serializing/parse-fragment-skip-root-50.xq
134: test/rbkt/Queries/zorba/parsing_and_serializing/fn-parse-xml-fragment-02.xq
134: test/rbkt/Queries/zorba/parsing_and_serializing/parse-fragment-skip-root-04.xq
134: test/rbkt/Queries/zorba/parsing_and_serializing/parse-fragment-doctype-11.xq
134: test/rbkt/Queries/zorba/parsing_and_serializing/parse-xml-fragment-06.xq
134: test/rbkt/Queries/zorba/parsing_and_serializing/parse-xml-fragment-21.xq
134: test/rbkt/Queries/zorba/parsing_and_serializing/parse-fragment-skip-root-10.xq
134: test/rbkt/Queries/zorba/parsing_and_serializing/parse-fragment-skip-root-18.xq
134: test/rbkt/Queries/zorba/parsing_and_serializing/parse-fragment-skip-root-09.xq
134: test/rbkt/Queries/zorba/parsing_and_serializing/parse-fragment-skip-root-22.xq
139: test/rbkt/Queries/zorba/parsing_and_serializing/parse-xml-fragment-13.xq
134: test/rbkt/Queries/zorba/parsing_and_serializing/fn-parse-xml-fragment-23.xq
134: test/rbkt/Queries/zorba/parsing_and_serializing/parse-fragment-skip-root-54.xq
134: test/rbkt/Queries/zorba/parsing_and_serializing/fn-parse-xml-fragment-16.xq
134: test/rbkt/Queries/zorba/parsing_and_serializing/parse-fragment-skip-root-47.xq
134: test/rbkt/Queries/zorba/parsing_and_serializing/parse-xml-fragment-24.xq
134: test/rbkt/Queries/zorba/parsing...

[Paul J. Lucas (paul-lucas)]

Sorry for the delay... other things came up. I think I've made some progress. Try the code at: https://github.com/28msec/zorba/tree/gcc-4.9

[Martin Gieseking (martin-gieseking)]

Thank you again for looking into this. I really appreciate your help.

I just tried to build the sources from the gcc-4.9 branch. I'm sorry to tell that the resulting binary segfaults or aborts abnormally on more test queries than zorba 3.0.

Here is the valgrind output for val-inplace2.xq2.xq:

$ valgrind ./zorba ../../test/rbkt/Queries/zorba/schemas/val-inplace2.xq2.xq
==24657== Memcheck, a memory error detector
==24657== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==24657== Using Valgrind-3.10.0.SVN and LibVEX; rerun with -h for copyright info
==24657== Command: ./zorba ../../test/rbkt/Queries/zorba/schemas/val-inplace2.xq
==24657==
<?xml version="1.0" encoding="UTF-8"?>
==24657== Invalid read of size 8
==24657== at 0x57E77A4: zorba::store::Item::removeReference() (item.h:125)
==24657== by 0x55BD903: zorba::ForVarState::~ForVarState() (item_handle.h:62)
==24657== by 0x55C0F0E: zorba::NoaryBaseIterator<zorba::ForVarIterator, zorba::ForVarState>::closeImpl(zorba::PlanState&) (plan_iterator.h:314)
==24657== by 0x5564FA7: zorba::UnaryBaseIterator<zorba::ApplyIterator, zorba::ApplyIteratorState>::closeImpl(zorba::PlanState&) (plan_iterator.h:441)
==24657== by 0x563AB09: zorba::TransformIterator::closeImpl(zorba::PlanState&) (plan_iterator.h:441)
==24657== by 0x5556B54: zorba::PlanWrapper::close() (plan_iterator.h:441)
==24657== by 0x505D731: zorba::XQueryImpl::dispose(zorba::rchandle<zorba::PlanWrapper> const&) (xqueryimpl.cpp:726)
==24657== by 0x50627E4: zorba::XQueryImpl::execute(std::ostream&, Zorba_SerializerOptions const*) (xqueryimpl.cpp:1145)
==24657== by 0x40D89B: compileAndExecute(zorba::Zorba*, zorba::SmartPtr<zorba::StaticContext>&, std::string const&, std::istream&, std::ostream&, Timers&) (zorbacmd.cpp:525)
==24657== by 0x40FD4A: main (zorbacmd.cpp:824)
==24657== Address 0xa9a90c0 is 16 bytes inside a block of size 432 free'd
==24657== at 0x4C2D103: operator delete(void*) (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==24657== by 0x58641F2: zorba::simplestore::PULImpl::~PULImpl() (simple_pul.cpp:152)
==24657== by 0x57E76C0: zorba::store::Item::free() (item.cpp:45)
==24657== by 0x57E7877: zorba::store::Item::removeReference() (item.cpp:207)
==24657== by 0x55BD903: zorba::ForVarState::~ForVarState() (item_handle.h:62)
==24657== by 0x55C0F0E: zorba::NoaryBaseIterator<zorba::ForVarIterator, zorba::ForVarState>::closeImpl(zorba::PlanState&) (plan_iterator.h:314)
==24657== by 0x563AAFC: zorba::TransformIterator::closeImpl(zorba::PlanState&) (plan_iterator.h:441)
==24657== by 0x5556B54: zorba::PlanWrapper::close() (plan_iterator.h:441)
==24657== by 0x505D731: zorba::XQueryImpl::dispose(zorba::rchandle<zorba::PlanWrapper> const&) (xqueryimpl.cpp:726)
==24657== by 0x50627E4: zorba::XQueryImpl::execute(std::ostream&, Zorba_SerializerOptions const*) (xqueryimpl.cpp:1145)
==24657== by 0x40D89B: compileAndExecute(zorba::Zorba*, zorba::SmartPtr<zorba::StaticContext>&, std::string const&, std::istream&, std::ostream&, Timers&) (zorbacmd.cpp:525)
==24657== by 0x40FD4A: main (zorbacmd.cpp:824)
==24657==
==24657== Invalid read of size 8
==24657== ...

[Paul J. Lucas (paul-lucas)]

I build using gcc-4.9 with the following options to cmake:

-G"Unix Makefiles"
-DCMAKE_BUILD_TYPE=Debug
-DCMAKE_C_FLAGS="-O1 -g -fsanitize=address -fsanitize=undefined -fno-omit-frame-pointer"
-DCMAKE_CXX_FLAGS="-O1 -g -fsanitize=address -fsanitize=undefined -fno-omit-frame-pointer"
-DCMAKE_SHARED_LINKER_FLAGS="-Wl,-fsanitize=address,-fsanitize=undefined"

and when I run the exact command, it runs without error. If I rebuild with:

-G"Unix Makefiles"
-DCMAKE_BUILD_TYPE=Debug
-DCMAKE_C_FLAGS="-O1 -g"
-DCMAKE_CXX_FLAGS="-O1 -g"

and use valgrind, it produces no error output.

[Martin Gieseking (martin-gieseking)]

That's strange. Does the GitHub repo contain all the latest changesets? Did you build on Ubuntu with the above mentioned gcc-4.9 packages?
I also just rebuilt everything on Ubuntu 14.04 using the same cmake flags plus -DBISON_EXECUTABLE=/dummy in order to prevent running bison because bison 3.0 fails to process parser.y.

The resulting zorba binary still crashes when processing val-inplace2.xq. However, the valgrind output is different from those of the Fedora build:

$ valgrind ./zorba ../../test/rbkt/Queries/zorba/schemas/val-inplace2.xq
==29363== Memcheck, a memory error detector
==29363== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==29363== Using Valgrind-3.10.0.SVN and LibVEX; rerun with -h for copyright info
==29363== Command: ./zorba ../../test/rbkt/Queries/zorba/schemas/val-inplace2.xq
==29363==
==29363== Conditional jump or move depends on uninitialised value(s)
==29363== at 0x42D28B6: zorba::TranslatorImpl::check_boundary_whitespace(zorba::DirElemContent const&) (rchandle.h:214)
==29363== by 0x425FBC5: zorba::DirElemContentList::accept(zorba::parsenode_visitor&) const (parsenodes.cpp:3736)
==29363== by 0x425C3B8: zorba::DirElemConstructor::accept(zorba::parsenode_visitor&) const (parsenodes.cpp:3705)
==29363== by 0x425D06D: zorba::VarBinding::accept(zorba::parsenode_visitor&) const (parsenodes.cpp:5085)
==29363== by 0x425F571: zorba::CopyVarList::accept(zorba::parsenode_visitor&) const (parsenodes.cpp:5064)
==29363== by 0x425CFF4: zorba::TransformExpr::accept(zorba::parsenode_visitor&) const (parsenodes.cpp:5041)
==29363== by 0x425AA75: zorba::QueryBody::accept(zorba::parsenode_visitor&) const (parsenodes.cpp:1174)
==29363== by 0x425A152: zorba::MainModule::accept(zorba::parsenode_visitor&) const (parsenodes.cpp:147)
==29363== by 0x427FF36: zorba::translate_aux(zorba::TranslatorImpl*, zorba::parsenode const&, zorba::static_context*, unsigned int, zorba::ModulesInfo*, std::map<zorba::rstring<zorba::rstring_classes::rep<zorba::atomic_int, std::char_traits<char>, std::allocator<char> > >, zorba::rstring<zorba::rstring_classes::rep<zorba::atomic_int, std::char_traits<char>, std::allocator<char> > >, std::less<zorba::rstring<zorba::rstring_classes::rep<zorba::atomic_int, std::char_traits<char>, std::allocator<char> > > >, std::allocator<std::pair<zorba::rstring<zorba::rstring_classes::rep<zorba::atomic_int, std::char_traits<char>, std::allocator<char> > > const, zorba::rstring<zorba::rstring_classes::rep<zorba::atomic_int, std::char_traits<char>, std::allocator<char> > > > > > const&, bool, zorba::StaticContextConsts::xquery_version_t) (translator.cpp:16432)
==29363== by 0x42803AF: zorba::translate(zorba::parsenode const&, zorba::CompilerCB*) (translator.cpp:16463)
==29363== by 0x43A2FF2: zorba::XQueryCompiler::translate(zorba::rchandle<zorba::parsenode>, zorba::audit::ScopedRecord&) (compiler_api.cpp:359)
==29363== by 0x43A3646: zorba::XQueryCompiler::compile(zorba::rchandle<zorba::parsenode> const&, bool, unsigned long&, zorba::audit::ScopedRecord&) (compiler_api.cpp:335)
==29363==
==29363== Conditional jump or move depends on uninitialised value(s)
==29363== at 0x42D28B6: ...

[Paul J. Lucas (paul-lucas)]

You did remember to "git checkout gcc-4.9" before building, right?

[Martin Gieseking (martin-gieseking)]

Yes, I did. :) It's actually the build from the gcc-4.9 branch that segfaults for me on Fedora and Ubuntu.

[Paul J. Lucas (paul-lucas)]

I just merged other fixes into the gcc-4.9 branch. Can you try again, please?

[Martin Gieseking (martin-gieseking)]

OK, done. Unfortunately, I get the same results. On Ubuntu, valgrind reports the same problems as shown in comment #16.
I'm still wondering why these issues don't occur in your testing environment. Do you use Ubuntu 14.04 and gcc 4.9-20140406-1ubuntu1, too?

[Paul J. Lucas (paul-lucas)]

If I do:

$ uname -a; gcc --version

I get:

Linux sanjose 3.13.0-27-generic #50-Ubuntu SMP Thu May 15 18:06:16 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
gcc (Ubuntu 4.9-20140406-1ubuntu1) 4.9.0 20140405 (experimental) [trunk revision 209157]

I configure with:

cmake -G"Unix Makefiles" -DCMAKE_BUILD_TYPE=Debug -DCMAKE_C_FLAGS="-O1" -DCMAKE_CXX_FLAGS="-O1" ..

I *used* to get the same errors you are seeing, but now I don't.

[Martin Gieseking (martin-gieseking)]

I think, I'm stuck here. The zorba binary I get when using exactly the same configuration options is still buggy.
Does "make doc" also work correctly in your Ubuntu environment? I get a segmentation fault here as well.

Here is my output of cmake which looks fine as far as I can tell:

-- Looking for LibXml2
-- Found LIBXML2 library -- /usr/lib/i386-linux-gnu/libxml2.so
--
-- Looking for ICU
-- Found ICU library -- /usr/lib/i386-linux-gnu/libicuuc.so
-- ICU Version: 52.1.0
-- Found ICU internationalization library -- /usr/lib/i386-linux-gnu/libicui18n.so
-- Found ICU data library -- /usr/lib/i386-linux-gnu/libicudata.so
-- Found ICU library -- /usr/lib/i386-linux-gnu/libicuuc.so
--
-- Looking for Xerces-C
--
-- Executable: PHP5_EXECUTABLE-NOTFOUND
-- Library: PHP5_LIBRARY-NOTFOUND
-- *NIX Environment Configuration...
-- Could NOT find PythonLibs (missing: PYTHON_LIBRARIES PYTHON_INCLUDE_DIRS)
-- ZORBA_NATIVE_STORE: ON
-- ZORBA_STORE_DIR: /home/mgieseki/zorba/src/store/naive
-- ZORBA_STORE_NAME: simplestore
-- ZORBA_BATCHING_TYPE: 0
-- ZORBA_FLOAT_POINT_PRECISION: 18
-- ZORBA_WITH_CODE_COVERAGE: OFF
-- ZORBA_WITH_CODE_PROFILE: OFF
-- ZORBA_FOR_ONE_THREAD_ONLY: ON
-- ZORBA_WITH_FILE_ACCESS [ON/OFF]: ON
-- ZORBA_WITH_DEBUGGER [ON/OFF]: ON
-- ZORBA_TEST_TIMEOUT_VALUE: 60
-- ZORBA_VERIFY_PEER_SSL_CERTIFICATE: OFF
-- ZORBA_WIN_DLL: ON
-- ZORBA_EXE: /home/mgieseki/zorba/build/bin/zorba
-- ZORBA_DEBUG_PARSER: OFF
-- ZORBA_DEBUG_STRING: ON
-- ZORBA_NO_ICU: OFF
-- ZORBA_NO_FULL_TEXT: OFF
-- ZORBA_NO_XMLSCHEMA: OFF
-- ZORBA_WITH_BIG_INTEGER: OFF
-- ZORBA_WITH_THESAURUS: ON
-- ZORBA_SUPPRESS_SWIG [ON/OFF]: OFF
-- ZORBA_XQUERYX: OFF
-- ZORBA_TEST_XQUERYX: OFF
-- CMAKE_BINARY_DIR: /home/mgieseki/zorba/build
-- CMAKE_SOURCE_DIR: /home/mgieseki/zorba
-- CMAKE_SYSTEM: Linux-3.13.0-29-generic
-- CMAKE_SYSTEM_NAME: Linux
-- CMAKE_SYSTEM_VERSION: 3.13.0-29-generic
-- CMAKE_SYSTEM_PROCESSOR: i686
-- UNIX: 1
-- WIN32:
-- APPLE:
-- MINGW:
-- CYGWIN:
-- BORLAND:
-- MSVC:
-- MSVC_IDE:
-- MSVC60:
-- MSVC70:
-- MSVC71:
-- MSVC80:
-- CMAKE_COMPILER_2005:
-- CMAKE_C_FLAGS: -O1
-- CMAKE_CXX_FLAGS: -O1
-- CMAKE_BUILD_TYPE: Debug
-- CMAKE_INSTALL_PREFIX: /home/mgieseki/zorba/build/dist
-- BUILD_SHARED_LIBS:
--...

[Paul J. Lucas (paul-lucas)]

Here is my cmake output:

-- The C compiler identification is GNU 4.9.0
-- The CXX compiler identification is Clang 3.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check if the system is big endian
-- Searching 16 bit integer
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for stddef.h
-- Looking for stddef.h - found
-- Check size of unsigned short
-- Check size of unsigned short - done
-- Using unsigned short
-- Check if the system is big endian - little endian
-- Looking for include file unicode/coll.h
-- Looking for include file unicode/coll.h - not found
-- Looking for include file iconv.h
-- Looking for include file iconv.h - found
-- Looking for include file inttypes.h
-- Looking for include file inttypes.h - found
-- Looking for include file limits.h
-- Looking for include file limits.h - found
-- Looking for include file stdint.h
-- Looking for include file stdint.h - found
-- Looking for include file stdlib.h
-- Looking for include file stdlib.h - found
-- Looking for include files sys/param.h, sys/mount.h
-- Looking for include files sys/param.h, sys/mount.h - found
-- Looking for include file sys/types.h
-- Looking for include file sys/types.h - found
-- Looking for include file tzfile.h
-- Looking for include file tzfile.h - not found
-- Looking for include file unicode/ustring.h
-- Looking for include file unicode/ustring.h - found
-- Looking for include file unicode/utypes.h
-- Looking for include file unicode/utypes.h - found
-- Looking for include file execinfo.h
-- Looking for include file execinfo.h - found
-- Looking for C++ include FlexLexer.h
-- Looking for C++ include FlexLexer.h - found
-- Looking for include file uuid/uuid.h
-- Looking for include file uuid/uuid.h - found
-- Looking for strtof
-- Looking for strtof - found
-- Looking for strtoll
-- Looking for strtoll - found
-- Looking for _stricmp
-- Looking for _stricmp - not found
-- Looking for strcat_s
-- Looking for strcat_s - not found
-- Looking for strcpy_s
-- Looking for strcpy_s - not found
-- Looking for include file pthread.h
-- Looking for include file pthread.h - found
-- Looking for pthread_spin_init
-- Looking for pthread_spin_init - found
-- Looking for pthread_mutex_init
-- Looking for pthread_mutex_init - found
-- Looking for getrusage
-- Looking for getrusage - found
-- Check size of int32_t
-- Check size of int32_t - done
-- Check size of uint32_t
-- Check size of uint32_t - done
-- Check size of unsigned __int32
-- Check size of unsigned __int32 - failed
-- Check size of __int32
-- Check size of __int32 - failed
-- Check size of int64_t
-- Check size of int64_t - done
-- Performing Test ZORBA_HAVE_STRUCT_TM_TM_GMTOFF
-- Performing Test ZORBA_HAVE_STRUCT_TM_TM_GMTOFF - Success
-- Performing Test ZORBA_HAVE_STRUCT_TM___TM_GMTOFF
-- Performing Test ZORBA_HAVE_...

[Paul J. Lucas (paul-lucas)]

Never mind. I just noticed that it was picking up clang++ and not g++. clang++ was installed earlier this week. Now I get the same results you do. Sorry. More investigation.

[Paul J. Lucas (paul-lucas)]

Thanks for your patience. Try doing a "git pull" from the gcc-4.9 branch and try it now.

[Martin Gieseking (martin-gieseking)]

Yes, that's great! The latest commits fix the problem. Thank you very much for your hard work on this issue. I'm going to rebuild the Fedora packages right now.

[Paul J. Lucas (paul-lucas)]

Actually, we discovered other new failures. Those have since been fixed, however. So re-pull.

[Martin Gieseking (martin-gieseking)]

Done. Unfortunately, with the latest changes , the binary segfaults again when it's built in release mode with option -O2.
These are the most important cmake options applied when building the Fedora packages:

-DCMAKE_BUILD_TYPE=Release
-DCMAKE_C_FLAGS="-O2 -g"
-DCMAKE_CXX_FLAGS="-O2 -g"
-DCMAKE_CXX_FLAGS_RELEASE='-DNDEBUG'
-DCMAKE_C_FLAGS_RELEASE='-DNDEBUG'

[Paul J. Lucas (paul-lucas)]

Are you saying that with the previous changes it did not segfault in release mode with -O2?

[Martin Gieseking (martin-gieseking)]

Yes, exactly. With the changes I pulled this morning (changeset 858e97125d...), the packages built without errors. Especially, "make doc" succeeded which fails now.

Here are the corresponding Fedora builds:
http://koji.fedoraproject.org/koji/taskinfo?taskID=7058112
http://koji.fedoraproject.org/koji/taskinfo?taskID=7059185

[Paul J. Lucas (paul-lucas)]

I just tried building with the options you gave and did a "make doc" and it worked fine. Do you have any other specific queries or tests that fail?

[Martin Gieseking (martin-gieseking)]

I just tried it on Ubuntu again and got the same error:
*** Error in `bin/zorba': free(): corrupted unsorted chunks: 0x0934e278 ***

Here's the complete cmake call I used:
cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_FLAGS="-O2 -g" -DCMAKE_CXX_FLAGS="-O2 -g" -DCMAKE_CXX_FLAGS_RELEASE='-DNDEBUG' -DCMAKE_C_FLAGS_RELEASE='-DNDEBUG' ..

The error occurs with query file test/rbkt/Queries/zorba/schemas/val-inplace2.xq, for example. "make doc" fails in my Ubuntu environment too (same error as mentioned above).

$ valgrind ./zorba ../../test/rbkt/Queries/zorba/schemas/val-inplace2.xq
==30909== Memcheck, a memory error detector
==30909== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==30909== Using Valgrind-3.10.0.SVN and LibVEX; rerun with -h for copyright info
==30909== Command: ./zorba ../../test/rbkt/Queries/zorba/schemas/val-inplace2.xq
==30909==
<?xml version="1.0" encoding="UTF-8"?>
==30909== Invalid read of size 4
==30909== at 0x492E873: zorba::store::Item::removeReference() (item.cpp:109)
==30909== by 0x46E08CC: zorba::NoaryBaseIterator<zorba::ForVarIterator, zorba::ForVarState>::closeImpl(zorba::PlanState&) (item_handle.h:62)
==30909== by 0x467CE43: zorba::UnaryBaseIterator<zorba::ApplyIterator, zorba::ApplyIteratorState>::closeImpl(zorba::PlanState&) (plan_iterator.h:450)
==30909== by 0x4763CEF: zorba::TransformIterator::closeImpl(zorba::PlanState&) (plan_iterator.h:450)
==30909== by 0x466CFA6: zorba::PlanWrapper::close() (plan_iterator.h:450)
==30909== by 0x4137FDE: zorba::XQueryImpl::dispose(zorba::rchandle<zorba::PlanWrapper> const&) (xqueryimpl.cpp:726)
==30909== by 0x413D7CE: zorba::XQueryImpl::execute(std::ostream&, Zorba_SerializerOptions const*) (xqueryimpl.cpp:1145)
==30909== by 0x8057F35: compileAndExecute(zorba::Zorba*, zorba::SmartPtr<zorba::StaticContext>&, std::string const&, std::istream&, std::ostream&, Timers&) (zorbacmd.cpp:525)
==30909== by 0x804D65A: main (zorbacmd.cpp:824)
==30909== Address 0x75b2990 is 8 bytes inside a block of size 216 free'd
==30909== at 0x402B838: operator delete(void*) (in /usr/lib/valgrind/vgpreload_memcheck-x86-linux.so)
==30909== by 0x49B4011: zorba::simplestore::PULImpl::~PULImpl() (simple_pul.cpp:152)
==30909== by 0x492E5D3: zorba::store::Item::free() (item.cpp:49)
==30909== by 0x492E8B6: zorba::store::Item::removeReference() (item.cpp:211)
==30909== by 0x46E08CC: zorba::NoaryBaseIterator<zorba::ForVarIterator, zorba::ForVarState>::closeImpl(zorba::PlanState&) (item_handle.h:62)
==30909== by 0x4763CDF: zorba::TransformIterator::closeImpl(zorba::PlanState&) (plan_iterator.h:450)
==30909== by 0x466CFA6: zorba::PlanWrapper::close() (plan_iterator.h:450)
==30909== by 0x4137FDE: zorba::XQueryImpl::dispose(zorba::rchandle<zorba::PlanWrapper> const&) (xqueryimpl.cpp:726)
==30909== by 0x413D7CE: zorba::XQueryImpl::execute(std::ostream&, Zorba_SerializerOptions const*) (xqueryimpl.cpp:1145)
==30909== by 0x8057F35: compileAndExecute(zorba::Zorba*, zorba::SmartPtr<zorba::StaticContext>&, std::string const&, std::istream&, std::ostream&, Timers&) (zorbacmd.cpp:525)
==30909== by 0x804D65A: main (zorbacmd.cpp:824)
==30909==
==30...

[Paul J. Lucas (paul-lucas)]

OK, try a "git pull" again.

[Martin Gieseking (martin-gieseking)]

That's it. The latest changeset fixes the crash so that the packages can be built successfully. Thanks again for your help.