FTBFS on zesty

[Dimitri John Ledkov]

Hello,

A no change rebuild on zesty FTBFS, is "zesty" not encoded somehow in the test suite exclusions?

Could you please take a look and fix? This is blocking boost1.62 transition.

The following tests FAILED:
   1 - art_extractor (Failed)
   3 - dbus (Failed)
   9 - qml (Failed)
  10 - libthumbnailer-qt (Failed)
  14 - thumbnailer (SEGFAULT)
  15 - thumbnailer-admin (Failed)
Errors while running CTest
Makefile:130: recipe for target 'test' failed
make[2]: *** [test] Error 8

[Michi Henning (michihenning)]

Thanks for that, will fix now.

[Michi Henning (michihenning)]

Fix is in a silo: https://bileto.ubuntu.com/#/ticket/2175

[Michi Henning (michihenning)]

Unfortunately, just suppressing the Qt tests doesn't fix this. It turns out that, on zesty, the tests for all architectures fail. It looks like a problem with taglib. Every test that tries to extract artwork from an ogg video file fails.

There is also a segfault in the thumbnailer test. That failure is either caused by a problem with max_size_in_bytes(), or by a problem in gsettings on zesty. Need to follow up.

There are also a bunch of gstreamer critical messages in the test log:

10: (vs-thumb:9542): GStreamer-CRITICAL **: Registering meta implementation 'GstVideoMeta' without init function
10: thumbnailer-service: [22:50:37.499] "thumbnail: /<<BUILDDIR>>/thumbnailer-2.4+17.04.20161108/tests/media/testvideo.ogg (256,256): 0.832444 [q: 0.000001, d: 0.524390] sec (MISS)"

These are not normal, need to figure out what is going on there.

It doesn't look like this will be a quick fix :-(

[Dimitri John Ledkov (xnox)]

Adding taglib task for inspection, there was a big version bump in taglib in zesty from 1.9 -> 1.11

[Flames_in_Paradise (ellisistfroh-deactivatedaccount)]

@ Dimitri

24th of October the taglib-team released a bugfix version [1]:

TagLib 1.11.1 Release - October 24, 2016

    Fixed binary incompatible change in TagLib::String.
    Fixed reading ID3v2 CTOC frames with a lot of entries.
    Fixed seeking ByteVectorStream from the end.

[1] http://taglib.org/

Probably this may help to fix those issues?

[Dimitri John Ledkov (xnox)]

Ubuntu and Debian are both at 1.11.1 upstream release version.

[Michi Henning (michihenning)]

Taglib 1.11 has broken extraction of VorbisComments.

I've opened a bug here: https://github.com/taglib/taglib/issues/770

Looking at the taglib source now to see whether there is an easy fix.

[Michi Henning (michihenning)]

I have a fix for the taglib issue that does not require changes to taglib. (taglib 1.11 added a new API for getting images out of VorbisComments and broke the behavior (not API) of the existing one.)

Still looking at the other issues.

[Michi Henning (michihenning)]

Taglib fix is in the linked branch.

The thumbnailer test dumps core on all architectures on zesty. The fault keeps moving around, looks like a race condition. We are not using threads ourselves, so the problem likely is in a something we call into. Candidates would be QT and (unlikely) leveldb.

I've run the tests for several hours continuously on my zesty amd64 VM, without error. The fault shows up only in the silo.

For added amusement, the qml test for vivid arm64 is segfaulting. I have no idea what might be causing this. The thumbnailer code has been stable for many months and hasn't ever had this kind of problem.

I suspect that the issue is with something that is installed on the builder. But there is no way to get a core dump when something fails in a builder, so I don't know how to debug this. Without a core dump, we are stabbing around in the dark :-(

[Michi Henning (michihenning)]

I think we can rule out leveldb. It's at revision 1.18-5 in both xenial and zesty.

[Michi Henning (michihenning)]

I'm seeing this report from valgrind on zesty. On xenial, the test runs clean. Not sure whether this might be related.

==63566== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 10 from 10)
==63463== Conditional jump or move depends on uninitialised value(s)
==63463== at 0x4C33E26: rawmemchr (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==63463== by 0x792D701: _IO_str_init_static_internal (strops.c:41)
==63463== by 0x791F6B6: vsscanf (iovsscanf.c:40)
==63463== by 0x79198F6: sscanf (sscanf.c:32)
==63463== by 0x61D1486: aa_query_label (in /lib/x86_64-linux-gnu/libapparmor.so.1.4.0)
==63463== by 0x17B207: (anonymous namespace)::query_file((anonymous namespace)::Access, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (check_access.cpp:59)
==63463== by 0x17B379: unity::thumbnailer::internal::apparmor_can_read(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (check_access.cpp:80)
==63463== by 0x16B6B1: unity::thumbnailer::internal::(anonymous namespace)::LocalThumbnailRequest::check_client_credentials(unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (thumbnailer.cpp:546)
==63463== by 0x14E5CC: ThumbnailerTest_check_client_access_Test::TestBody() (thumbnailer_test.cpp:579)
==63463== by 0x1AAFC3: void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2078)
==63463== by 0x1A630E: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2114)
==63463== by 0x18BF7B: testing::Test::Run() (gtest.cc:2151)
==63463== by 0x18C7C7: testing::TestInfo::Run() (gtest.cc:2326)
==63463== by 0x18CE33: testing::TestCase::Run() (gtest.cc:2444)
==63463== by 0x193E91: testing::internal::UnitTestImpl::RunAllTests() (gtest.cc:4315)
==63463== by 0x1ABF06: bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (gtest.cc:2078)
==63463== by 0x1A70D2: bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (gtest.cc:2114)
==63463== by 0x192A8C: testing::UnitTest::Run() (gtest.cc:3926)
==63463== by 0x15B30B: RUN_ALL_TESTS() (gtest.h:2288)
==63463== by 0x15942E: main (thumbnailer_test.cpp:1042)

[Michi Henning (michihenning)]

Almost certainly not related. The test that triggers this is far away from the core dump. Still worth following up on though.

[Michi Henning (michihenning)]

After lots of debugging and tracing, I found the root cause with valgrind. The interesting thing is that this happens only in the silo. When I run the same code on my zesty VM, everything is fine. So, what is different in the silo builders from a vanilla zesty install? (Installed zesty about two days ago. Is it possible that the builders are missing some updates?)

1: ==8494== Process terminating with default action of signal 11 (SIGSEGV)
1: ==8494== Access not within mapped region at address 0x61632F6D6F6347
1: ==8494== at 0x4EB5804: leveldb::InternalFilterPolicy::CreateFilter(leveldb::Slice const*, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) const (in /usr/lib/aarch64-linux-gnu/libleveldb.so.1.19)
1: ==8494== by 0x4ECBBAF: leveldb::FilterBlockBuilder::GenerateFilter() (in /usr/lib/aarch64-linux-gnu/libleveldb.so.1.19)
1: ==8494== by 0x4ECBD87: leveldb::FilterBlockBuilder::StartBlock(unsigned long) (in /usr/lib/aarch64-linux-gnu/libleveldb.so.1.19)
1: ==8494== by 0x4ED1F8F: leveldb::TableBuilder::Flush() (in /usr/lib/aarch64-linux-gnu/libleveldb.so.1.19)
1: ==8494== by 0x4ED2113: leveldb::TableBuilder::Add(leveldb::Slice const&, leveldb::Slice const&) (in /usr/lib/aarch64-linux-gnu/libleveldb.so.1.19)
1: ==8494== by 0x4EA68AB: leveldb::BuildTable(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, leveldb::Env*, leveldb::Options const&, leveldb::TableCache*, leveldb::Iterator*, leveldb::FileMetaData*) (in /usr/lib/aarch64-linux-gnu/libleveldb.so.1.19)
1: ==8494== by 0x4EAD183: leveldb::DBImpl::WriteLevel0Table(leveldb::MemTable*, leveldb::VersionEdit*, leveldb::Version*) (in /usr/lib/aarch64-linux-gnu/libleveldb.so.1.19)
1: ==8494== by 0x4EAEECF: leveldb::DBImpl::CompactMemTable() (in /usr/lib/aarch64-linux-gnu/libleveldb.so.1.19)
1: ==8494== by 0x4EAFD3F: leveldb::DBImpl::BackgroundCompaction() (in /usr/lib/aarch64-linux-gnu/libleveldb.so.1.19)
1: ==8494== by 0x4EB051F: leveldb::DBImpl::BackgroundCall() (in /usr/lib/aarch64-linux-gnu/libleveldb.so.1.19)
1: ==8494== by 0x4ED7C3B: ??? (in /usr/lib/aarch64-linux-gnu/libleveldb.so.1.19)
1: ==8494== by 0x53E901B: start_thread (pthread_create.c:335)

[Michi Henning (michihenning)]

The builder uses libleveldb 1.19-2, but the zesty archives still have libleveldb 1.18-5.
After installing 1.19-2 locally, I instantly got the same failure.

[Michi Henning (michihenning)]

Upstream bug here: https://github.com/Level/leveldown/issues/321

[Michi Henning (michihenning)]

Sorry wrong link for upstream bug. It's here: https://github.com/google/leveldb/issues/425

[Dimitri John Ledkov (xnox)]

leveldb 1.19-2 is stuck in zesty-proposed over autopackagetest failures with the python bindings. Seems like it's stuffed. Requested the broken leveldb to be removed from zesty-proposed, thus reverting to stable 1.18-5 which is in zesty (release). After removal is published, will trigger thumbnailer retries.

[Chuck Short (zulcss)]

Marking as fixed released according tot he feedback here.