From IRC as documentation of the next steps:

TL;DR: rebuild llvm with -g1 but with -DNDEBUG and retry then

[09:05] <doko> LocutusOfBorg, cpaelzer: what are you trying? we are back with two one gigabyte debug builds on ppc64el and s390x, and of course the failing amd64 build
[09:09] <LocutusOfBorg> doko, -g1 is breaking clamav
[09:09] <LocutusOfBorg> -g works
[09:09] <LocutusOfBorg> I'm doing a -g -NDEBUG to see what happens
[09:12] <cpaelzer> doko: bug 1717574
[09:12] <ubottu> bug 1717574 in clamav (Ubuntu) "new llvm3.9 breaking freshclam bytecode execution" [Medium,Confirmed] https://launchpad.net/bugs/1717574
[09:14] <doko> cpaelzer: is this upstreamed to clamav, or are we trying to change things which we don't understand? why is a change for the debug information responsible for that?
[09:15] <cpaelzer> doko: I only rerun the test and found the link of the issue to llvm - there is no clamav change
[09:16] <cpaelzer> doko: so nothing to upstream there that we'd know yet
[09:16] <cpaelzer> doko: it is just that their bytecode worked before a certina llvm upload and fails since then
[09:16] <doko> cpaelzer: did you try to rebuild clamav with the same flags as llvm?
[09:17] <doko> cpaelzer: or did you try to rebuild clamav at all?
[09:17] <cpaelzer> we rebuilt clamav with old and new llvm, but not change flags
[09:17] <cpaelzer> none of the rebuilds had any effect
[09:17] <LocutusOfBorg> we also tried a new clamav just to be sure
[09:17] <LocutusOfBorg> but yeah, no changes in flags, except for llvm
[09:18] <doko> please can you report that upstream?
[09:25] <doko> cpaelzer: we can't accomodate these builds if they take too much disk space to build, and even if, each of these packages take 1 gb, that makes 4gb for every upload ...
[09:28] <cpaelzer> doko: I did never say "please make them big" I'm only naively debugged a clamav crash and found that it is broken since these llvm uploads
[09:28] <cpaelzer> I beg your pardon if that got to you differently
[09:29] <cpaelzer> I'm reporting to clamav and will link it up
[09:30] <cpaelzer> doko: if you follow the bug I linked maybe you can see what the actual root cause might be the reason for that incompatibility
[09:31] <doko> cpaelzer: what is the change in compiler and linker flags?
[09:32] <LocutusOfBorg> cpaelzer, https://launchpad.net/~costamagnagianfranco/+archive/ubuntu/costamagnagianfranco-ppa/+packages
[09:32] <LocutusOfBorg> I uploaded a new llvm with -g1 (to make doko and infra happy) and with -DNDEBUG
[09:32] <LocutusOfBorg> what I discovered, is that ppassing RELWITHDEBINFO flags, overrides the cmake default ones
[09:32] <LocutusOfBorg> and that -DNDEBUG is stripped then
[09:32] <LocutusOfBorg> so the change has probably been a side-effect of that -g1, but not directly related
[09:34] <cpaelzer> I'll test that once it is built LocutusOfBorg
[09:34] <doko> cpaelzer: there's nothing about this -DNDEBUG in the bug report
[09:35] <doko> and then find out why clamav is relying on that ...
[09:35] <LocutusOfBorg> I discovered it some minutes ago, melding the failed amd64 and the succeeded one
[09:36] <LocutusOfBorg> because in my ppa it was good that build
[09:36] <doko> you should turn on creation of debug package in your ppa if you want to simulate the builders
[09:38] <LocutusOfBorg> both enabled, build and publish
[09:38] <LocutusOfBorg> thanks for the reminder, I admit I didn't recheck it
[09:47] <LocutusOfBorg> interesting, libclamav is using some NDEBUG stuff
[09:48] <cpaelzer> LocutusOfBorg: doko: I reported to clamav and linked it up in the bug
[09:49] <cpaelzer> LocutusOfBorg: let me know if/once there is a build I shall try for you
[09:49] <LocutusOfBorg> configure: CXXFLAGS from llvm-config: -I/usr/lib/llvm-3.9/include -std=c++0x -gsplit-dwarf -Wl,-fuse-ld=gold -fPIC -fvisibility-inlines-hidden -Wall -W -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wno-missing-field-initializers -pedantic -Wno-long-long -Wno-maybe-uninitialized -Wdelete-non-virtual-dtor -Wno-comment -Werror=date-time -std=c++11 -ffunction-sections -fdata-sections -O2 -g -DNDEBUG
[09:49] <LocutusOfBorg> -fno-exceptions -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS
[09:49] <LocutusOfBorg> interestingly, clamav is getting the build flags from llvm, so a no change rebuild should change them too
[09:50] <LocutusOfBorg> and in fact this seems to happen
[09:50] <cpaelzer> but you rebuilt a new clamav
[09:50] <cpaelzer> also I did with a local build vs the new library and -dev packages
[09:52] <LocutusOfBorg> but clamav not picking up NDEBUG from llvm flags, means that the code ifdefs are not triggered
[09:52] <LocutusOfBorg> this means less asserts
[09:53] <cpaelzer> but the clamav currently in artful uploaded by mdeslaur still built by picking -DNDEBUG up as it was before the llvm changes
[09:53] <LocutusOfBorg> I think I'll directly upload to Ubuntu this one
[09:53] <cpaelzer> https://launchpadlibrarian.net/333379617/buildlog_ubuntu-artful-amd64.clamav_0.99.2+dfsg-6ubuntu2_BUILDING.txt.gz
[09:54] <cpaelzer> LocutusOfBorg: do you mean we have lost the -NDEBUG in llvm and those "extra" asserts might be the reason to break it now?
[09:54] <LocutusOfBorg> somewhat this is a possible reason
[09:54] <LocutusOfBorg> rebuilding makes it assert because the current llvm in artful proposed is built without
[09:54] <LocutusOfBorg> probably rebuilding it with -DNDEBUG makes it pass
[09:56] <cpaelzer> LocutusOfBorg: ok, so since clamav is already built with -DNDEBUG from the old llvm builds that would mean likely no re-upload there. Once your ppa is ready I would install from it and then retry + rebuild+retry - ok?
[09:57] <LocutusOfBorg> ok
[09:57] <cpaelzer> so waiting on https://launchpad.net/~costamagnagianfranco/+archive/ubuntu/costamagnagianfranco-ppa/+build/13399109 now
[09:57] <LocutusOfBorg> I could also upload directly to Ubuntu
[09:57] <cpaelzer> I personally prefer uploading to ubuntu what is known or at least expected to work
[09:57] <LocutusOfBorg> the current proposed one has debug symbols too huge
[09:57] <cpaelzer> ah, ok I see the reason
[09:58] <LocutusOfBorg> even if not working, it is worth an upload
[09:58] <cpaelzer> well, are you sure the ubuntu5 at least builds and is as good/bad than the last one that had small symbols?
[09:58] <cpaelzer> I'll carry all this discussion into the bug for documentation