Glusterfs 1.0 breaks libvirt tests on s390x

[Christian Ehrhardt ]

On building libvirt I got these fails on s390x (all other architectures are fine):

 88/162 virdrivermoduletest FAIL 0.04s exit status 1
128/162 storagepoolxml2argvtest FAIL 0.03s exit status 1
129/162 storagepoolxml2xmltest FAIL 0.03s exit status 1
132/162 virstoragetest FAIL 0.03s exit status 1

I checked and it also breaks without my recent changes (regression in release without a libvirt upload)

The error messages themselve are not helpful (just no output)

LP jammy - works
  https://launchpad.net/~paelzer/+archive/ubuntu/libvirt-regressed-build-jammy/+build/22402236

LP jammy proposed - fails
  https://launchpad.net/~paelzer/+archive/ubuntu/libvirt-regressed-build-jammy-proposed/+build/22402343

VM / Container doesn't matter, but the config must be ok
For a local repro run:

./debian/rules clean; export DEB_HOST_ARCH=s390x; export DEB_HOST_ARCH_OS=linux; ./debian/rules build

The most likely suspect seemed to be glib2 2.68.4 -> 2.70.1
But downgrading that still hit the issue, next I found an actual error message

$ VIR_TEST_DEBUG=1 VIR_TEST_RANGE=3 ./debian/build/tests/virdrivermoduletest
TEST: virdrivermoduletest
 3) Test driver "storage" ...
libvirt: error : internal
error: Failed to load module '/root/libvirt-7.6.0/debian/build/src/libvirt_storage_backend_gluster.so':
/lib/s390x-linux-gnu/libtcmalloc.so.4: cannot allocate memory in static TLS block
FAILED

The other three tests do not have any useful extended info.

But gluster was in that candidate list I was checking.

Downgrade Gluster
v=9.2-1; apt install libglusterfs-dev=$v libglusterfs0=$v libglusterd0=$v libgfapi0=$v libgfchangelog0=$v libgfrpc0=$v libgfxdr0=$v

Working fine with 9.2-1

[Christian Ehrhardt  (paelzer)]

Thanks do Danpb I have found https://bugzilla.redhat.com/show_bug.cgi?id=2018439

So we really might want to disable tcmalloc in glusterfs.

[Christian Ehrhardt  (paelzer)]

Test derived from the BZ

9.2-1 and 10.0-1 both do work for x86 (as expected)

$ apt install libglusterfs-dev gcc
$ cat gtest.c
#include <stdio.h>
#include <dlfcn.h>
#include <assert.h>

int main(int argc, char **argv)
{
  void *h = dlopen("/usr/lib/x86_64-linux-gnu/libglusterfs.so.0", RTLD_NOW);
  if (!h) fprintf (stderr, "dlerror = %s\n", dlerror());
  return 0;
}
$ gcc -Wall gtest.c -o gtest -ldl -g
$ ./gtest

But on s390x (we only see the FTBFS on s390x+arm64)
(Path obviously has to be adapted)

#include <stdio.h>
#include <dlfcn.h>
#include <assert.h>

int main(int argc, char **argv)
{
  void *h = dlopen("/usr/lib/s390x-linux-gnu/libglusterfs.so.0", RTLD_NOW);
  if (!h) fprintf (stderr, "dlerror = %s\n", dlerror());
  return 0;
}

Impish fine
root@i-gluster:~# ./gtest

Jammy broken
root@j-gluster:~# ./gtest
dlerror = /lib/s390x-linux-gnu/libtcmalloc.so.4: cannot allocate memory in static TLS block

[Christian Ehrhardt  (paelzer)]

Right now we have in the config step on all architectures:

GlusterFS configure summary
===========================
...
Link with TCMALLOC : yes

I think we should follow fedoras choice to - for now - only enable this on x86 where it is proven to provide a benefit. It is dysfunctional on arm64/s390x and we only have ppc64 left where IMHO the chances are high it might also have problems - better relying on the safer option here.

I have uploaded a version with that to the PPA [1] and rebuild libvirt against that.

Furthermore I have checked and this affects Debian just as much.
Therefore I have reported it there as well [2]

[1]: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4710/+packages
[2]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=999700

[Christian Ehrhardt  (paelzer)]

The PPA build works - non-x86 builds now have:

dh_auto_configure -- --disable-linux-io_uring --enable-firewalld --libexecdir=/usr/lib/s390x-linux-gnu --without-tcmalloc
 ./configure --build=s390x-linux-gnu --prefix=/usr --includedir=\${prefix}/include --mandir=\${prefix}/share/man --infodir=\${prefix}/share/info --sysconfdir=/etc --localstatedir=/var --disable-option-checking --disable-silent-rules --libdir=\${prefix}/lib/s390x-linux-gnu --runstatedir=/run --disable-maintainer-mode --disable-dependency-tracking --disable-linux-io_uring --enable-firewalld --libexecdir=/usr/lib/s390x-linux-gnu --without-tcmalloc
...
Link with TCMALLOC : no

And against that libvirt now also builds fine again.

[Christian Ehrhardt  (paelzer)]

This isn't in git-ubuntu yet, but the fix is small - therefore here as .debdiff for a second pair of eyes to have a look at.

I have reported the very same fix to the Debian bug that I have filed

P.S. Unrelated but FYI please do note that for 10.x to migrate in proposed we will also need https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=999619 to be resolved.

[Utkarsh Gupta (utkarsh)]

Hello,

The diff looks okay. I saw there's also a documented line to build rpms w/o tcmalloc so I guess it should be okay here, too. Good to go. Thank you.

[Paride Legovini (paride)]

Another +1 on the debdiff. IIUC this is a straight FTBFS on s390x, so in the future it should be easy to verify if the delta can be dropped.

OT:

-Maintainer: Patrick Matthäi <email address hidden>
+Maintainer: Ubuntu Developers <email address hidden>
+XSBC-Original-Maintainer: Patrick Matthäi <email address hidden>

I wonder what's the piece of software that needs to be updated to handle UTF-8 correctly...

[Christian Ehrhardt  (paelzer)]

That is a readable "ä" for me Paride :-)
It is what update-maintainer+debdiff creates, I didn't mess with it in any special way.

Thank you both, uploaded to resolve this and potentially more.

[Andreas Hasenack (ahasenack)]

Since you guys are working on it, removing me from the assignment slot.

[Christian Ehrhardt  (paelzer)]

This is now fixed in Debian-unstable and Jammy-proposed.
It doesn't have to go to -release to consider it fixed.
The old version 9.x currently in -release didn't have the issue and the version now in -proposed is fixed as well. Therefore this can be considered fix-released for Jammy.