Unable to start or initialise any ceph process (ceph-mon)

[James Page]

Latest yakkety server build, with ceph 10.2.2; creation of the ceph-mon FS hangs:

ceph-mon --mkfs -i juju-3da7bf0b-0062-4a2f-8949-d85e6579b759-machine-15 --keyring /var/lib/ceph/tmp/juju-3da7bf0b-0062-4a2f-8949-d85e6579b759-machine-15.mon.keyring

spinning in:

nanosleep({0, 2000001}, NULL) = 0

top of backtrace:

#0 0x00007f5a619efd10 in __nanosleep_nocancel () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1 0x00007f5a624b9193 in base::internal::SpinLockDelay(int volatile*, int, int) () from /usr/lib/libtcmalloc.so.4
No symbol table info available.
#2 0x00007f5a624b9026 in SpinLock::SlowLock() () from /usr/lib/libtcmalloc.so.4
No symbol table info available.
#3 0x00007f5a624adb78 in tcmalloc::ThreadCache::InitModule() () from /usr/lib/libtcmalloc.so.4
No symbol table info available.
#4 0x00007f5a624bbe55 in tc_malloc () from /usr/lib/libtcmalloc.so.4
No symbol table info available.
#5 0x00007f5a5fc05f7d in __fopen_internal (filename=0x7f5a6291c4a0 "/usr/lib/libtcmalloc.so.4", mode=0x7f5a5f984c6a "r", is32=1) at iofopen.c:69
        new_f = <optimized out>
#6 0x00007f5a5f982ab8 in ?? () from /usr/lib/x86_64-linux-gnu/libunwind.so.8
No symbol table info available.
#7 0x00007f5a5f983624 in ?? () from /usr/lib/x86_64-linux-gnu/libunwind.so.8
No symbol table info available.
#8 0x00007f5a5f983710 in _ULx86_64_dwarf_find_debug_frame () from /usr/lib/x86_64-linux-gnu/libunwind.so.8
No symbol table info available.
#9 0x00007f5a5f983cde in ?? () from /usr/lib/x86_64-linux-gnu/libunwind.so.8
No symbol table info available.
#10 0x00007f5a5fcdc0a4 in __GI___dl_iterate_phdr (callback=0x7f5a5f983940, data=0x7fff3a02ba90) at dl-iteratephdr.c:76
        __clframe = {__cancel_routine = <optimized out>, __cancel_arg = 0x0, __do_it = 1, __cancel_type = <optimized out>}
        nloaded = 26
        ns = <optimized out>
        caller = <optimized out>
        l = 0x7f5a6291c4c0
        info = {dlpi_addr = 140026172706816, dlpi_name = 0x7f5a6291c4a0 "/usr/lib/libtcmalloc.so.4", dlpi_phdr = 0x7f5a62489040, dlpi_phnum = 8,
          dlpi_adds = 26, dlpi_subs = 0, dlpi_tls_modid = 2, dlpi_tls_data = 0x7f5a62906760}
        ret = 0

looks like initial setup of tcmalloc is spinning - I've seen the same happen on start of the ceph-mon daemon (which I thought was the original problem).

ProblemType: Bug
DistroRelease: Ubuntu 16.10
Package: ceph-mon 10.2.2-0ubuntu5
ProcVersionSignature: Ubuntu 4.8.0-17.19-generic 4.8.0-rc7
Uname: Linux 4.8.0-17-generic x86_64
ApportVersion: 2.20.3-0ubuntu7
Architecture: amd64
Date: Thu Sep 29 20:43:51 2016
Ec2AMI: ami-0000045c
Ec2AMIManifest: FIXME
Ec2AvailabilityZone: nova
Ec2InstanceType: m1.small.osci
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
SourcePackage: ceph
UpgradeStatus: No upgrade log present (probably fresh install)

[James Page (james-page)]

Reverting back to libunwind 1.1-4.1 resolves this problem for me - the changelog entry is:

libunwind (1.1-4.1ubuntu1) yakkety; urgency=medium

  * Configure with --enable-debug-frame. LP: #1590236.

 -- Matthias Klose <email address hidden> Mon, 19 Sep 2016 13:28:08 +0200

ceph was functional on yakkety not that long ago (this landed into updates on the 28th).

[James Page (james-page)]

I reverted that change and uploaded to:

 https://launchpad.net/~openstack-ubuntu-testing/+archive/ubuntu/ceph-sru

to test and confirm that appears to resolve this problem for me.

[Matthias Klose (doko)]

does the issue persist when rebuilding ceph against the new libunwind?

[James Page (james-page)]

No, the packages in:

 https://launchpad.net/~openstack-ubuntu-testing/+archive/ubuntu/ceph-sru/+build/10968224

where built using the 4.1ubuntu1 version from the archive and have the same symptom.

The dependency is via google-perftools rather that direct from ceph.

[Matthias Klose (doko)]

my bad, so maybe check with a rebuilt google-perftools?

[James Page (james-page)]

Somehow this bug just went away - I'm not 100% sure why but marking invalid for ceph.

[Graham Inggs (ginggs)]

libunwind (1.1-4.1ubuntu2) yakkety; urgency=medium

  * Don't configure with --enable-debug-frame on amd64.

 -- Matthias Klose <email address hidden> Fri, 30 Sep 2016 21:30:21 +0200

[Launchpad Janitor (janitor)]

[Expired for google-perftools (Ubuntu) because there has been no activity for 60 days.]

[Launchpad Janitor (janitor)]

[Expired for libunwind (Ubuntu) because there has been no activity for 60 days.]