radosgw crashed with SIGABRT in __GI_raise() when uploading small file via Horizon/CLI

[Ed Stewart]

[Impact]
radosgw is non functional - files cannot be uploaded.

[Test Case]
Deploy ceph + radosgw
Upload file using swift cli tooling
radosgw daemon SIGABRT's.

[Regression Potential]
The cause of this bug is a distro patch for 32bit compatibility which mis-aligns the type for the retrieval of a configuration option with the underlying type of the option - this results in a SIGABRT.

The update to the distro patch realigns the types and aligns with a broader patch in upstream ceph in master branch development.

[Original Bug Report]
Hi,

Juju deployed OpenStack using a modified openstack-core bundle - Rocky on bionic. All components (except neutron-gateway) are deployed in LXD containers.

Horizon/CLI can list and create new containers, however, if a small file (in this case a cloud.init file of 82 bytes) is uploaded, the radosgw process crashes with attached crash file.

NB this is running within GCP

Please advise if any further information required. Here's the snippet out of the end of radosgw.log

d9fa6cacbb94b96b4941ec2af38a49f$ad9fa6cacbb94b96b4941ec2af38a49f:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e80) v8 -- 0x5587e73f9740 con 0
   -27> 2018-11-26 12:17:31.262 7f87d596b700 5 -- 252.0.89.179:0/1413084365 >> 252.0.80.134:6800/7301 conn(0x5587e6c51800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.0 seq 10832 0x5587e73f9740 osd_op_reply(132197 ad9fa6cacbb94b96b4941ec2af38a49f$ad9fa6cacbb94b96b4941ec2af38a49f [getxattrs,stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8
   -26> 2018-11-26 12:17:31.262 7f87d596b700 1 -- 252.0.89.179:0/1413084365 <== osd.0 252.0.80.134:6800/7301 10832 ==== osd_op_reply(132197 ad9fa6cacbb94b96b4941ec2af38a49f$ad9fa6cacbb94b96b4941ec2af38a49f [getxattrs,stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 251+0+0 (2451902153 0 0) 0x5587e73f9740 con 0x5587e6c51800
   -25> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001665:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:normalizing buckets and tenants
   -24> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001701:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:init permissions
   -23> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001755:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:recalculating target
   -22> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001779:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:reading permissions
   -21> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001786:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:init op
   -20> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001788:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:verifying op mask
   -19> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001789:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:verifying op permissions
   -18> 2018-11-26 12:17:31.262 7f87b633a700 5 content type wasn't provided, trying to guess
   -17> 2018-11-26 12:17:31.262 7f87b633a700 5 Searching permissions for identity=rgw::auth::ThirdPartyAccountApplier() -> rgw::auth::SysReqApplier -> rgw::auth::RemoteApplier(acct_user=ad9fa6cacbb94b96b4941ec2af38a49f, acct_name=xxxx_mgmt, perm_mask=15, is_admin=0) mask=50
   -16> 2018-11-26 12:17:31.262 7f87b633a700 5 Searching permissions for uid=ad9fa6cacbb94b96b4941ec2af38a49f
   -15> 2018-11-26 12:17:31.262 7f87b633a700 5 Found permission: 15
   -14> 2018-11-26 12:17:31.262 7f87b633a700 5 Searching permissions for uid=ad9fa6cacbb94b96b4941ec2af38a49f$ad9fa6cacbb94b96b4941ec2af38a49f
   -13> 2018-11-26 12:17:31.262 7f87b633a700 5 Permissions for user not found
   -12> 2018-11-26 12:17:31.262 7f87b633a700 5 Searching permissions for group=1 mask=50
   -11> 2018-11-26 12:17:31.262 7f87b633a700 5 Permissions for group not found
   -10> 2018-11-26 12:17:31.262 7f87b633a700 5 Searching permissions for group=2 mask=50
    -9> 2018-11-26 12:17:31.262 7f87b633a700 5 Permissions for group not found
    -8> 2018-11-26 12:17:31.262 7f87b633a700 5 -- Getting permissions done for identity=rgw::auth::ThirdPartyAccountApplier() -> rgw::auth::SysReqApplier -> rgw::auth::RemoteApplier(acct_user=ad9fa6cacbb94b96b4941ec2af38a49f, acct_name=xxxxx_mgmt, perm_mask=15, is_admin=0), owner=ad9fa6cacbb94b96b4941ec2af38a49f, perm=2
    -7> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001818:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:verifying op params
    -6> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001820:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:pre-executing
    -5> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001821:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:executing
    -4> 2018-11-26 12:17:31.262 7f87b633a700 1 -- 252.0.89.179:0/1413084365 --> 252.0.90.230:6800/17596 -- osd_op(unknown.0.0:132198 6.0 6:21b9d9f7:::.dir.c9c6424e-a4bd-44de-b972-6cd4539a0a61.5322.1:head [call rgw.bucket_list] snapc 0=[] ondisk+read+known_if_redirected e80) v8 -- 0x5587e745a9c0 con 0
    -3> 2018-11-26 12:17:31.266 7f87d616c700 5 -- 252.0.89.179:0/1413084365 >> 252.0.90.230:6800/17596 conn(0x5587e6c50300 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=10 cs=1 l=1). rx osd.1 seq 54313 0x5587e745a9c0 osd_op_reply(132198 .dir.c9c6424e-a4bd-44de-b972-6cd4539a0a61.5322.1 [call] v0'0 uv1 ondisk = 0) v8
    -2> 2018-11-26 12:17:31.266 7f87d616c700 1 -- 252.0.89.179:0/1413084365 <== osd.1 252.0.90.230:6800/17596 54313 ==== osd_op_reply(132198 .dir.c9c6424e-a4bd-44de-b972-6cd4539a0a61.5322.1 [call] v0'0 uv1 ondisk = 0) v8 ==== 192+0+71 (1074935246 0 1943469035) 0x5587e745a9c0 con 0x5587e6c50300
    -1> 2018-11-26 12:17:31.266 7f87b633a700 5 NOTICE: call to do_aws4_auth_completion
     0> 2018-11-26 12:17:31.266 7f87b633a700 -1 *** Caught signal (Aborted) **
 in thread 7f87b633a700 thread_name:civetweb-worker

 ceph version 13.2.1 (5533ecdc0fda920179d7ad84e0aa65a127b20d77) mimic (stable)
 1: (()+0x293e70) [0x5587e4514e70]
 2: (()+0x12890) [0x7f87eb99d890]
 3: (gsignal()+0xc7) [0x7f87e0e96e97]
 4: (abort()+0x141) [0x7f87e0e98801]
 5: (()+0x8c8b7) [0x7f87e14ed8b7]
 6: (()+0x92a06) [0x7f87e14f3a06]
 7: (()+0x92a41) [0x7f87e14f3a41]
 8: (()+0x92c74) [0x7f87e14f3c74]
 9: (()+0x35e30d) [0x5587e45df30d]
 10: (Option::size_t const md_config_t::get_val<Option::size_t>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const+0x53) [0x5587e461fab3]
 11: (()+0x37df4f) [0x5587e45fef4f]
 12: (RGWPutObj::execute()+0x1adb) [0x5587e4600efb]
 13: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, bool)+0x1cd) [0x5587e4620cfd]
 14: (process_request(RGWRados*, RGWREST*, RGWRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, int*)+0x15f0) [0x5587e46223f0]
 15: (RGWCivetWebFrontend::process(mg_connection*)+0x381) [0x5587e44976e1]
 16: (()+0x2851ee) [0x5587e45061ee]
 17: (()+0x286e78) [0x5587e4507e78]
 18: (()+0x287308) [0x5587e4508308]
 19: (()+0x76db) [0x7f87eb9926db]
 20: (clone()+0x3f) [0x7f87e0f7988f]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 1 lockdep
   0/ 1 context
   1/ 1 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 1 buffer
   0/ 1 timer
   0/ 1 filer
   0/ 1 striper
   0/ 1 objecter
   0/ 5 rados
   0/ 5 rbd
   0/ 5 rbd_mirror
   0/ 5 rbd_replay
   0/ 5 journaler
   0/ 5 objectcacher
   0/ 5 client
   1/ 5 osd
   0/ 5 optracker
   0/ 5 objclass
   1/ 3 filestore
   1/ 3 journal
   0/ 5 ms
   1/ 5 mon
   0/10 monc
   1/ 5 paxos
   0/ 5 tp
   1/ 5 auth
   1/ 5 crypto
   1/ 1 finisher
   1/ 1 reserver
   1/ 5 heartbeatmap
   1/ 5 perfcounter
   1/ 5 rgw
   1/ 5 rgw_sync
   1/10 civetweb
   1/ 5 javaclient
   1/ 5 asok
   1/ 1 throttle
   0/ 0 refs
   1/ 5 xio
   1/ 5 compressor
   1/ 5 bluestore
   1/ 5 bluefs
   1/ 3 bdev
   1/ 5 kstore
   4/ 5 rocksdb
   4/ 5 leveldb
   4/ 5 memdb
   1/ 5 kinetic
   1/ 5 fuse
   1/ 5 mgr
   1/ 5 mgrc
   1/ 5 dpdk
   1/ 5 eventtrace
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
  max_recent 10000
  max_new 1000
  log_file /var/log/ceph/radosgw.log
--- end dump of recent events ---

ProblemType: Crash
DistroRelease: Ubuntu 18.04
Package: radosgw 13.2.1+dfsg1-0ubuntu2~cloud0 [origin: Canonical]
ProcVersionSignature: Ubuntu 4.15.0-1024.25-gcp 4.15.18
Uname: Linux 4.15.0-1024-gcp x86_64
ApportVersion: 2.20.9-0ubuntu7.5
Architecture: amd64
CrashDB:
 {
                "impl": "launchpad",
                "project": "cloud-archive",
                "bug_pattern_url": "http://people.canonical.com/~ubuntu-archive/bugpatterns/bugpatterns.xml",
             }
Date: Mon Nov 26 12:17:31 2018
ExecutablePath: /usr/bin/radosgw
ProcCmdline: /usr/bin/radosgw -n client.radosgw.gateway
ProcEnviron:
 PATH=(custom, no user)
 LANG=C.UTF-8
Signal: 6
SourcePackage: ceph
StacktraceTop:
 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
 __GI_abort () at abort.c:79
 ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
 ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
 std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
Title: radosgw crashed with SIGABRT in __GI_raise()
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:

[James Page (james-page)]

I was able to confirm this issue on a rocky/bionic deployment outside of GCP

[James Page (james-page)]

I think I see the cause of this issue; we're carrying a distro patch for 32 bit compatibility which changes the type of a configuration option on use to a Option:size_t - however the underlying type of the configuration option is a UINT still which might be causing the SIGABRT.

Producing a test build to validate.

[James Page (james-page)]

https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3534

[Apport retracing service (apport)]

StacktraceTop:
 ?? ()
 ?? ()
 ?? ()
 ?? ()
 ?? ()

[Apport retracing service (apport)]

Thank you for your report!

However, processing it in order to get sufficient information for the
developers failed (it does not generate a useful symbolic stack trace). This
might be caused by some outdated packages which were installed on your system
at the time of the report:

radosgw version 13.2.1+dfsg1-0ubuntu2~cloud0 required, but 12.2.8-0ubuntu0.18.04.1 is available
ceph-common version 13.2.1+dfsg1-0ubuntu2~cloud0 required, but 12.2.8-0ubuntu0.18.04.1 is available
no debug symbol package found for gcc-8-base
libcephfs2 version 13.2.1+dfsg1-0ubuntu2~cloud0 required, but 12.2.8-0ubuntu0.18.04.1 is available
no debug symbol package found for libgssapi-krb5-2
no debug symbol package found for libk5crypto3
no debug symbol package found for libkrb5-3
no debug symbol package found for libkrb5support0
no debug symbol package found for libmagic-mgc
no debug symbol package found for libnl-route-3-200
no debug symbol package found for libpython-stdlib
no debug symbol package found for libpython2.7-minimal
no debug symbol package found for libpython2.7-stdlib
librados2 version 13.2.1+dfsg1-0ubuntu2~cloud0 required, but 12.2.8-0ubuntu0.18.04.1 is available
libradosstriper1 version 13.2.1+dfsg1-0ubuntu2~cloud0 required, but 12.2.8-0ubuntu0.18.04.1 is available
librbd1 version 13.2.1+dfsg1-0ubuntu2~cloud0 required, but 12.2.8-0ubuntu0.18.04.1 is available
librgw2 version 13.2.1+dfsg1-0ubuntu2~cloud0 required, but 12.2.8-0ubuntu0.18.04.1 is available
no debug symbol package found for perl-base
python-cephfs version 13.2.1+dfsg1-0ubuntu2~cloud0 required, but 12.2.8-0ubuntu0.18.04.1 is available
no debug symbol package found for python-minimal
python-rados version 13.2.1+dfsg1-0ubuntu2~cloud0 required, but 12.2.8-0ubuntu0.18.04.1 is available
python-rbd version 13.2.1+dfsg1-0ubuntu2~cloud0 required, but 12.2.8-0ubuntu0.18.04.1 is available
no debug symbol package found for python2.7-minimal
no debug symbol package found for libnl-route-3-200
no debug symbol package found for libkrb5support0
librados2 version 13.2.1+dfsg1-0ubuntu2~cloud0 required, but 12.2.8-0ubuntu0.18.04.1 is available
no debug symbol package found for libkrb5-3
no debug symbol package found for libgssapi-krb5-2
no debug symbol package found for libk5crypto3
radosgw version 13.2.1+dfsg1-0ubuntu2~cloud0 required, but 12.2.8-0ubuntu0.18.04.1 is available

Please upgrade your system to the latest package versions. If you still
encounter the crash, please file a new report.

Thank you for your understanding, and sorry for the inconvenience!

[James Page (james-page)]

Also reported upstream: https://tracker.ceph.com/issues/37386

[James Page (james-page)]

Revised patch resolves the issue; I can upload content to the radosgw.

I'll start work on the SRU's now.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ceph - 13.2.2+dfsg1-0ubuntu2

---------------
ceph (13.2.2+dfsg1-0ubuntu2) disco; urgency=medium

  * d/tests/python-ceph: Fix python3 test support resolving
    autopkgtest failure.

 -- James Page <email address hidden> Fri, 30 Nov 2018 13:00:55 +0000

[James Page (james-page)]

Testing packages in:

  https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3535

[Chris MacNaughton (chris.macnaughton)]

I can confirm that ceph-radosgw behaves correctly on a bionic-rocky cloud, with radosgw using the package from the linked ppa (source: "ppa:ci-train-ppa-service/3535"), allowing images to be synced in correctly

[James Page (james-page)]

Hello Ed, or anyone else affected,

Accepted ceph into rocky-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:rocky-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-rocky-needed to verification-rocky-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-rocky-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Chris MacNaughton (chris.macnaughton)]

I can confirm that radosgw works as expected when deploying the cloud from rocky-proposed

[Chris MacNaughton (chris.macnaughton)]

package version: radosgw/bionic-proposed,now 13.2.1+dfsg1-0ubuntu2.18.10.1~cloud0 amd64 [installed]

[James Page (james-page)]

The verification of the Stable Release Update for ceph has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[James Page (james-page)]

This bug was fixed in the package ceph - 13.2.1+dfsg1-0ubuntu2.18.10.1~cloud0
---------------

 ceph (13.2.1+dfsg1-0ubuntu2.18.10.1~cloud0) bionic; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 ceph (13.2.1+dfsg1-0ubuntu2.18.10.1) cosmic; urgency=medium
 .
   * Re-introduce Ceph Mimic to Ubuntu Cosmic (LP: #1805355).
   * d/p/misc-32-bit-fixes.patch: Update type of rgw_max_attr_name_len,
     resolving SIGABRT in radosgw (LP: #1805145).

[Brian Murray (brian-murray)]

Hello Ed, or anyone else affected,

Accepted ceph into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ceph/13.2.1+dfsg1-0ubuntu2.18.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[James Page (james-page)]

Deployed cosmic proposed and uploaded a large git repository of files into the rados gateway using the CLI tooling; all uploaded OK and daemon did not SIGABRT.

[James Page (james-page)]

Marking cosmic tags for verification as done.

[James Page (james-page)]

$ swift stat test
                      Account: v1
                    Container: test
                      Objects: 6251
                        Bytes: 133947693
                     Read ACL:
                    Write ACL:
                      Sync To:
                     Sync Key:
       X-Openstack-Request-Id: tx000000000000000003048-005c10e76d-3f4a-default
                Last-Modified: Wed, 12 Dec 2018 10:44:59 GMT
                 Content-Type: text/plain; charset=utf-8
                   X-Trans-Id: tx000000000000000003048-005c10e76d-3f4a-default
                Accept-Ranges: bytes
X-Container-Bytes-Used-Actual: 149155840
                  X-Timestamp: 1544611499.76620
             X-Storage-Policy: default-placement

[Launchpad Janitor (janitor)]

This bug was fixed in the package ceph - 13.2.1+dfsg1-0ubuntu2.18.10.1

---------------
ceph (13.2.1+dfsg1-0ubuntu2.18.10.1) cosmic; urgency=medium

  * Re-introduce Ceph Mimic to Ubuntu Cosmic (LP: #1805355).
  * d/p/misc-32-bit-fixes.patch: Update type of rgw_max_attr_name_len,
    resolving SIGABRT in radosgw (LP: #1805145).

 -- James Page <email address hidden> Wed, 28 Nov 2018 10:35:58 +0000