radosgw crashed with SIGABRT in __GI_raise() when uploading small file via Horizon/CLI
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Ubuntu Cloud Archive |
Fix Released
|
High
|
James Page | ||
Rocky |
Fix Released
|
High
|
James Page | ||
Stein |
Fix Released
|
High
|
James Page | ||
ceph (Ubuntu) |
Fix Released
|
High
|
James Page | ||
Cosmic |
Fix Released
|
High
|
James Page | ||
Disco |
Fix Released
|
High
|
James Page |
Bug Description
[Impact]
radosgw is non functional - files cannot be uploaded.
[Test Case]
Deploy ceph + radosgw
Upload file using swift cli tooling
radosgw daemon SIGABRT's.
[Regression Potential]
The cause of this bug is a distro patch for 32bit compatibility which mis-aligns the type for the retrieval of a configuration option with the underlying type of the option - this results in a SIGABRT.
The update to the distro patch realigns the types and aligns with a broader patch in upstream ceph in master branch development.
[Original Bug Report]
Hi,
Juju deployed OpenStack using a modified openstack-core bundle - Rocky on bionic. All components (except neutron-gateway) are deployed in LXD containers.
Horizon/CLI can list and create new containers, however, if a small file (in this case a cloud.init file of 82 bytes) is uploaded, the radosgw process crashes with attached crash file.
NB this is running within GCP
Please advise if any further information required. Here's the snippet out of the end of radosgw.log
d9fa6cacbb94b96
-27> 2018-11-26 12:17:31.262 7f87d596b700 5 -- 252.0.89.
-26> 2018-11-26 12:17:31.262 7f87d596b700 1 -- 252.0.89.
-25> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001665:
-24> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001701:
-23> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001755:
-22> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001779:
-21> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001786:
-20> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001788:
-19> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001789:
-18> 2018-11-26 12:17:31.262 7f87b633a700 5 content type wasn't provided, trying to guess
-17> 2018-11-26 12:17:31.262 7f87b633a700 5 Searching permissions for identity=
-16> 2018-11-26 12:17:31.262 7f87b633a700 5 Searching permissions for uid=ad9fa6cacbb
-15> 2018-11-26 12:17:31.262 7f87b633a700 5 Found permission: 15
-14> 2018-11-26 12:17:31.262 7f87b633a700 5 Searching permissions for uid=ad9fa6cacbb
-13> 2018-11-26 12:17:31.262 7f87b633a700 5 Permissions for user not found
-12> 2018-11-26 12:17:31.262 7f87b633a700 5 Searching permissions for group=1 mask=50
-11> 2018-11-26 12:17:31.262 7f87b633a700 5 Permissions for group not found
-10> 2018-11-26 12:17:31.262 7f87b633a700 5 Searching permissions for group=2 mask=50
-9> 2018-11-26 12:17:31.262 7f87b633a700 5 Permissions for group not found
-8> 2018-11-26 12:17:31.262 7f87b633a700 5 -- Getting permissions done for identity=
-7> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001818:
-6> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001820:
-5> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001821:
-4> 2018-11-26 12:17:31.262 7f87b633a700 1 -- 252.0.89.
-3> 2018-11-26 12:17:31.266 7f87d616c700 5 -- 252.0.89.
-2> 2018-11-26 12:17:31.266 7f87d616c700 1 -- 252.0.89.
-1> 2018-11-26 12:17:31.266 7f87b633a700 5 NOTICE: call to do_aws4_
0> 2018-11-26 12:17:31.266 7f87b633a700 -1 *** Caught signal (Aborted) **
in thread 7f87b633a700 thread_
ceph version 13.2.1 (5533ecdc0fda92
1: (()+0x293e70) [0x5587e4514e70]
2: (()+0x12890) [0x7f87eb99d890]
3: (gsignal()+0xc7) [0x7f87e0e96e97]
4: (abort()+0x141) [0x7f87e0e98801]
5: (()+0x8c8b7) [0x7f87e14ed8b7]
6: (()+0x92a06) [0x7f87e14f3a06]
7: (()+0x92a41) [0x7f87e14f3a41]
8: (()+0x92c74) [0x7f87e14f3c74]
9: (()+0x35e30d) [0x5587e45df30d]
10: (Option::size_t const md_config_
11: (()+0x37df4f) [0x5587e45fef4f]
12: (RGWPutObj:
13: (rgw_process_
14: (process_
15: (RGWCivetWebFro
16: (()+0x2851ee) [0x5587e45061ee]
17: (()+0x286e78) [0x5587e4507e78]
18: (()+0x287308) [0x5587e4508308]
19: (()+0x76db) [0x7f87eb9926db]
20: (clone()+0x3f) [0x7f87e0f7988f]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
1/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 1 reserver
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/ 5 rgw_sync
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
4/ 5 rocksdb
4/ 5 leveldb
4/ 5 memdb
1/ 5 kinetic
1/ 5 fuse
1/ 5 mgr
1/ 5 mgrc
1/ 5 dpdk
1/ 5 eventtrace
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/
--- end dump of recent events ---
ProblemType: Crash
DistroRelease: Ubuntu 18.04
Package: radosgw 13.2.1+
ProcVersionSign
Uname: Linux 4.15.0-1024-gcp x86_64
ApportVersion: 2.20.9-0ubuntu7.5
Architecture: amd64
CrashDB:
{
}
Date: Mon Nov 26 12:17:31 2018
ExecutablePath: /usr/bin/radosgw
ProcCmdline: /usr/bin/radosgw -n client.
ProcEnviron:
PATH=(custom, no user)
LANG=C.UTF-8
Signal: 6
SourcePackage: ceph
StacktraceTop:
__GI_raise (sig=sig@entry=6) at ../sysdeps/
__GI_abort () at abort.c:79
?? () from /usr/lib/
?? () from /usr/lib/
std::terminate() () from /usr/lib/
Title: radosgw crashed with SIGABRT in __GI_raise()
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:
information type: | Private → Public |
Changed in ceph (Ubuntu Disco): | |
assignee: | nobody → James Page (james-page) |
Changed in ceph (Ubuntu Cosmic): | |
assignee: | nobody → James Page (james-page) |
Changed in ceph (Ubuntu Disco): | |
status: | Invalid → In Progress |
Changed in ceph (Ubuntu Cosmic): | |
status: | Triaged → In Progress |
I was able to confirm this issue on a rocky/bionic deployment outside of GCP