Juju deployed OpenStack using a modified openstack-core bundle - Rocky on bionic. All components (except neutron-gateway) are deployed in LXD containers.
Horizon/CLI can list and create new containers, however, if a small file (in this case a cloud.init file of 82 bytes) is uploaded, the radosgw process crashes with attached crash file.
NB this is running within GCP
Please advise if any further information required. Here's the snippet out of the end of radosgw.log
Hi,
Juju deployed OpenStack using a modified openstack-core bundle - Rocky on bionic. All components (except neutron-gateway) are deployed in LXD containers.
Horizon/CLI can list and create new containers, however, if a small file (in this case a cloud.init file of 82 bytes) is uploaded, the radosgw process crashes with attached crash file.
NB this is running within GCP
Please advise if any further information required. Here's the snippet out of the end of radosgw.log
d9fa6cacbb94b96 b4941ec2af38a49 f$ad9fa6cacbb94 b96b4941ec2af38 a49f:head [getxattrs,stat] snapc 0=[] ondisk+ read+known_ if_redirected e80) v8 -- 0x5587e73f9740 con 0 179:0/141308436 5 >> 252.0.80. 134:6800/ 7301 conn(0x5587e6c51800 :-1 s=STATE_ OPEN_MESSAGE_ READ_FOOTER_ AND_DISPATCH pgs=20 cs=1 l=1). rx osd.0 seq 10832 0x5587e73f9740 osd_op_reply(132197 ad9fa6cacbb94b9 6b4941ec2af38a4 9f$ad9fa6cacbb9 4b96b4941ec2af3 8a49f [getxattrs,stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 179:0/141308436 5 <== osd.0 252.0.80. 134:6800/ 7301 10832 ==== osd_op_reply(132197 ad9fa6cacbb94b9 6b4941ec2af38a4 9f$ad9fa6cacbb9 4b96b4941ec2af3 8a49f [getxattrs,stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 251+0+0 (2451902153 0 0) 0x5587e73f9740 con 0x5587e6c51800 swift:PUT /swift/ v1/test/ cloudinit. txt:put_ obj:normalizing buckets and tenants swift:PUT /swift/ v1/test/ cloudinit. txt:put_ obj:init permissions swift:PUT /swift/ v1/test/ cloudinit. txt:put_ obj:recalculati ng target swift:PUT /swift/ v1/test/ cloudinit. txt:put_ obj:reading permissions swift:PUT /swift/ v1/test/ cloudinit. txt:put_ obj:init op swift:PUT /swift/ v1/test/ cloudinit. txt:put_ obj:verifying op mask swift:PUT /swift/ v1/test/ cloudinit. txt:put_ obj:verifying op permissions rgw::auth: :ThirdPartyAcco untApplier( ) -> rgw::auth: :SysReqApplier -> rgw::auth: :RemoteApplier( acct_user= ad9fa6cacbb94b9 6b4941ec2af38a4 9f, acct_name= xxxx_mgmt, perm_mask=15, is_admin=0) mask=50 94b96b4941ec2af 38a49f 94b96b4941ec2af 38a49f$ ad9fa6cacbb94b9 6b4941ec2af38a4 9f rgw::auth: :ThirdPartyAcco untApplier( ) -> rgw::auth: :SysReqApplier -> rgw::auth: :RemoteApplier( acct_user= ad9fa6cacbb94b9 6b4941ec2af38a4 9f, acct_name= xxxxx_mgmt, perm_mask=15, is_admin=0), owner=ad9fa6cac bb94b96b4941ec2 af38a49f, perm=2 swift:PUT /swift/ v1/test/ cloudinit. txt:put_ obj:verifying op params swift:PUT /swift/ v1/test/ cloudinit. txt:put_ obj:pre- executing swift:PUT /swift/ v1/test/ cloudinit. txt:put_ obj:executing 179:0/141308436 5 --> 252.0.90. 230:6800/ 17596 -- osd_op( unknown. 0.0:132198 6.0 6:21b9d9f7: ::.dir. c9c6424e- a4bd-44de- b972-6cd4539a0a 61.5322. 1:head [call rgw.bucket_list] snapc 0=[] ondisk+ read+known_ if_redirected e80) v8 -- 0x5587e745a9c0 con 0 179:0/141308436 5 >> 252.0.90. 230:6800/ 17596 conn(0x5587e6c50300 :-1 s=STATE_ OPEN_MESSAGE_ READ_FOOTER_ AND_DISPATCH pgs=10 cs=1 l=1). rx osd.1 seq 54313 0x5587e745a9c0 osd_op_reply(132198 .dir.c9c6424e- a4bd-44de- b972-6cd4539a0a 61.5322. 1 [call] v0'0 uv1 ondisk = 0) v8 179:0/141308436 5 <== osd.1 252.0.90. 230:6800/ 17596 54313 ==== osd_op_reply(132198 .dir.c9c6424e- a4bd-44de- b972-6cd4539a0a 61.5322. 1 [call] v0'0 uv1 ondisk = 0) v8 ==== 192+0+71 (1074935246 0 1943469035) 0x5587e745a9c0 con 0x5587e6c50300 auth_completion name:civetweb- worker
-27> 2018-11-26 12:17:31.262 7f87d596b700 5 -- 252.0.89.
-26> 2018-11-26 12:17:31.262 7f87d596b700 1 -- 252.0.89.
-25> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001665:
-24> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001701:
-23> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001755:
-22> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001779:
-21> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001786:
-20> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001788:
-19> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001789:
-18> 2018-11-26 12:17:31.262 7f87b633a700 5 content type wasn't provided, trying to guess
-17> 2018-11-26 12:17:31.262 7f87b633a700 5 Searching permissions for identity=
-16> 2018-11-26 12:17:31.262 7f87b633a700 5 Searching permissions for uid=ad9fa6cacbb
-15> 2018-11-26 12:17:31.262 7f87b633a700 5 Found permission: 15
-14> 2018-11-26 12:17:31.262 7f87b633a700 5 Searching permissions for uid=ad9fa6cacbb
-13> 2018-11-26 12:17:31.262 7f87b633a700 5 Permissions for user not found
-12> 2018-11-26 12:17:31.262 7f87b633a700 5 Searching permissions for group=1 mask=50
-11> 2018-11-26 12:17:31.262 7f87b633a700 5 Permissions for group not found
-10> 2018-11-26 12:17:31.262 7f87b633a700 5 Searching permissions for group=2 mask=50
-9> 2018-11-26 12:17:31.262 7f87b633a700 5 Permissions for group not found
-8> 2018-11-26 12:17:31.262 7f87b633a700 5 -- Getting permissions done for identity=
-7> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001818:
-6> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001820:
-5> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001821:
-4> 2018-11-26 12:17:31.262 7f87b633a700 1 -- 252.0.89.
-3> 2018-11-26 12:17:31.266 7f87d616c700 5 -- 252.0.89.
-2> 2018-11-26 12:17:31.266 7f87d616c700 1 -- 252.0.89.
-1> 2018-11-26 12:17:31.266 7f87b633a700 5 NOTICE: call to do_aws4_
0> 2018-11-26 12:17:31.266 7f87b633a700 -1 *** Caught signal (Aborted) **
in thread 7f87b633a700 thread_
ceph version 13.2.1 (5533ecdc0fda92 0179d7ad84e0aa6 5a127b20d77) mimic (stable) t::get_ val<Option: :size_t> (std::_ _cxx11: :basic_ string< char, std::char_ traits< char>, std::allocator< char> > const&) const+0x53) [0x5587e461fab3] :execute( )+0x1adb) [0x5587e4600efb] authenticated( RGWHandler_ REST*, RGWOp*&, RGWRequest*, req_state*, bool)+0x1cd) [0x5587e4620cfd] request( RGWRados* , RGWREST*, RGWRequest*, std::__ cxx11:: basic_string< char, std::char_ traits< char>, std::allocator< char> > const&, rgw::auth: :StrategyRegist ry const&, RGWRestfulIO*, OpsLogSocket*, int*)+0x15f0) [0x5587e46223f0] ntend:: process( mg_connection* )+0x381) [0x5587e44976e1]
1: (()+0x293e70) [0x5587e4514e70]
2: (()+0x12890) [0x7f87eb99d890]
3: (gsignal()+0xc7) [0x7f87e0e96e97]
4: (abort()+0x141) [0x7f87e0e98801]
5: (()+0x8c8b7) [0x7f87e14ed8b7]
6: (()+0x92a06) [0x7f87e14f3a06]
7: (()+0x92a41) [0x7f87e14f3a41]
8: (()+0x92c74) [0x7f87e14f3c74]
9: (()+0x35e30d) [0x5587e45df30d]
10: (Option::size_t const md_config_
11: (()+0x37df4f) [0x5587e45fef4f]
12: (RGWPutObj:
13: (rgw_process_
14: (process_
15: (RGWCivetWebFro
16: (()+0x2851ee) [0x5587e45061ee]
17: (()+0x286e78) [0x5587e4507e78]
18: (()+0x287308) [0x5587e4508308]
19: (()+0x76db) [0x7f87eb9926db]
20: (clone()+0x3f) [0x7f87e0f7988f]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels --- ceph/radosgw. log
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
1/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 1 reserver
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/ 5 rgw_sync
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
4/ 5 rocksdb
4/ 5 leveldb
4/ 5 memdb
1/ 5 kinetic
1/ 5 fuse
1/ 5 mgr
1/ 5 mgrc
1/ 5 dpdk
1/ 5 eventtrace
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/
--- end dump of recent events ---
ProblemType: Crash dfsg1-0ubuntu2~ cloud0 [origin: Canonical] ature: Ubuntu 4.15.0-1024.25-gcp 4.15.18
"impl" : "launchpad",
"project" : "cloud-archive",
"bug_ pattern_ url": "http:// people. canonical. com/~ubuntu- archive/ bugpatterns/ bugpatterns. xml", radosgw. gateway unix/sysv/ linux/raise. c:51 x86_64- linux-gnu/ libstdc+ +.so.6 x86_64- linux-gnu/ libstdc+ +.so.6 x86_64- linux-gnu/ libstdc+ +.so.6
DistroRelease: Ubuntu 18.04
Package: radosgw 13.2.1+
ProcVersionSign
Uname: Linux 4.15.0-1024-gcp x86_64
ApportVersion: 2.20.9-0ubuntu7.5
Architecture: amd64
CrashDB:
{
}
Date: Mon Nov 26 12:17:31 2018
ExecutablePath: /usr/bin/radosgw
ProcCmdline: /usr/bin/radosgw -n client.
ProcEnviron:
PATH=(custom, no user)
LANG=C.UTF-8
Signal: 6
SourcePackage: ceph
StacktraceTop:
__GI_raise (sig=sig@entry=6) at ../sysdeps/
__GI_abort () at abort.c:79
?? () from /usr/lib/
?? () from /usr/lib/
std::terminate() () from /usr/lib/
Title: radosgw crashed with SIGABRT in __GI_raise()
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: