Comment 0 for bug 1805145

Revision history for this message
Ed Stewart (emcs2) wrote :

Hi,

Juju deployed OpenStack using a modified openstack-core bundle - Rocky on bionic. All components (except neutron-gateway) are deployed in LXD containers.

Horizon/CLI can list and create new containers, however, if a small file (in this case a cloud.init file of 82 bytes) is uploaded, the radosgw process crashes with attached crash file.

NB this is running within GCP

Please advise if any further information required. Here's the snippet out of the end of radosgw.log

d9fa6cacbb94b96b4941ec2af38a49f$ad9fa6cacbb94b96b4941ec2af38a49f:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e80) v8 -- 0x5587e73f9740 con 0
   -27> 2018-11-26 12:17:31.262 7f87d596b700 5 -- 252.0.89.179:0/1413084365 >> 252.0.80.134:6800/7301 conn(0x5587e6c51800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.0 seq 10832 0x5587e73f9740 osd_op_reply(132197 ad9fa6cacbb94b96b4941ec2af38a49f$ad9fa6cacbb94b96b4941ec2af38a49f [getxattrs,stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8
   -26> 2018-11-26 12:17:31.262 7f87d596b700 1 -- 252.0.89.179:0/1413084365 <== osd.0 252.0.80.134:6800/7301 10832 ==== osd_op_reply(132197 ad9fa6cacbb94b96b4941ec2af38a49f$ad9fa6cacbb94b96b4941ec2af38a49f [getxattrs,stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 251+0+0 (2451902153 0 0) 0x5587e73f9740 con 0x5587e6c51800
   -25> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001665:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:normalizing buckets and tenants
   -24> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001701:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:init permissions
   -23> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001755:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:recalculating target
   -22> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001779:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:reading permissions
   -21> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001786:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:init op
   -20> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001788:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:verifying op mask
   -19> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001789:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:verifying op permissions
   -18> 2018-11-26 12:17:31.262 7f87b633a700 5 content type wasn't provided, trying to guess
   -17> 2018-11-26 12:17:31.262 7f87b633a700 5 Searching permissions for identity=rgw::auth::ThirdPartyAccountApplier() -> rgw::auth::SysReqApplier -> rgw::auth::RemoteApplier(acct_user=ad9fa6cacbb94b96b4941ec2af38a49f, acct_name=xxxx_mgmt, perm_mask=15, is_admin=0) mask=50
   -16> 2018-11-26 12:17:31.262 7f87b633a700 5 Searching permissions for uid=ad9fa6cacbb94b96b4941ec2af38a49f
   -15> 2018-11-26 12:17:31.262 7f87b633a700 5 Found permission: 15
   -14> 2018-11-26 12:17:31.262 7f87b633a700 5 Searching permissions for uid=ad9fa6cacbb94b96b4941ec2af38a49f$ad9fa6cacbb94b96b4941ec2af38a49f
   -13> 2018-11-26 12:17:31.262 7f87b633a700 5 Permissions for user not found
   -12> 2018-11-26 12:17:31.262 7f87b633a700 5 Searching permissions for group=1 mask=50
   -11> 2018-11-26 12:17:31.262 7f87b633a700 5 Permissions for group not found
   -10> 2018-11-26 12:17:31.262 7f87b633a700 5 Searching permissions for group=2 mask=50
    -9> 2018-11-26 12:17:31.262 7f87b633a700 5 Permissions for group not found
    -8> 2018-11-26 12:17:31.262 7f87b633a700 5 -- Getting permissions done for identity=rgw::auth::ThirdPartyAccountApplier() -> rgw::auth::SysReqApplier -> rgw::auth::RemoteApplier(acct_user=ad9fa6cacbb94b96b4941ec2af38a49f, acct_name=xxxxx_mgmt, perm_mask=15, is_admin=0), owner=ad9fa6cacbb94b96b4941ec2af38a49f, perm=2
    -7> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001818:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:verifying op params
    -6> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001820:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:pre-executing
    -5> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001821:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:executing
    -4> 2018-11-26 12:17:31.262 7f87b633a700 1 -- 252.0.89.179:0/1413084365 --> 252.0.90.230:6800/17596 -- osd_op(unknown.0.0:132198 6.0 6:21b9d9f7:::.dir.c9c6424e-a4bd-44de-b972-6cd4539a0a61.5322.1:head [call rgw.bucket_list] snapc 0=[] ondisk+read+known_if_redirected e80) v8 -- 0x5587e745a9c0 con 0
    -3> 2018-11-26 12:17:31.266 7f87d616c700 5 -- 252.0.89.179:0/1413084365 >> 252.0.90.230:6800/17596 conn(0x5587e6c50300 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=10 cs=1 l=1). rx osd.1 seq 54313 0x5587e745a9c0 osd_op_reply(132198 .dir.c9c6424e-a4bd-44de-b972-6cd4539a0a61.5322.1 [call] v0'0 uv1 ondisk = 0) v8
    -2> 2018-11-26 12:17:31.266 7f87d616c700 1 -- 252.0.89.179:0/1413084365 <== osd.1 252.0.90.230:6800/17596 54313 ==== osd_op_reply(132198 .dir.c9c6424e-a4bd-44de-b972-6cd4539a0a61.5322.1 [call] v0'0 uv1 ondisk = 0) v8 ==== 192+0+71 (1074935246 0 1943469035) 0x5587e745a9c0 con 0x5587e6c50300
    -1> 2018-11-26 12:17:31.266 7f87b633a700 5 NOTICE: call to do_aws4_auth_completion
     0> 2018-11-26 12:17:31.266 7f87b633a700 -1 *** Caught signal (Aborted) **
 in thread 7f87b633a700 thread_name:civetweb-worker

 ceph version 13.2.1 (5533ecdc0fda920179d7ad84e0aa65a127b20d77) mimic (stable)
 1: (()+0x293e70) [0x5587e4514e70]
 2: (()+0x12890) [0x7f87eb99d890]
 3: (gsignal()+0xc7) [0x7f87e0e96e97]
 4: (abort()+0x141) [0x7f87e0e98801]
 5: (()+0x8c8b7) [0x7f87e14ed8b7]
 6: (()+0x92a06) [0x7f87e14f3a06]
 7: (()+0x92a41) [0x7f87e14f3a41]
 8: (()+0x92c74) [0x7f87e14f3c74]
 9: (()+0x35e30d) [0x5587e45df30d]
 10: (Option::size_t const md_config_t::get_val<Option::size_t>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const+0x53) [0x5587e461fab3]
 11: (()+0x37df4f) [0x5587e45fef4f]
 12: (RGWPutObj::execute()+0x1adb) [0x5587e4600efb]
 13: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, bool)+0x1cd) [0x5587e4620cfd]
 14: (process_request(RGWRados*, RGWREST*, RGWRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, int*)+0x15f0) [0x5587e46223f0]
 15: (RGWCivetWebFrontend::process(mg_connection*)+0x381) [0x5587e44976e1]
 16: (()+0x2851ee) [0x5587e45061ee]
 17: (()+0x286e78) [0x5587e4507e78]
 18: (()+0x287308) [0x5587e4508308]
 19: (()+0x76db) [0x7f87eb9926db]
 20: (clone()+0x3f) [0x7f87e0f7988f]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 1 lockdep
   0/ 1 context
   1/ 1 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 1 buffer
   0/ 1 timer
   0/ 1 filer
   0/ 1 striper
   0/ 1 objecter
   0/ 5 rados
   0/ 5 rbd
   0/ 5 rbd_mirror
   0/ 5 rbd_replay
   0/ 5 journaler
   0/ 5 objectcacher
   0/ 5 client
   1/ 5 osd
   0/ 5 optracker
   0/ 5 objclass
   1/ 3 filestore
   1/ 3 journal
   0/ 5 ms
   1/ 5 mon
   0/10 monc
   1/ 5 paxos
   0/ 5 tp
   1/ 5 auth
   1/ 5 crypto
   1/ 1 finisher
   1/ 1 reserver
   1/ 5 heartbeatmap
   1/ 5 perfcounter
   1/ 5 rgw
   1/ 5 rgw_sync
   1/10 civetweb
   1/ 5 javaclient
   1/ 5 asok
   1/ 1 throttle
   0/ 0 refs
   1/ 5 xio
   1/ 5 compressor
   1/ 5 bluestore
   1/ 5 bluefs
   1/ 3 bdev
   1/ 5 kstore
   4/ 5 rocksdb
   4/ 5 leveldb
   4/ 5 memdb
   1/ 5 kinetic
   1/ 5 fuse
   1/ 5 mgr
   1/ 5 mgrc
   1/ 5 dpdk
   1/ 5 eventtrace
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
  max_recent 10000
  max_new 1000
  log_file /var/log/ceph/radosgw.log
--- end dump of recent events ---

ProblemType: Crash
DistroRelease: Ubuntu 18.04
Package: radosgw 13.2.1+dfsg1-0ubuntu2~cloud0 [origin: Canonical]
ProcVersionSignature: Ubuntu 4.15.0-1024.25-gcp 4.15.18
Uname: Linux 4.15.0-1024-gcp x86_64
ApportVersion: 2.20.9-0ubuntu7.5
Architecture: amd64
CrashDB:
 {
                "impl": "launchpad",
                "project": "cloud-archive",
                "bug_pattern_url": "http://people.canonical.com/~ubuntu-archive/bugpatterns/bugpatterns.xml",
             }
Date: Mon Nov 26 12:17:31 2018
ExecutablePath: /usr/bin/radosgw
ProcCmdline: /usr/bin/radosgw -n client.radosgw.gateway
ProcEnviron:
 PATH=(custom, no user)
 LANG=C.UTF-8
Signal: 6
SourcePackage: ceph
StacktraceTop:
 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
 __GI_abort () at abort.c:79
 ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
 ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
 std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
Title: radosgw crashed with SIGABRT in __GI_raise()
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: