swift-proxy-server is not working when rabbitmq-server service is stopped

Bug #1661511 reported by Swati Bansal on 2017-02-03
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
ceilometermiddleware
Undecided
Unassigned

Bug Description

In NormalCase execution if we have environment without ceilometer component and trying to upload the object using command "swift upload abc clean.sh" it is uploaded successfully whether rabbitmq-server service is in "inactive" or "active" state. But when we have environment with ceilometer component and if rabbitmq-server service is unavailable and then run the command "swift upload abc clean.sh", the system hangs and if we check the status of swift-proxy service, it is in active state but it is not working. Swift-proxy service requests for AMQP server repeatedly and CLI user doesn't receive any kind of response. After some time when the rabbitmq-server service is up, swift-proxy-server will successfully upload the object.

[root@controller ~(keystone_admin)]# systemctl status openstack-swift-proxy.service
? openstack-swift-proxy.service - OpenStack Object Storage (swift) - Proxy Server
   Loaded: loaded (/usr/lib/systemd/system/openstack-swift-proxy.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2017-02-02 14:51:59 IST; 21h ago
Main PID: 1397 (swift-proxy-ser)
   CGroup: /system.slice/openstack-swift-proxy.service
           +-1397 /usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
           +-3448 /usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf

Feb 03 11:53:26 controller proxy-server[3448]: STDERR: ERROR:oslo.messaging._drivers.impl_rabbit:[74ef04b2-fa35-42ed-a3dd-9464c640a731] AMQP server on 10.0...port: None
Feb 03 11:53:48 controller proxy-server[3448]: STDERR: ERROR:oslo.messaging._drivers.impl_rabbit:[74ef04b2-fa35-42ed-a3dd-9464c640a731] AMQP server on 10.0...port: None
Feb 03 11:54:12 controller proxy-server[3448]: ERROR with Account server 10.0.2.15:6002/swiftloopback re: Trying to HEAD /v1/AUTH_5436c00da9424368ac4afbd9e...0589421ca)
Feb 03 11:54:12 controller proxy-server[3448]: STDERR: ERROR:oslo.messaging._drivers.impl_rabbit:[74ef04b2-fa35-42ed-a3dd-9464c640a731] AMQP server on 10.0...port: None
Feb 03 11:54:38 controller proxy-server[3448]: Account HEAD returning 503 for [] (txn: txcfc5817381f5453584815-00589421ca)
Feb 03 11:54:38 controller proxy-server[3448]: STDERR: ERROR:root:Timeout talking to memcached: 127.0.0.1:11211 (txn: txf27f47c35a744193a5247-00589421f4)
Feb 03 11:54:38 controller proxy-server[3448]: Deferring reject downstream
Feb 03 11:54:38 controller proxy-server[3448]: STDERR: ERROR:oslo.messaging._drivers.impl_rabbit:[74ef04b2-fa35-42ed-a3dd-9464c640a731] AMQP server on 10.0...port: None
Feb 03 11:55:06 controller proxy-server[3448]: STDERR: ERROR:root:Timeout talking to memcached: 127.0.0.1:11211 (txn: txcfc5817381f5453584815-00589421ca)
Feb 03 11:55:06 controller proxy-server[3448]: STDERR: ERROR:oslo.messaging._drivers.impl_rabbit:[74ef04b2-fa35-42ed-a3dd-9464c640a731] AMQP server on 10.0...port: None
Hint: Some lines were ellipsized, use -l to show in full.

And this is really wrong, since even if the status of the service shows *running*, the service is not working.

Correct behavior should be:
When rabbitmq-server service is unavailable, then swift-proxy-server should upload the object because rabbitmq-server service is used only for sending notification to ceilometer services. So execution should wait only after uploading the object, not in the starting.

Julien Danjou (jdanjou) on 2017-02-03
affects: ceilometer → ceilometermiddleware
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers