OpenStack Object Storage (swift)

grenade tests fail with error trying to create container

Bug #1209086 reported by Avishay Traeger
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Object Storage (swift)
Fix Released
Medium
Unassigned

Bug Description

2013-08-07 05:13:05.726 | No handlers could be found for logger "swiftclient"
2013-08-07 05:13:05.727 | Account HEAD failed: http://127.0.0.1:8080/v1/AUTH_f7ed2d7e936b4ced85d992de21a67b22 503 Internal Server Error
2013-08-07 05:13:05.843 | + swift post ex-swift
2013-08-07 05:13:06.419 | No handlers could be found for logger "swiftclient"
2013-08-07 05:13:06.442 | Container PUT failed: http://127.0.0.1:8080/v1/AUTH_f7ed2d7e936b4ced85d992de21a67b22/ex-swift 404 Not Found [first 60 chars of response] <html><h1>Not Found</h1><p>The resource could not be found.<
2013-08-07 05:13:06.567 | + swift upload ex-swift /etc/issue
2013-08-07 05:13:07.106 | No handlers could be found for logger "swiftclient"
2013-08-07 05:13:07.108 | Error trying to create container 'ex-swift': 404 Not Found: <html><h1>Not Found</h1><p>The resource could not be found.<
2013-08-07 05:13:07.367 | Object PUT failed: http://127.0.0.1:8080/v1/AUTH_f7ed2d7e936b4ced85d992de21a67b22/ex-swift/etc/issue 404 Not Found [first 60 chars of response] <html><h1>Not Found</h1><p>The resource could not be found.<
2013-08-07 05:13:07.492 | + die 54 'Failure uploading file to container ex-swift'
2013-08-07 05:13:07.493 | + local exitcode=1
2013-08-07 05:13:07.493 | + set +o xtrace
2013-08-07 05:13:07.531 | [ERROR] /opt/stack/new/devstack/exercises/swift.sh:54 Failure uploading file to container ex-swift

http://logs.openstack.org/72/40272/1/gate/gate-grenade-devstack-vm/8ad48f0/console.html

Revision history for this message
Chmouel Boudjnah (chmouel) wrote :

So I am not totally sure how grenade works, does it take a version released around grizzly do test do an update and do test and those test fails?

Revision history for this message
clayg (clay-gerrard) wrote :

There is no s-account log here:

http://logs.openstack.org/72/40272/1/gate/gate-grenade-devstack-vm/8ad48f0/logs/old/

that explains why the account request 503'd

but not why the account server didn't start?

Can you reproduce the error? What does `swift-init account-server start` say after the upgrade?

Revision history for this message
Chmouel Boudjnah (chmouel) wrote :

clay: On devstack we launch the service in foreground :

stack@devstack:/opt/stack/python-cinderclient$ cd /opt/stack/swift && /opt/stack/swift/bin/swift-account-server /etc/swift/account-server/1.conf -v || touch "/opt/stack/status/stack/s-account.failure"

this is the log of the screen session:

http://logs.openstack.org/72/40272/1/gate/gate-grenade-devstack-vm/8ad48f0/logs/old/screen-s-account.txt.gz

so it seems that it was running but we still getting the 503, I'd like to know as well what the context to reproduce that.

Revision history for this message
clayg (clay-gerrard) wrote :

http://logs.openstack.org/72/40272/1/gate/gate-grenade-devstack-vm/8ad48f0/logs/new/

^ Chmouel and I agree now this is the right link and account didn't start post upgrade

Would love some assistance.

Revision history for this message
Samuel Merritt (torgomatic) wrote :

Hm; without any account-server logs at all, it's really hard to say what happened. Is grenade even trying to start an account server?

Chmouel Boudjnah (chmouel)
Changed in swift:
status: New → Incomplete
importance: Undecided → Medium
Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for OpenStack Object Storage (swift) because there has been no activity for 60 days.]

Changed in swift:
status: Incomplete → Expired
Revision history for this message
Sean Dague (sdague) wrote :

3 hits in the last week on this in elastic recheck - http://logstash.openstack.org/#eyJzZWFyY2giOiJcIkZhaWx1cmUgdXBsb2FkaW5nIGZpbGUgdG8gY29udGFpbmVyIGV4LXN3aWZ0XCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjEzODk0OTEzMTc5MTl9

Changed in swift:
status: Expired → Confirmed
Revision history for this message
clayg (clay-gerrard) wrote :

Interesting. When I try to follow the logstash those error messages to their logs (e.g. log_url: http://logs.openstack.org/49/64749/7/check/check-grenade-dsvm/a4c71c0/console.html) I get a 404?

How can we help troubleshoot these failures? Is the problem still the account-server did not start? Does the client try to create the container and retry the upload when it gets the 404?

Revision history for this message
Peter Portante (peter-a-portante) wrote :

I think you can just append a .gz to the end.

Revision history for this message
clayg (clay-gerrard) wrote :

doh! thanks pete!

And we *are* missing the account screen again!?

http://logs.openstack.org/49/64749/7/check/check-grenade-dsvm/a4c71c0/logs/new/

I see s-account listed in enabled in in localrc

I see some weird messages in syslog that might relate to how swift is being shutdown/restarted? I can see SIGTERM and account server shutdown, but when I see the other services starting up I don't see the account-server? Who's driving that?

Revision history for this message
Alan Pevec (apevec) wrote :

> Who's driving that?

Just debugging similar issue where I don't get s-object in http://logs.openstack.org/05/62205/2/gate/gate-grenade-dsvm/86e638b/logs/new/
NB this is for stable/havana change, so upgrade test is grizzly->havana

The driver is grenade script: https://github.com/openstack-dev/grenade/blob/master/grenade.sh
It installs "old" version devstack, runs exercise and stops[1] it.
Then "new" version is installed, db schema migrated (configs are kept unchanged), everything is started again and tempest smoketests executed.

[1] https://github.com/openstack-dev/grenade/blob/master/stop-base - kills devstack screens and runs devstack's stop_swift (from https://github.com/openstack-dev/devstack/blob/master/lib/swift )

Revision history for this message
Chmouel Boudjnah (chmouel) wrote : Re: [Bug 1209086] Re: grenade tests fail with error trying to create container

I *think* that should fix it :

https://github.com/openstack-dev/devstack/commit/bb8227ce69b9b040b98dbe339e4f5c02172d19ac

Revision history for this message
Chmouel Boudjnah (chmouel) wrote :

I *think* that should fix it :

https://github.com/openstack-dev/devstack/commit/bb8227ce69b9b040b98dbe339e4f5c02172d19ac

Revision history for this message
Peter Portante (peter-a-portante) wrote :

@Chmouel, not sure where that "type" variable is used, but that pkill is gonna all the swift processes four times. :)

Revision history for this message
Chmouel Boudjnah (chmouel) wrote :

Yeah right, so I guess that's better than not being killed :) as before we could not unmount cleanly the loop file system since it was busy with a swift process attached to it.

Revision history for this message
Joe Gordon (jogo) wrote :

We haven't seen this in a while in CI

John Dickinson (notmyname)
Changed in swift:
status: Confirmed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.