OpenStack Compute (Nova)

ec2 metadata service extremely unreliable (sometimes)

Reported by Scott Moser on 2011-12-12
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
High
Jesse Andrews

Bug Description

$ for x in $(seq 1 10); do
  echo -n $x:;
  wget -q http://169.254.169.254/latest/meta-data -O x; echo;
  sleep 1;
done
1:
2:wget: server returned error: HTTP/1.1 500 Internal Server Error
3:wget: server returned error: HTTP/1.1 500 Internal Server Error
4:wget: server returned error: HTTP/1.1 500 Internal Server Error
5:wget: server returned error: HTTP/1.1 500 Internal Server Error
6:wget: server returned error: HTTP/1.1 500 Internal Server Error
7:wget: server returned error: HTTP/1.1 500 Internal Server Error
8:wget: server returned error: HTTP/1.1 500 Internal Server Error
9:
10:

7 out of 10 times, I got a 500 error. the other 3 times it responded
with the correct output (a list of items).

This is with devstack deployed system up to date as of this morning, and nova tip at 1b4a0f859e76de4c6e5cb7b51a12e9bf2fb3e295 .

Tags: ec2 Edit Tag help
Chuck Short (zulcss) on 2011-12-13
tags: added: ec2
Todd Willey (xtoddx) wrote :

https://review.openstack.org/#change,809

I have a very out of date change to fix this. I can re-work it soon to apply cleanly and solve this.
This is also not the first time you've reported this bug? https://bugs.launchpad.net/nova/+bug/851159

Scott Moser (smoser) wrote :

@Todd,
  Unreliable (this bug) and slow (bug 851159) are 2 very different things.
  This bug I saw 7 out of 10 attempts to get something from the metadata service fail with 500 error.

   Prior to that, to my knowledge I'd never seen a 500, just a very slow response.

Thierry Carrez (ttx) on 2011-12-13
Changed in nova:
importance: Undecided → Critical
status: New → Confirmed
Thierry Carrez (ttx) wrote :

Actually this can't be reproduced easily.

Changed in nova:
importance: Critical → High
summary: - ec2 metadata service extremely unreliable
+ ec2 metadata service extremely unreliable (sometimes)
Scott Moser (smoser) wrote :

I'm not able to easily reproduce this unfortunately.

I had seen it in nova on tip in a devstack install from sometime yesterday.

The only thing that I did that seemed to be maybe funny was I had added some images, run some instances, added some images with duplicate glance names, deleted them, and run some more instances.

I think that it is possible that the image that the instance had booted from had been deleted, but that would'nt explain to me the transient results that I saw.

Jesse Andrews (anotherjesse) wrote :

with https://review.openstack.org/#change,5363 I'm able to do:

    ab -c 5 -n 100 http://169.254.169.254/latest/meta-data

and get consistent responses and speed (the first request takes 200ms or so)

Changed in nova:
assignee: nobody → Jesse Andrews (anotherjesse)
milestone: none → essex-rc1
Changed in nova:
status: Confirmed → Fix Committed
Thierry Carrez (ttx) on 2012-03-20
Changed in nova:
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2012-04-05
Changed in nova:
milestone: essex-rc1 → 2012.1
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers