Requests to meta-data service do not timeout
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Canonical Juju |
Expired
|
Wishlist
|
Unassigned | ||
juju-core |
Won't Fix
|
Medium
|
Unassigned |
Bug Description
Description:
Requests made by juju to the Openstack metadata service do not timeout (or take too long to timeout) if the service stops responding. This eventually results in a DOS of this service.
How we discovered the issue:
We encountered a problem with Openstack (Folsom) metadata service not responding to requests. On closer inspection, we noticed that hundreds of TCP connections to the metadata service were "stuck" in the ESTABLISHED state causing the service not to respond to clients; effectively DOS'ing the service. After tracing the connections we noticed that they all originated from juju-core deployed instances. These instances all had multiple connections to the metadata service which seems unusual given that the service is intended for short-lived HTTP requests for simple instance metadata.
Conclusions and assumptions:
Without doubt the nova-api-metadata service has a bug and shouldn't maintain connections for so long, however juju should timeout connections if they don't respond after a period of time. We can't expect all services to be good citizens.
A quick search reveals that the lack of connection (and other) timeouts seem to be a common problem with the Go net http library.
System and other information:
I tried to capture relevant information before restarting the nova-api-metadata service.
ubuntu@
[...]
tcp 0 0 10.33.16.66:43086 169.254.169.254:80 ESTABLISHED 9467/jujud
tcp 0 0 10.33.16.66:43112 169.254.169.254:80 ESTABLISHED 15419/jujud
tcp 0 0 10.33.16.66:43120 169.254.169.254:80 ESTABLISHED 16558/jujud
tcp 0 0 10.33.16.66:43126 169.254.169.254:80 ESTABLISHED 17071/jujud
[...]
ubuntu@
1.13.2.
root@meta-
1002
root@meta-
23 10.33.16.241
23 10.33.16.63
24 10.33.16.231
24 10.33.16.233
24 10.33.16.234
24 10.33.16.235
24 10.33.16.236
24 10.33.16.66
25 10.33.16.120
30 10.33.16.128
Please let me know if you need futher information?
Changed in juju-core: | |
importance: | Undecided → High |
status: | New → Triaged |
tags: |
added: canonistack removed: prodstack |
tags: | added: preformance security |
tags: |
added: performance removed: preformance |
Changed in juju-core: | |
importance: | High → Medium |
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 2013-08-30 5:02, Ian Booth wrote:
> ** Changed in: juju-core Importance: Undecided => High
>
> ** Changed in: juju-core Status: New => Triaged
>
Unfortunately, the standard go idiom for handling a connection that
might block is:
go readFromSocketO ntoChannel( sock, fromSockChan) After(timeout) :
select {
case data := <-fromSockChan:
do stuff
case <-time.
do other stuff
}
AFAIK that means the readFromSockOnt oChannel never times out and stops
reading from the channel, we just stop waiting for the data to come
out of it.
Now it is possible that we should be doing:
case <-time. After(timeout) :
sock.Close() // We didn't hear back in time
// do other stuff
Or some other mechanism.
I don't see an explicit way to SetTimeout, but you do have SetDeadline golang. org/pkg/ net/#TCPConn. SetDeadline golang. org/pkg/ net/#Conn
and SetNoDelay
http://
The main problem with SetDeadline is that you have to continually
refresh it.
http://
Note that to implement timeout directly we have to wrap
net/http/Transport (which is a RoundTripper) and then wrap Dial to
return a net.Conn. We would then need to have a custom net.Conn that
did some sort of SetDeadline before every Read or Write call.
Client does seem to expose its Transport object, but Transport doesn't
expose its underlying connection cache for us to call Conn.SetDeadline
ourselves.
http:// stackoverflow. com/questions/ 16895294/ how-to- set-timeout- for-http- get-requests- in-golang
John www.enigmail. net/
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.13 (Cygwin)
Comment: Using GnuPG with Thunderbird - http://
iEYEARECAAYFAlI jLcYACgkQJdeBCY SNAAO6QQCggqeB1 1dGuBLTrNL8Bjcj aajv B+WHlV4ZLEXJKun 3L2gbxSx
rcUAn0bZ/
=J0L6
-----END PGP SIGNATURE-----