reverse dns leads to low response of all api in some network

[ZhiQiang Fan]

when test ceilometer havana on sles 11 sp3, I found that there is about 10 seconds latency in each request, while nova, neutron does not

I found it is because ceilometer-api uses wsgiref.simple_server, which depends on BaseHTTPServer, which will do a dns query when log message (I'm not sure whether I understand it properly)

even in the controller node, if I use ip address instead of domain name to query the api, it still has 10 seconds latency, because it try to resolve the domain name of the ip address...

simply patch the BaseHTTPServer.BaseHTTPRequestHandler.address_string will solve it.

Is there anyone else has come across such weird problem?

[ZhiQiang Fan (aji-zqfan)]

# time curl -i -X GET -H 'X-Auth-Token: 49d692370e1f49cfab3e14b041b2f5d8' -H 'Content-Type: application/json' -H 'Accept: application/json' -H 'User-Agent: python-ceilometerclient' http://160.132.0.12:8777/v2/alarms
real 0m10.081s
user 0m0.004s
sys 0m0.004s
# tcpdump port 8777
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
16:04:48.719846 IP 160.132.0.12.42549 > controller.8777: Flags [S], seq 3154548310, win 14600, options [mss 1460,sackOK,TS val 146248549 ecr 0,nop,wscale 7], length 0
16:04:48.719929 IP controller.8777 > 160.132.0.12.42549: Flags [S.], seq 1919198764, ack 3154548311, win 14480, options [mss 1460,sackOK,TS val 638406787 ecr 146248549,nop,wscale 7], length 0
16:04:48.720033 IP 160.132.0.12.42549 > controller.8777: Flags [.], ack 1, win 115, options [nop,nop,TS val 146248549 ecr 638406787], length 0
16:04:48.720099 IP 160.132.0.12.42549 > controller.8777: Flags [P.], seq 1:196, ack 1, win 115, options [nop,nop,TS val 146248549 ecr 638406787], length 195
16:04:48.720115 IP controller.8777 > 160.132.0.12.42549: Flags [.], ack 196, win 122, options [nop,nop,TS val 638406787 ecr 146248549], length 0
2014-03-11 16:04:58.732 18263 DEBUG ceilometer.api.controllers.v2 [-] zqfan: enter alarm list get_all /usr/lib/python2.7/dist-packages/ceilometer/api/controllers/v2.py:1491
2014-03-11 16:04:58.732 18263 DEBUG ceilometer.openstack.common.policy [-] Rule context_is_admin will be now enforced enforce /usr/lib/python2.7/dist-packages/ceilometer/openstack/common/policy.py:253
2014-03-11 16:04:58.733 18263 DEBUG ceilometer.openstack.common.policy [-] Rule context_is_admin will be now enforced enforce /usr/lib/python2.7/dist-packages/ceilometer/openstack/common/policy.py:253
2014-03-11 16:04:58.734 18263 DEBUG ceilometer.openstack.common.policy [-] Rule context_is_admin will be now enforced enforce /usr/lib/python2.7/dist-packages/ceilometer/openstack/common/policy.py:253
2014-03-11 16:04:58.735 18263 DEBUG ceilometer.openstack.common.policy [-] Rule context_is_admin will be now enforced enforce /usr/lib/python2.7/dist-packages/ceilometer/openstack/common/policy.py:253
16:04:58.736818 IP controller.8777 > 160.132.0.12.42549: Flags [P.], seq 1:18, ack 196, win 122, options [nop,nop,TS val 638409291 ecr 146248549], length 17
16:04:58.736924 IP controller.8777 > 160.132.0.12.42549: Flags [.], seq 18:1466, ack 196, win 122, options [nop,nop,TS val 638409291 ecr 146248549], length 1448
16:04:58.736980 IP 160.132.0.12.42549 > controller.8777: Flags [.], ack 18, win 115, options [nop,nop,TS val 146251053 ecr 638409291], length 0
16:04:58.737013 IP controller.8777 > 160.132.0.12.42549: Flags [P.], seq 1466:1493, ack 196, win 122, options [nop,nop,TS val 638409291 ecr 146251053], length 27
16:04:58.737021 IP 160.132.0.12.42549 > controller.8777: Flags [.], ack 1466, win 137, options [nop,nop,TS val 146251053 ecr 638409291], length 0
16:04:58.737064 IP 160.132.0.12.42549 > controller.8777: Flags [.], ack 1493, win 137, options [nop,nop,TS val 146251053 ecr 638409291], length 0
16:04:58.737129 IP 160.132.0.12.42549 > controller.8777: Flags [F.], se...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/79876

[Openstack Gerrit (openstack-gerrit)]

Reviewed: https://review.openstack.org/79876
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=bad95ca2bc94aedbcf235f56d6a0c4c2c72b7d08
Submitter: Jenkins
Branch: master

commit bad95ca2bc94aedbcf235f56d6a0c4c2c72b7d08
Author: ZhiQiang Fan <email address hidden>
Date: Wed Mar 12 16:38:54 2014 +0800

    Disable reverse dns lookup

    The BaseHTTPServer will do a reverse dns lookup when log message, which
    will cause 10 seconds latency in some network environment. This will
    significantly slow down the api request when wsgiref.simple_server is
    used.

    This patch creates a class, which inherits from
    simple_server.WSGIRequestHandler but overrides address_string method,
    and passes it to handler_class param of simple_server.make_server.

    Change-Id: Ia38c7719ed700ab5867dcc4bcf3700bff282a907
    Closes-Bug: #1291229

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/363320

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/363320
Committed: https://git.openstack.org/cgit/openstack/cloudpulse/commit/?id=342265f9b8d1c462305107eaeb76af334c4514d5
Submitter: Jenkins
Branch: master

commit 342265f9b8d1c462305107eaeb76af334c4514d5
Author: Anand Shanmugam <email address hidden>
Date: Tue Aug 30 16:29:21 2016 -0700

    Disable reverse dns lookup

    The BaseHTTPServer will do a reverse dns lookup when log message, which
    will cause 10 seconds latency in some network environment. This will
    significantly slow down the api request when wsgiref.simple_server is
    used.

    Change-Id: I0c8de84665f9a68290a26874aa555a881e8f1ca2
    Closes-Bug: #1291229

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/363958

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/363958
Committed: https://git.openstack.org/cgit/openstack/cloudpulse/commit/?id=3a69e062fbd7d9e9d2a002c243d77dc8b796c2fb
Submitter: Jenkins
Branch: stable/liberty

commit 3a69e062fbd7d9e9d2a002c243d77dc8b796c2fb
Author: Anand Shanmugam <email address hidden>
Date: Tue Aug 30 16:29:21 2016 -0700

    Disable reverse dns lookup

    The BaseHTTPServer will do a reverse dns lookup when log message, which
    will cause 10 seconds latency in some network environment. This will
    significantly slow down the api request when wsgiref.simple_server is
    used.

    Change-Id: I0c8de84665f9a68290a26874aa555a881e8f1ca2
    Closes-Bug: #1291229
    (cherry picked from commit 342265f9b8d1c462305107eaeb76af334c4514d5)