statsd timing unhandled exception operation not permitted

[clayg]

    -container-5.2..com: May 22 22:49:25 container-server ERROR __ll__ error with DELETE /d84/58775/AUTH_benchmark/bench_000078/tiny_020610 :
    Traceback (most recent ll last):
      File "/opt//lib/python2.7/site-packages/swift/container/server.py", line 474, in __ll__
        res = method(req)
      File "/opt//lib/python2.7/site-packages/swift/common/utils.py", line 1458, in wrapped
        return func(*a, **kw)
      File "/opt//lib/python2.7/site-packages/swift/common/utils.py", line 480, in _timing_stats
        ctrl.logger.timing_since(method + '.timing', start_time)
      File "/opt//lib/python2.7/site-packages/swift/common/utils.py", line 610, in wrapped
        return func(self.logger.statsd_client, *a, **kw)
      File "/opt//lib/python2.7/site-packages/swift/common/utils.py", line 464, in timing_since
        sample_rate)
      File "/opt//lib/python2.7/site-packages/swift/common/utils.py", line 460, in timing
        return self._send(metric, timing_ms, 'ms', sample_rate)
      File "/opt//lib/python2.7/site-packages/swift/common/utils.py", line 445, in _send
        return sock.sendto('|'.join(parts), self._target)
      File "/opt//lib/python2.7/site-packages/eventlet/greenio.py", line 290, in sendto
        return self.fd.sendto(*args)
    error: [Errno 1] Operation not permitted (txn: tx60ec62fa962d4d96ba5dace5d81eede2)

Just need to audit the error handling, I think this happened under high load and the network was saturated maybe unable to send new udp packets or something...

[Samuel Merritt (torgomatic)]

Could be nf_conntrack's connection tables being full, at least according to http://stackoverflow.com/questions/6240951/sendto-operation-not-permitted-netsnmp

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/113180
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=1bc4fe891a359c626ebc8049403d4d72a75b6be7
Submitter: Jenkins
Branch: master

commit 1bc4fe891a359c626ebc8049403d4d72a75b6be7
Author: John Dickinson <email address hidden>
Date: Sun Aug 10 23:50:17 2014 -0700

    Catch permissions errors when writing StatsD packets

    Closes-Bug: #1183152

    Change-Id: I4b2c6e947241c987779a385fdff270d037470a57

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: feature/ec
Review: https://review.openstack.org/114651

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/114651
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=343ab4f757f52cce0b2b12e810dcb7ef578767e1
Submitter: Jenkins
Branch: feature/ec

commit 1bc4fe891a359c626ebc8049403d4d72a75b6be7
Author: John Dickinson <email address hidden>
Date: Sun Aug 10 23:50:17 2014 -0700

    Catch permissions errors when writing StatsD packets

    Closes-Bug: #1183152

    Change-Id: I4b2c6e947241c987779a385fdff270d037470a57

commit 41d851387cec122f4795d447458fd81e48e256b0
Author: Takashi Kajinami <email address hidden>
Date: Wed Apr 9 16:33:02 2014 +0900

    Add timestamp checking in AccountBroker.is_status_deleted

    Account-reaper works only at account-server with the first replica, and reaps
    account with "deleted" status.

    On the other hand, account-replicator doesn't replicate the status, only
    replicates *_timestamp.
    When swift fails to delete the first account replica, account-reaper never
    reaps the account, because the first replica never gets marked as "deleted".

    This patch adds a timestamp checking into is_status_deleted method, and
    account-reaper will start to reap the account after account-replicator
    replicates *_timestamp.

    Change-Id: I75e3f15ad217a71b4fd39552cf6db2957597efca
    Closes-Bug: #1304755

commit 048d46e609c7a2d3d1a837875f6787dddf03c8be
Author: Thiago da Silva <email address hidden>
Date: Fri Aug 1 15:31:03 2014 -0400

    fix to handle object-expiration headers in one method

    There was some duplication of code in both POST and PUT
    methods to handle object-expiration headers.
    A method was created to remove this duplication,
    which should help with maintainability of code.

    Change-Id: I85cc4a7b0d688760c97598d80b9e9a39288c5f34
    Signed-off-by: Thiago da Silva <email address hidden>

commit 134e864fa132b4d29e8fed6c54526cf1777307b2
Author: Samuel Merritt <email address hidden>
Date: Fri Jul 11 11:27:11 2014 -0700

    Add POST and DELETE to tempurl default methods

    The tempurl middleware supports any configured HTTP methods, but the
    default set was only GET, PUT, and HEAD, so cluster operators had to
    take action to enable POST and DELETE. This commit changes the
    defaults to include POST and DELETE.

    Note that this doesn't affect any existing temporary URLs at all; the
    method is baked into the signature (temp_url_sig query param), so no
    new access is granted to a holder of a temporary URL by this
    change. It simply gives more flexibility to creators of temporary
    URLs.

    Change-Id: I5bc15bbd2968ab7bedcd7c0df10f2ec825537191

commit 3e04606f87c8cb3207ffeb717a6f5407b77b4d12
Author: Peter Portante <email address hidden>
Date: Tue May 6 10:54:05 2014 -0400

    Add a env var to use in-memory obj server in func

    Add an environment variable to enable the use of the in-memory object
    server during in-process functional test runs.

    It might be worth-while to just run under both object servers in-tree,
    but this at least enables it, without having to figure out how to make
    two test runs in two different environments.

    D...