MemcacheRing logs ValueError: invalid literal for int() with base 10: 'NOT_FOUND'

[Alistair Coles]

We've seen tracebacks like:

```
proxy-server: err Error talking to memcached: 127.0.0.1:5556: #012Traceback (most recent call last):#012 File "/opt/ss/lib/python2.7/site-packages/swift/common/memcached.py", line 445, in incr#012 ret = int(line[0].strip())#012ValueError: invalid literal for int() with base 10: 'NOT_FOUND' (txn: tx49e7666440b2483d9b085-00643c8bbf)
```

The MemcacheRing.incr method first tries a memcached incr. This may fail if the key does not exist. A memcached add is then tried. This may fail if another actor has concurrently added the key. A further incr is then tried, which fails and raises the ValueError exception if the key has now expired.

We've only seen this in the context of a proprietary middleware that creates keys with relatively short TTLs, and very occasionally the MemcacheRing.incr thread executes slowly enough for the key to concurrently expire in this way.

[Alistair Coles (alistair-coles)]

https://review.opendev.org/c/openstack/swift/+/880677

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/swift/+/880677
Committed: https://opendev.org/openstack/swift/commit/49a5293cbd7d731eba816aebec47ffe65066dc51
Submitter: "Zuul (22348)"
Branch: master

commit 49a5293cbd7d731eba816aebec47ffe65066dc51
Author: Alistair Coles <email address hidden>
Date: Mon Apr 17 19:10:48 2023 +0100

    memcached: handle failed incr+add+incr better

    Improve the exception handling for the case when:
      * incr fails because key is not found
      * add fails because key is found (another actor added it
        concurrently)
      * incr fails because key is not found (the key expired since
        the add, presumably because the TTL was short relative to
        the execution time of the incr method)

    Previously this resulted in an error log and a traceback. This patch
    eliminates the traceback and adds the TTL to the log message.

    It's not clear that this failure scenario is due to a server problem:
    slow execution of the MemcacheRing.incr code path due to the green
    thread being suspended could cause the second incr to be delayed more
    than the TTL. This patch therefore prevents this particular failure
    scenario causing the server to be error limited.

    Closes-Bug: #2019095
    Change-Id: I3086e3453b13411304e431e83d6379e255c1777c

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/swift 2.32.0 release.