Wrong Swift config caused swift can't connect to memched

[JohnsonYi]

Environment: Fuel 8.0 with mu1 patch

Log:
<11>Sep 20 03:40:29 node-1 epmd: epmd: got partial packet only on file descriptor 5 (0)
<131>Sep 20 03:40:58 node-1 container-reconciler: STDERR: ERROR:root:Error connecting to memcached: 127.0.0.1:11211#012Traceback (most recent call last):#012 File "/usr/lib/python2.7/dist-packages/swift/common/memcached.py", line 209, in _get_conns#012 fp, sock = self._client_cache[server].get()#012 File "/usr/lib/python2.7/dist-packages/swift/common/memcached.py", line 127, in get#012 fp, sock = self.create()#012 File "/usr/lib/python2.7/dist-packages/swift/common/memcached.py", line 120, in create#012 sock.connect((host, int(port)))#012 File "/usr/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 240, in connect#012 socket_checkerr(fd)#012 File "/usr/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 44, in socket_checkerr#012 raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: tx1d6ab41d35fb4be1be2a9-0057e0afca)
<27>Sep 20 03:40:58 node-1 ocf-mysql-wss: ERROR: Not enough arguments [1] to ocf_log.
<27>Sep 20 03:40:58 node-1 ocf-mysql-wss: ERROR: Not enough arguments [1] to ocf_log.

swift connect to memcached with 127.0.0.1:1121, but iptables only support connection from br_mgmt ip for port 11211, and memcached also listen on br_mgmt ip for port 11211.

Impact：
Service swift-container-reconciler exhausted all sockets and the controller node will be offline for other nodes every two days(cluster with 100+ nodes),

Expected result:
No error like " STDERR: ERROR:root:Error connecting to memcached: 127.0.0.1:11211#"

Actual result:
Lots of errors like " STDERR: ERROR:root:Error connecting to memcached: 127.0.0.1:11211#", and controller node will be offline every two days.

Configuration in memcached.conf under /etc/swift/ should be corrected.
/etc/swift/memcached.conf
memcached_server = <br_mgmt_ip>:11211

[Vitaly Sedelnik (vsedelnik)]

JohnsonYi - please provide diagnostic snapshot and describe impact and expected result. Why swift needs to connect to memcached?

[JohnsonYi (yichengli)]

The impact is service swift-container-reconciler exhausted all sockets and the controller node will be offline for other nodes every two days(cluster with 100+ nodes),

Expected result:
No error like " STDERR: ERROR:root:Error connecting to memcached: 127.0.0.1:11211#"

[Rudy McComb (rmccomb)]

I have also experienced this issue.

<131>Feb 7 14:55:29 node-45 container-reconciler: STDERR: ERROR:root:Error connecting to memcached: 127.0.0.1:11211#012Traceback (most recent call last):#0
12 File "/usr/lib/python2.7/dist-packages/swift/common/memcached.py", line 209, in _get_conns#012 fp, sock = self._client_cache[server].get()#012 File "/usr/lib/python2.7/dist-p
ackages/swift/common/memcached.py", line 127, in get#012 fp, sock = self.create()#012 File "/usr/lib/python2.7/dist-packages/swift/common/memcached.py", line 120, in create#012
  sock.connect((host, int(port)))#012 File "/usr/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 240, in connect#012 socket_checkerr(fd)#012 File "/usr/lib/python2.7
/dist-packages/eventlet/greenio/base.py", line 44, in socket_checkerr#012 raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: tx02e810cda3714ee6
93709-005899dfe1)

ode-46 ocf-mysql-wss: ERROR: Not enough arguments [1] to ocf_log
node-46 object-updater: ERROR: Unable to access /srv/node: [Errno 2] No such file or directory: '/srv/node'

[Oleksiy Molchanov (omolchanov)]

Please provide diagnostic snapshot, marking as Incomplete