Swift is not using memcache

[Sergey Arkhipov]

Swift is not using memcache

MOS 8.0 build #496

I observe a lot of errors of container-reconciler Swift process trying to connect to memcached in syslog

<131>Feb 10 11:19:57 node-13 container-reconciler: STDERR: ERROR:root:Error connecting to memcached: 127.0.0.1:11211#012Traceback (most recent call last):#012 File "/usr/lib/python2.7/dist-packages/swift/common/memcached.py", line 209, in _get_conns#012 fp, sock = self._client_cache[server].get()#012 File "/usr/lib/python2.7/dist-packages/swift/common/memcached.py", line 127, in get#012 fp, sock = self.create()#012 File "/usr/lib/python2.7/dist-packages/swift/common/memcached.py", line 120, in create#012 sock.connect((host, int(port)))#012 File "/usr/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 240, in connect#012 socket_checkerr(fd)#012 File "/usr/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 44, in socket_checkerr#012 raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: tx7d52329d79644a6db2b37-0056bb1cdd)
<131>Feb 10 11:19:57 node-13 container-reconciler: STDERR: ERROR:root:Error connecting to memcached: 127.0.0.1:11211#012Traceback (most recent call last):#012 File "/usr/lib/python2.7/dist-packages/swift/common/memcached.py", line 209, in _get_conns#012 fp, sock = self._client_cache[server].get()#012 File "/usr/lib/python2.7/dist-packages/swift/common/memcached.py", line 127, in get#012 fp, sock = self.create()#012 File "/usr/lib/python2.7/dist-packages/swift/common/memcached.py", line 120, in create#012 sock.connect((host, int(port)))#012 File "/usr/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 240, in connect#012 socket_checkerr(fd)#012 File "/usr/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 44, in socket_checkerr#012 raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: tx7d52329d79644a6db2b37-0056bb1cdd)
<131>Feb 10 11:19:57 node-13 container-reconciler: STDERR: ERROR:root:Error connecting to memcached: 127.0.0.1:11211#012Traceback (most recent call last):#012 File "/usr/lib/python2.7/dist-packages/swift/common/memcached.py", line 209, in _get_conns#012 fp, sock = self._client_cache[server].get()#012 File "/usr/lib/python2.7/dist-packages/swift/common/memcached.py", line 127, in get#012 fp, sock = self.create()#012 File "/usr/lib/python2.7/dist-packages/swift/common/memcached.py", line 120, in create#012 sock.connect((host, int(port)))#012 File "/usr/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 240, in connect#012 socket_checkerr(fd)#012 File "/usr/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 44, in socket_checkerr#012 raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: tx80e21f6346674fb28ad02-0056bb1cdd)

Actually, memcached is bind to another IP address

root@node-13:/etc/swift# netstat -lntp | grep memcached
tcp 0 0 192.168.0.8:11211 0.0.0.0:* LISTEN 12981/memcached

Environment:
    * 10 baremetal nodes
    * Network - Neutron with VLAN segmentation
    * Storage Backends - Cinder LVM over iSCSI for volumes (Ceph is not used)

[root@fuel ~]# fuel node
id | status | name | cluster | ip | mac | roles | pending_roles | online | group_id
---|--------|---------------|---------|------------|-------------------|----------------------|---------------|--------|---------
16 | ready | elasticsearch | 2 | 10.20.0.7 | 0c:c4:7a:1d:92:54 | elasticsearch_kibana | | True | 2
20 | ready | controller_1 | 2 | 10.20.0.8 | 0c:c4:7a:1d:ff:2c | controller | | True | 2
11 | ready | influxdb | 2 | 10.20.0.11 | 0c:c4:7a:1e:9f:0e | influxdb_grafana | | True | 2
13 | ready | controller_0 | 2 | 10.20.0.10 | 0c:c4:7a:1e:3e:fc | controller | | True | 2
14 | ready | compute_0 | 2 | 10.20.0.5 | 0c:c4:7a:1d:92:ba | cinder, compute | | True | 2
12 | ready | controller_2 | 2 | 10.20.0.13 | 00:25:90:eb:d7:ee | controller | | True | 2
17 | ready | compute_4 | 2 | 10.20.0.6 | 0c:c4:7a:1d:93:44 | cinder, compute | | True | 2
19 | ready | compute_1 | 2 | 10.20.0.12 | 0c:c4:7a:1e:9d:40 | cinder, compute | | True | 2
15 | ready | compute_3 | 2 | 10.20.0.4 | 00:25:90:eb:d8:24 | cinder, compute | | True | 2
18 | ready | compute_2 | 2 | 10.20.0.9 | 0c:c4:7a:1e:41:9c | cinder, compute | | True | 2

The same error is observed on node-13, node-12 and node-20 (all controllers).

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "529"
  build_id: "529"
  fuel-nailgun_sha: "baec8643ca624e52b37873f2dbd511c135d236d9"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "658be72c4b42d3e1436b86ac4567ab914bfb451b"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "e2d79330d5d708796330fac67722c21f85569b87"
  fuel-ostf_sha: "3bc76a63a9e7d195ff34eadc29552f4235fa6c52"
  fuel-mirror_sha: "fb45b80d7bee5899d931f926e5c9512e2b442749"
  fuelmenu_sha: "e071216cb214e34b4d861478033425ee6a54a3be"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "a43cf96cd9532f10794dce736350bf5bed350e9d"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "a365f05b903368225da3fea9aa42afc1d50dc9b4"

I've attached /etc/swift content of node-13, please check

[Matthew Mosesohn (raytrac3r)]

Please include diagnostic snapshot

[Sergey Arkhipov (sarkhipov)]

Diagnostic snapshot:
https://drive.google.com/a/mirantis.com/file/d/0B9tzODpFABxkOHhZNFR2R2ZMTzQ/view?usp=sharing

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/279498

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/279498
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=3a7e93c5c6c45c68882187cde65a6b446460e042
Submitter: Jenkins
Branch: master

commit 3a7e93c5c6c45c68882187cde65a6b446460e042
Author: Kyrylo Galanov <email address hidden>
Date: Fri Feb 12 13:53:00 2016 +0200

    Swift must use common memcached resource

    Swift will now use correct memcached interfaces.

    Change-Id: I6ccf4d5c70cd82ef8e3378ae34d316d6f5bfe364
    Closes-Bug: #1544007

[Kyrylo Galanov (kgalanov)]

Verified on ISO #465 RC1