haproxy HTTP 500 error for Nova

[Alexander Evseev]

# fuel --fuel-version
api: '1.0'
astute_sha: 16b252d93be6aaa73030b8100cf8c5ca6a970a91
auth_required: true
build_id: 2014-12-26_14-25-46
build_number: '58'
feature_groups:
- mirantis
fuellib_sha: fde8ba5e11a1acaf819d402c645c731af450aff0
fuelmain_sha: 81d38d6f2903b5a8b4bee79ca45a54b76c1361b8
nailgun_sha: 5f91157daa6798ff522ca9f6d34e7e135f150a90
ostf_sha: a9afb68710d809570460c29d6c3293219d3624d4
production: docker
release: '6.0'
release_versions:
  2014.2-6.0:
    VERSION:
      api: '1.0'
      astute_sha: 16b252d93be6aaa73030b8100cf8c5ca6a970a91
      build_id: 2014-12-26_14-25-46
      build_number: '58'
      feature_groups:
      - mirantis
      fuellib_sha: fde8ba5e11a1acaf819d402c645c731af450aff0
      fuelmain_sha: 81d38d6f2903b5a8b4bee79ca45a54b76c1361b8
      nailgun_sha: 5f91157daa6798ff522ca9f6d34e7e135f150a90
      ostf_sha: a9afb68710d809570460c29d6c3293219d3624d4
      production: docker
      release: '6.0'

Baremetal, Ubuntu, HA, Neutron+GRE, Сeph-all, Debug, nova-quotas, 6.0-58
Controllers:3 Computes:97

Sample trace from Rally:

Traceback (most recent call last):
  File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/runners/base.py", line 77, in _run_scenario_once
    method_name)(**kwargs) or scenario_output
  File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/scenarios/nova/security_group.py", line 113, in boot_and_delete_server_with_secgroups
    rules_per_security_group)
  File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/scenarios/nova/utils.py", line 621, in _create_rules_for_security_group
    cidr=cidr)
  File "/opt/stack/.venv/lib/python2.7/site-packages/novaclient/v2/security_group_rules.py", line 70, in create
    'security_group_rule')
  File "/opt/stack/.venv/lib/python2.7/site-packages/novaclient/base.py", line 99, in _create
    _resp, body = self.api.client.post(url, body=body)
  File "/opt/stack/.venv/lib/python2.7/site-packages/novaclient/client.py", line 518, in post
    return self._cs_request(url, 'POST', **kwargs)
  File "/opt/stack/.venv/lib/python2.7/site-packages/novaclient/client.py", line 493, in _cs_request
    resp, body = self._time_request(url, method, **kwargs)
  File "/opt/stack/.venv/lib/python2.7/site-packages/novaclient/client.py", line 467, in _time_request
    resp, body = self.request(url, method, **kwargs)
  File "/opt/stack/.venv/lib/python2.7/site-packages/novaclient/client.py", line 461, in request
    raise exceptions.from_response(resp, body, url, method)
ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-ef1584e5-6efe-485f-8eb7-dfcfa8f8f3f6)

A lot of 500 errors in haproxy log for service "nova-api-2":

<134>Feb 26 05:14:46 node-46 haproxy[16488]: 172.16.44.10:51196 [26/Feb/2015:05:14:16.478] nova-api-2 nova-api-2/node-47 2/0/0/30028/30031 500 354 - - ---- 1111/1/1/1/0 0/0 "POST /v2/f1d1562e1f994bf2873b62246c09f8de/os-security-group-rules HTTP/1.1"
<134>Feb 26 05:14:46 node-46 haproxy[16488]: 172.16.44.10:51204 [26/Feb/2015:05:14:16.493] nova-api-2 nova-api-2/node-47 4/0/0/30022/30027 500 354 - - ---- 1110/0/0/0/0 0/0 "POST /v2/f1d1562e1f994bf2873b62246c09f8de/os-security-group-rules HTTP/1.1"
<134>Feb 26 05:16:29 node-46 haproxy[16488]: 172.16.44.10:54067 [26/Feb/2015:05:15:59.096] nova-api-2 nova-api-2/node-72 0/0/0/30027/30029 500 354 - - ---- 1149/17/17/5/0 0/0 "POST /v2/e086636f3a0e4dcfb3eb5a6f288ce129/os-security-groups HTTP/1.1"
<134>Feb 26 05:16:29 node-46 haproxy[16488]: 172.16.44.10:54068 [26/Feb/2015:05:15:59.155] nova-api-2 nova-api-2/node-47 0/0/0/30079/30079 500 354 - - ---- 1145/17/17/5/0 0/0 "POST /v2/f1d1562e1f994bf2873b62246c09f8de/os-security-group-rules HTTP/1.1"
<134>Feb 26 05:18:43 node-46 haproxy[16488]: 172.16.44.10:57833 [26/Feb/2015:05:18:13.008] nova-api-2 nova-api-2/node-46 26/0/0/30174/30201 500 354 - - ---- 1141/17/17/7/0 0/0 "POST /v2/f1d1562e1f994bf2873b62246c09f8de/os-security-group-rules HTTP/1.1"
<134>Feb 26 05:18:43 node-46 haproxy[16488]: 172.16.44.10:57846 [26/Feb/2015:05:18:13.059] nova-api-2 nova-api-2/node-46 0/0/0/30174/30175 500 354 - - ---- 1143/17/17/6/0 0/0 "POST /v2/e086636f3a0e4dcfb3eb5a6f288ce129/os-security-group-rules HTTP/1.1"
<134>Feb 26 05:21:28 node-46 haproxy[16488]: 172.16.44.10:34444 [26/Feb/2015:05:20:58.686] nova-api-2 nova-api-2/node-72 4/0/0/30025/30030 500 354 - - ---- 1117/2/2/0/0 0/0 "POST /v2/e086636f3a0e4dcfb3eb5a6f288ce129/os-security-groups HTTP/1.1"

Nova logs on all three controllers doesn't contain nor error 500, nor any requests with IDs from traces.

Snapshot will be later.

[Alexander Evseev (aevseev)]

Sorry. Cluster has keystone bug https://bugs.launchpad.net/mos/+bug/1413341