[vCenter]In HA mode on vCenter's machine haproxy returns 504 gateway timeout in attempts to get access to api of different services

Bug #1364898 reported by Tatyana Dubyk
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Invalid
Critical
Stepan Rogov

Bug Description

Fuel version:

api: '1.0'
astute_sha: bc60b7d027ab244039f48c505ac52ab8eb0a990c
auth_required: true
build_id: 2014-09-01_00-01-17
build_number: '491'
feature_groups:
- mirantis
fuellib_sha: 2cfa83119ae90b13a5bac6a844bdadfaf5aeb13f
fuelmain_sha: 109812be3425408dd7be192b5debf109cb1edd4c
nailgun_sha: d25ed02948a8be773e2bd87cfe583ef7be866bb2
ostf_sha: 4dcd99cc4bfa19f52d4b87ed321eb84ff03844da
production: docker
release: '5.1'
--------------------------------------------------

Steps to reproduce:
1. Create new environment (Ubuntu, HA mode)
     Network: Nova Network Flat DHCP

     setting for vcenter creation:
                    VCENTER_IP='172.16.0.254'
                    <email address hidden>'
                    VCENTER_PASSWORD='Qwer!1234'
                    VCENTER_CLUSTERS='Cluster1'

2. Add 3 nodes, all with the same roles:
   (1 controller+1 cinder) on each of node

3.Make deploy of environment , and then stop deploy on provisioning
   Then re-deploy again
4. Verify network connectivity
5. Run ostf

--------------------------------------------------

Traceback:
root@node-6:~# nova --debug image-list

REQ: curl -i 'http://192.168.0.3:5000/v2.0/tokens' -X POST -H "Content-Type: application/json" -H "Accept: application/json" -H "User-Agent: python-novaclient" -d '{"auth": {"tenantName": "admin", "passwordCredentials": {"username": "admin", "password": "admin"}}}'

New session created for: (http://192.168.0.3:5000)
INFO (connectionpool:202) Starting new HTTP connection (1): 192.168.0.3
DEBUG (connectionpool:296) "POST /v2.0/tokens HTTP/1.1" 200 3738
RESP: [200] CaseInsensitiveDict({'date': 'Wed, 03 Sep 2014 09:52:41 GMT', 'vary': 'X-Auth-Token', 'content-length': '3738', 'content-type': 'application/json', 'x-distribution': 'Ubuntu'})
RESP BODY: {"access": {"token": {"issued_at": "2014-09-03T09:52:41.700865", "expires": "2014-09-03T10:52:41Z", "id": "a3a7f6ca40244fc2bc048f8b6ddb19f5", "tenant": {"id": "69be24868b684045bc5907e6bc5e1d1c", "enabled": true, "name": "admin", "description": "admin tenant"}}, "serviceCatalog": [{"endpoints": [{"adminURL": "http://192.168.0.3:8774/v2/69be24868b684045bc5907e6bc5e1d1c", "region": "RegionOne", "internalURL": "http://192.168.0.3:8774/v2/69be24868b684045bc5907e6bc5e1d1c", "id": "52e65ead6f59469390f0cdc76d381442", "publicURL": "http://172.16.0.4:8774/v2/69be24868b684045bc5907e6bc5e1d1c"}], "endpoints_links": [], "type": "compute", "name": "nova"}, {"endpoints": [{"adminURL": "http://192.168.0.3:8776/v2/69be24868b684045bc5907e6bc5e1d1c", "region": "RegionOne", "internalURL": "http://192.168.0.3:8776/v2/69be24868b684045bc5907e6bc5e1d1c", "id": "12d169b5f3d74177bd5e4b278ead1097", "publicURL": "http://172.16.0.4:8776/v2/69be24868b684045bc5907e6bc5e1d1c"}], "endpoints_links": [], "type": "volumev2", "name": "cinderv2"}, {"endpoints": [{"adminURL": "http://192.168.0.3:8080", "region": "RegionOne", "internalURL": "http://192.168.0.3:8080", "id": "00dab2258da14bc891048fdf18ffcab7", "publicURL": "http://172.16.0.4:8080"}], "endpoints_links": [], "type": "s3", "name": "swift_s3"}, {"endpoints": [{"adminURL": "http://192.168.0.3:9292", "region": "RegionOne", "internalURL": "http://192.168.0.3:9292", "id": "335a6004cb7d4957a7b77b10c6a1113f", "publicURL": "http://172.16.0.4:9292"}], "endpoints_links": [], "type": "image", "name": "glance"}, {"endpoints": [{"adminURL": "http://192.168.0.3:8000/v1/", "region": "RegionOne", "internalURL": "http://192.168.0.3:8000/v1/", "id": "134777116f7f47fca56475c9439482b5", "publicURL": "http://172.16.0.4:8000/v1/"}], "endpoints_links": [], "type": "cloudformation", "name": "heat-cfn"}, {"endpoints": [{"adminURL": "http://192.168.0.3:8776/v1/69be24868b684045bc5907e6bc5e1d1c", "region": "RegionOne", "internalURL": "http://192.168.0.3:8776/v1/69be24868b684045bc5907e6bc5e1d1c", "id": "00dc482d00d94236bc25ca7480be55af", "publicURL": "http://172.16.0.4:8776/v1/69be24868b684045bc5907e6bc5e1d1c"}], "endpoints_links": [], "type": "volume", "name": "cinder"}, {"endpoints": [{"adminURL": "http://192.168.0.3:8773/services/Admin", "region": "RegionOne", "internalURL": "http://192.168.0.3:8773/services/Cloud", "id": "24ce95b62e034e578b72104113add2f4", "publicURL": "http://172.16.0.4:8773/services/Cloud"}], "endpoints_links": [], "type": "ec2", "name": "nova_ec2"}, {"endpoints": [{"adminURL": "http://192.168.0.3:8004/v1/69be24868b684045bc5907e6bc5e1d1c", "region": "RegionOne", "internalURL": "http://192.168.0.3:8004/v1/69be24868b684045bc5907e6bc5e1d1c", "id": "0aaf5a1c6d25424e8986e0a0fc19b2b9", "publicURL": "http://172.16.0.4:8004/v1/69be24868b684045bc5907e6bc5e1d1c"}], "endpoints_links": [], "type": "orchestration", "name": "heat"}, {"endpoints": [{"adminURL": "http://192.168.0.3:8080/", "region": "RegionOne", "internalURL": "http://192.168.0.3:8080/v1/AUTH_69be24868b684045bc5907e6bc5e1d1c", "id": "0b0246b74fb34d098fae9b55e5b5b340", "publicURL": "http://172.16.0.4:8080/v1/AUTH_69be24868b684045bc5907e6bc5e1d1c"}], "endpoints_links": [], "type": "object-store", "name": "swift"}, {"endpoints": [{"adminURL": "http://192.168.0.3:35357/v2.0", "region": "RegionOne", "internalURL": "http://192.168.0.3:5000/v2.0", "id": "8bc05fd779d845d491628d28bb859dcc", "publicURL": "http://172.16.0.4:5000/v2.0"}], "endpoints_links": [], "type": "identity", "name": "keystone"}], "user": {"username": "admin", "roles_links": [], "id": "71ee7b4697544c83863633f87d9ff8bb", "roles": [{"name": "admin"}], "name": "admin"}, "metadata": {"is_admin": 0, "roles": ["0e382a67c8ad44e0bd799ef833fc0f06"]}}}

REQ: curl -i 'http://172.16.0.4:8774/v2/69be24868b684045bc5907e6bc5e1d1c/images/detail' -X GET -H "X-Auth-Project-Id: admin" -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: a3a7f6ca40244fc2bc048f8b6ddb19f5"

New session created for: (http://172.16.0.4:8774)
INFO (connectionpool:202) Starting new HTTP connection (1): 172.16.0.4

DEBUG (connectionpool:296) "GET /v2/69be24868b684045bc5907e6bc5e1d1c/images/detail HTTP/1.1" 504 None
RESP: [504] CaseInsensitiveDict({'connection': 'close', 'content-type': 'text/html', 'cache-control': 'no-cache'})
RESP BODY: <html><body><h1>504 Gateway Time-out</h1>
The server didn't respond in time.
</body></html>

DEBUG (shell:777) <attribute 'message' of 'exceptions.BaseException' objects> (HTTP 504)
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/novaclient/shell.py", line 774, in main
    OpenStackComputeShell().main(map(strutils.safe_decode, sys.argv[1:]))
  File "/usr/lib/python2.7/dist-packages/novaclient/shell.py", line 710, in main
    args.func(self.cs, args)
  File "/usr/lib/python2.7/dist-packages/novaclient/v1_1/shell.py", line 905, in do_image_list
    image_list = cs.images.list(limit=limit)
  File "/usr/lib/python2.7/dist-packages/novaclient/v1_1/images.py", line 69, in list
    return self._list('/images%s%s' % (detail, query), 'images')
  File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 64, in _list
    _resp, body = self.api.client.get(url)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 283, in get
    return self._cs_request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 260, in _cs_request
    **kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 242, in _time_request
    resp, body = self.request(url, method, **kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 236, in request
    raise exceptions.from_response(resp, body, url, method)
ClientException: <attribute 'message' of 'exceptions.BaseException' objects> (HTTP 504)
ERROR: <attribute 'message' of 'exceptions.BaseException' objects> (HTTP 504)

Tags: vcenter
Revision history for this message
Tatyana Dubyk (tdubyk) wrote :
Revision history for this message
Evgeniya Shumakher (eshumakher) wrote :

Please identify the Importance level.

Changed in fuel:
assignee: nobody → Stepan Rogov (srogov)
Revision history for this message
Stepan Rogov (srogov) wrote :

Do you have a enabled VMDK driver with cinder?

Revision history for this message
Tatyana Dubyk (tdubyk) wrote :

yep! guys, I've added one more shapshot for you with another configuration. Try it please for reproducing, please.

api: '1.0'
astute_sha: bc60b7d027ab244039f48c505ac52ab8eb0a990c
auth_required: true
build_id: 2014-09-01_00-01-17
build_number: '491'
feature_groups:
- mirantis
fuellib_sha: 2cfa83119ae90b13a5bac6a844bdadfaf5aeb13f
fuelmain_sha: 109812be3425408dd7be192b5debf109cb1edd4c
nailgun_sha: d25ed02948a8be773e2bd87cfe583ef7be866bb2
ostf_sha: 4dcd99cc4bfa19f52d4b87ed321eb84ff03844da
production: docker
release: '5.1'

Steps to reproduce:
1. Create new environment (Ubuntu, HA mode)
     Network: Nova Network Flat DHCP

     setting for vcenter creation:
                    VCENTER_IP='172.16.0.254'
                    <email address hidden>'
                    VCENTER_PASSWORD='Qwer!1234'
                    VCENTER_CLUSTERS='Cluster1'

2. Add 4 nodes:
   3 nodes with CONTROLLER role;
   1 node with Cinder role

3.Make deploy of environment
4. Verify network connectivity
5. Run ostf

Changed in fuel:
importance: Undecided → High
summary: - In HA mode on vCenter's machine haproxy returns 504 gateway timeout in
- attempts to get access to api of different services
+ [vCenter]In HA mode on vCenter's machine haproxy returns 504 gateway
+ timeout in attempts to get access to api of different services
Changed in fuel:
status: New → Confirmed
importance: High → Critical
Revision history for this message
Tatyana Dubyk (tdubyk) wrote :

Temporary Igor can't reproduce it, so we've decided to set it in Incomplete status.

Changed in fuel:
status: Confirmed → Incomplete
Revision history for this message
Stepan Rogov (srogov) wrote :

It was a bug in the nova-conductor.
It seems that it was fixed in the new version of ISO, now I'll check once again.

Revision history for this message
Stepan Rogov (srogov) wrote :

Ok.
In 499-2014-09-05_00-01-17 ISO all working correctly.

Stepan Rogov (srogov)
Changed in fuel:
status: Incomplete → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.