Juniper Openstack

SM:R3.1build6 Mitaka:Creation of subnet within a VN using neutron fails with NeutronClientException: <html><body><h1>504 Gateway Time-out</h1>

  1. Series r3.1
  2. Bug #1607327
Bug #1607327 reported by sundarkh
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R3.1
Fix Committed
Critical
Dheeraj Gautam
Juniper Openstack r3.1.0.0-fcs
Trunk
Fix Committed
Critical
Dheeraj Gautam
Juniper Openstack r3.2.0.0-fcs "r3.2.0.0"

Bug Description

SM:R3.1build6 Mitaka:Creation of subnet within a VN using neutron fails with NeutronClientException: <html><body><h1>504 Gateway Time-out</h1>

Multi Interface setup provisioned using ServerManager

Topology

server": [
        {
            "cluster_id": "cluster_multi",
            "id": "nodec57",
            "roles": [
                "compute"
            ]
        },
        {
            "cluster_id": "cluster_multi",
            "id": "nodec33",
            "roles": [
                "webui",
                "config",
                "database",
                "control",
                "collector"
            ]
        },
        {
            "cluster_id": "cluster_multi",
            "id": "nodec35",
            "roles": [
                "control",
                "config",
                "openstack"
            ]
        },
        {
            "cluster_id": "cluster_multi",
            "id": "nodea4",
            "roles": [
                "compute"
            ]
        }
    ]
}

1) from cfgm0 (nodec35) try to create a subnet in a VN with neutron command
2) hitting neutron client exception
2016-07-26 23:32:28,529 - ERROR - Neutron Exception while creating subnet for vn with id 70f70db3-74ec-4a56-bc86-ce41a1d71772

Traceback (most recent call last):

  File "/contrail-test/fixtures/quantum_test.py", line 103, in create_subnet

    subnet_rsp = self.obj.create_subnet({'subnet': subnet_req})

  File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 97, in with_params

    ret = self.function(instance, *args, **kwargs)

  File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 715, in create_subnet

    return self.post(self.subnets_path, body=body)

  File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 363, in post

    headers=headers, params=params)

  File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 298, in do_request

    self._handle_fault_response(status_code, replybody, resp)

  File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 273, in _handle_fault_response

    exception_handler_v20(status_code, error_body)

  File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 84, in exception_handler_v20

    request_ids=request_ids)

NeutronClientException: <html><body><h1>504 Gateway Time-out</h1>

The server didn't respond in time.

</body></html>

Neutron-server.log
---------------------

2016-07-28 04:11:04.526 17761 ERROR neutron.api.v2.resource [req-507059f8-8d2f-49b2-940a-9c6ca0c94e9e fb9d6071012a4480ae9f327175154dbd 548aed020125420f8b421543761900b3 - - -] create failed
2016-07-28 04:11:04.526 17761 ERROR neutron.api.v2.resource Traceback (most recent call last):
2016-07-28 04:11:04.526 17761 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/api/v2/resource.py", line 84, in resource
2016-07-28 04:11:04.526 17761 ERROR neutron.api.v2.resource result = method(request=request, **args)
2016-07-28 04:11:04.526 17761 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/api/v2/base.py", line 410, in create
2016-07-28 04:11:04.526 17761 ERROR neutron.api.v2.resource return self._create(request, body, **kwargs)
2016-07-28 04:11:04.526 17761 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 148, in wrapper
2016-07-28 04:11:04.526 17761 ERROR neutron.api.v2.resource ectxt.value = e.inner_exc
2016-07-28 04:11:04.526 17761 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-07-28 04:11:04.526 17761 ERROR neutron.api.v2.resource self.force_reraise()
2016-07-28 04:11:04.526 17761 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-07-28 04:11:04.526 17761 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2016-07-28 04:11:04.526 17761 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 138, in wrapper
2016-07-28 04:11:04.526 17761 ERROR neutron.api.v2.resource return f(*args, **kwargs)
2016-07-28 04:11:04.526 17761 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/api/v2/base.py", line 525, in _create
2016-07-28 04:11:04.526 17761 ERROR neutron.api.v2.resource obj)})
2016-07-28 04:11:04.526 17761 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/api/v2/base.py", line 463, in notify
2016-07-28 04:11:04.526 17761 ERROR neutron.api.v2.resource with request.context.session.begin():
2016-07-28 04:11:04.526 17761 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/context.py", line 121, in session
2016-07-28 04:11:04.526 17761 ERROR neutron.api.v2.resource self._session = db_api.get_session()
2016-07-28 04:11:04.526 17761 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 75, in get_session
2016-07-28 04:11:04.526 17761 ERROR neutron.api.v2.resource facade = _create_facade_lazily()
2016-07-28 04:11:04.526 17761 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 56, in _create_facade_lazily
2016-07-28 04:11:04.526 17761 ERROR neutron.api.v2.resource _FACADE = session.EngineFacade.from_config(cfg.CONF, sqlite_fk=True)
2016-07-28 04:11:04.526 17761 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1056, in from_config
2016-07-28 04:11:04.526 17761 ERROR neutron.api.v2.resource expire_on_commit=expire_on_commit, _conf=conf)
2016-07-28 04:11:04.526 17761 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 977, in __init__
2016-07-28 04:11:04.526 17761 ERROR neutron.api.v2.resource slave_connection=slave_connection)

2016-07-28 04:11:04.526 17761 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 338, in _start
2016-07-28 04:11:04.526 17761 ERROR neutron.api.v2.resource engine_args, maker_args)
2016-07-28 04:11:04.526 17761 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 362, in _setup_for_connection
2016-07-28 04:11:04.526 17761 ERROR neutron.api.v2.resource sql_connection=sql_connection, **engine_kwargs)
2016-07-28 04:11:04.526 17761 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py", line 152, in create_engine
2016-07-28 04:11:04.526 17761 ERROR neutron.api.v2.resource test_conn = _test_connection(engine, max_retries, retry_interval)
2016-07-28 04:11:04.526 17761 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py", line 334, in _test_connection
2016-07-28 04:11:04.526 17761 ERROR neutron.api.v2.resource six.reraise(type(de_ref), de_ref)
2016-07-28 04:11:04.526 17761 ERROR neutron.api.v2.resource File "<string>", line 2, in reraise
2016-07-28 04:11:04.526 17761 ERROR neutron.api.v2.resource DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '192.168.100.3' ([Errno 111] ECONNREFUSED)")
2016-07-28 04:11:04.526 17761 ERROR neutron.api.v2.resource
2016-07-28 04:11:04.528 17761 INFO neutron.wsgi [req-507059f8-8d2f-49b2-940a-9c6ca0c94e9e fb9d6071012a4480ae9f327175154dbd 548aed020125420f8b421543761900b3 - - -] Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 512, in handle_one_response
    write(b''.join(towrite))
  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 453, in write
    wfile.flush()
  File "/usr/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
  File "/usr/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 385, in sendall
    tail = self.send(data, flags)
  File "/usr/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 379, in send
    return self._send_loop(self.fd.send, data, flags)
  File "/usr/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 366, in _send_loop
    return send_method(data, *args)
error: [Errno 104] Connection reset by peer

2016-07-28 04:11:04.528 17761 INFO neutron.wsgi [req-507059f8-8d2f-49b2-940a-9c6ca0c94e9e fb9d6071012a4480ae9f327175154dbd 548aed020125420f8b421543761900b3 - - -] 192.168.100.3 - - [28/Jul/2016 04:11:04] "POST /v2.0/networks.json HTTP/1.1" 500 0 100.330614

sundarkh (sundar-kh)
tags: added: blocker
Revision history for this message
sundarkh (sundar-kh) wrote :

Issue not seen in Single Node

Sudheendra Rao (sudheendra-k)
Changed in juniperopenstack:
milestone: r3.1.0.0-fcs → none
Revision history for this message
sundarkh (sundar-kh) wrote :

Seen with kilo also

Revision history for this message
sundarkh (sundar-kh) wrote :
Download full text (13.5 KiB)

2016-08-03 02:23:52.408 28756 ERROR neutron.service [-] Unrecoverable error: please check log for details.
2016-08-03 02:23:52.408 28756 TRACE neutron.service Traceback (most recent call last):
2016-08-03 02:23:52.408 28756 TRACE neutron.service File "/usr/lib/python2.7/dist-packages/neutron/service.py", line 103, in serve_wsgi
2016-08-03 02:23:52.408 28756 TRACE neutron.service service.start()
2016-08-03 02:23:52.408 28756 TRACE neutron.service File "/usr/lib/python2.7/dist-packages/neutron/service.py", line 74, in start
2016-08-03 02:23:52.408 28756 TRACE neutron.service self.wsgi_app = _run_wsgi(self.app_name)
2016-08-03 02:23:52.408 28756 TRACE neutron.service File "/usr/lib/python2.7/dist-packages/neutron/service.py", line 169, in _run_wsgi
2016-08-03 02:23:52.408 28756 TRACE neutron.service app = config.load_paste_app(app_name)
2016-08-03 02:23:52.408 28756 TRACE neutron.service File "/usr/lib/python2.7/dist-packages/neutron/common/config.py", line 227, in load_paste_app
2016-08-03 02:23:52.408 28756 TRACE neutron.service app = deploy.loadapp("config:%s" % config_path, name=app_name)
2016-08-03 02:23:52.408 28756 TRACE neutron.service File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
2016-08-03 02:23:52.408 28756 TRACE neutron.service return loadobj(APP, uri, name=name, **kw)
2016-08-03 02:23:52.408 28756 TRACE neutron.service File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 272, in loadobj
2016-08-03 02:23:52.408 28756 TRACE neutron.service return context.create()
2016-08-03 02:23:52.408 28756 TRACE neutron.service File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
2016-08-03 02:23:52.408 28756 TRACE neutron.service return self.object_type.invoke(self)
2016-08-03 02:23:52.408 28756 TRACE neutron.service File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 144, in invoke
2016-08-03 02:23:52.408 28756 TRACE neutron.service **context.local_conf)
2016-08-03 02:23:52.408 28756 TRACE neutron.service File "/usr/lib/python2.7/dist-packages/paste/deploy/util.py", line 55, in fix_call
2016-08-03 02:23:52.408 28756 TRACE neutron.service val = callable(*args, **kw)
2016-08-03 02:23:52.408 28756 TRACE neutron.service File "/usr/lib/python2.7/dist-packages/paste/urlmap.py", line 28, in urlmap_factory
2016-08-03 02:23:52.408 28756 TRACE neutron.service app = loader.get_app(app_name, global_conf=global_conf)
2016-08-03 02:23:52.408 28756 TRACE neutron.service File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 350, in get_app
2016-08-03 02:23:52.408 28756 TRACE neutron.service name=name, global_conf=global_conf).create()
2016-08-03 02:23:52.408 28756 TRACE neutron.service File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
2016-08-03 02:23:52.408 28756 TRACE neutron.service return self.object_type.invoke(self)
2016-08-03 02:23:52.408 28756 TRACE neutron.service File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 144, in invoke
2016-08-03 02:23:52.408 28756 TRACE neutron.service **context.local_...

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.1

Review in progress for https://review.opencontrail.org/22820
Submitter: Dheeraj Gautam (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/22820
Committed: http://github.org/Juniper/contrail-puppet/commit/b1edb5d428459a064fe918bbb5b632104ea93884
Submitter: Zuul
Branch: R3.1

commit b1edb5d428459a064fe918bbb5b632104ea93884
Author: Dheeraj Gautam <email address hidden>
Date: Wed Aug 3 14:53:34 2016 -0700

SM-Neutron: Fix neutron.conf to correct mysql

Closes-Bug: #1607327

CHANGES:
1. calculate neutron_mysql based on openstack role, internal_vip, external_ip
or openstack_ip_list[0]

TESTING:
Liberty:
1. provisioned nodes with 1 openstack + config and 1 with config roles
, verified that provision is completed
2. added image, created a network, subnetwork and 2 VMs, verified VMs are
active.
3. Logged-in to instance using shell.
4. VMs are able to ping each other.
5. VM is able get http://169.254.269.254/openstack (metadata service)

PATCH 2:
Fixed host_control_ip to host_ip. This is not available in params.pp

Change-Id: I29f0536db80695fcecb2aac58d5274f8fed44ce9

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/22869
Submitter: Dheeraj Gautam (<email address hidden>)

Revision history for this message
sundarkh (sundar-kh) wrote :

in build 15, following trace is seen

raceback (most recent call last):
  File "/contrail-test/fixtures/quantum_test.py", line 77, in create_network
    net_rsp = self.obj.create_network({'network': net_req})
  File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 99, in with_params
    ret = self.function(instance, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 542, in create_network
    return self.post(self.networks_path, body=body)
  File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 295, in post
    headers=headers, params=params)
  File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 197, in do_request
    content_type=self.content_type())
  File "/usr/lib/python2.7/dist-packages/neutronclient/client.py", line 172, in do_request
    **kwargs)
  File "/usr/lib/python2.7/dist-packages/neutronclient/client.py", line 108, in _cs_request
    raise exceptions.ConnectionFailed(reason=e)
ConnectionFailed: Connection to neutron failed: HTTPConnectionPool(host='192.168.100.3', port=9696): Max retries exceeded with url: /v2.0/networks.json (Caused by <class 'httplib.BadStatusLine'>: '')
2016-08-04 09:31:30,647 - DEBUG - VN ctest-ctest-TestBasicVMVN0-59228348-86249799 not present

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/22869
Committed: http://github.org/Juniper/contrail-puppet/commit/0690a9a3e364be9af23d8851972eac35b246e4c3
Submitter: Zuul
Branch: master

commit 0690a9a3e364be9af23d8851972eac35b246e4c3
Author: Dheeraj Gautam <email address hidden>
Date: Wed Aug 3 14:53:34 2016 -0700

SM-Neutron: Fix neutron.conf to correct mysql

Closes-Bug: #1607327

CHANGES:
1. calculate neutron_mysql based on openstack role, internal_vip, external_ip
or openstack_ip_list[0]

TESTING:
Liberty:
1. provisioned nodes with 1 openstack + config and 1 with config roles
, verified that provision is completed
2. added image, created a network, subnetwork and 2 VMs, verified VMs are
active.
3. Logged-in to instance using shell.
4. VMs are able to ping each other.
5. VM is able get http://169.254.269.254/openstack (metadata service)

PATCH 2:
Fixed host_control_ip to host_ip. This is not available in params.pp

Change-Id: I29f0536db80695fcecb2aac58d5274f8fed44ce9

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.