neutron net-list bad request on large number of networks

Bug #1384418 reported by Ilya Shakhat
18
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Fix Committed
High
Sergey Kolekonov

Bug Description

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.0"
  api: "1.0"
  build_number: "156"
  build_id: "2014-10-21_22-46-19"
  astute_sha: "6a11a7c481d116e6cfdb422fab1d4bbb29cbea1c"
  fuellib_sha: "af2bf11d4a3a075fa4e9fa9b7b7209af29498a46"
  ostf_sha: "de177931b53fbe9655502b73d03910b8118e25f1"
  nailgun_sha: "f4bf25da24c4e5b0d9eb86493945200deba3d92e"
  fuelmain_sha: "dab17913263bbea7e9a3b55de8a0f3af5ac0e3e2"

OpenStack contains 1796 networks. Command 'neutron net-list' fails:
(.venv)developer@fuel:stack$ neutron net-list
Bad Request (HTTP 400)

In debug mode the command shows huge response with the list of networks followed by request "curl -i -X GET http://172.16.44.10:9696/v2.0/subnets.json?fields=id&fields=cidr&id=7ef6efcc-ee2a-481f-9097-0403c2994246&id=.... etc." where ids are subnet ids.

Stacktrace:
DEBUG: keystoneclient.session RESP:
DEBUG: keystoneclient.session Request returned failure status: 400
ERROR: neutronclient.shell Bad Request (HTTP 400)
Traceback (most recent call last):
  File "/opt/stack/.venv/lib/python2.7/site-packages/neutronclient/shell.py", line 691, in run_subcommand
    return run_command(cmd, cmd_parser, sub_argv)
  File "/opt/stack/.venv/lib/python2.7/site-packages/neutronclient/shell.py", line 90, in run_command
    return cmd.run(known_args)
  File "/opt/stack/.venv/lib/python2.7/site-packages/neutronclient/common/command.py", line 29, in run
    return super(OpenStackCommand, self).run(parsed_args)
  File "/opt/stack/.venv/lib/python2.7/site-packages/cliff/display.py", line 91, in run
    column_names, data = self.take_action(parsed_args)
  File "/opt/stack/.venv/lib/python2.7/site-packages/neutronclient/common/command.py", line 35, in take_action
    return self.get_data(parsed_args)
  File "/opt/stack/.venv/lib/python2.7/site-packages/neutronclient/neutron/v2_0/__init__.py", line 670, in get_data
    self.extend_list(data, parsed_args)
  File "/opt/stack/.venv/lib/python2.7/site-packages/neutronclient/neutron/v2_0/network.py", line 75, in extend_list
    _get_subnet_list(subnet_ids[i: i + chunk_size]))
  File "/opt/stack/.venv/lib/python2.7/site-packages/neutronclient/neutron/v2_0/network.py", line 60, in _get_subnet_list
    **search_opts).get('subnets', [])
  File "/opt/stack/.venv/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 98, in with_params
    ret = self.function(instance, *args, **kwargs)
  File "/opt/stack/.venv/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 365, in list_subnets
    **_params)
  File "/opt/stack/.venv/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1334, in list
    for r in self._pagination(collection, path, **params):
  File "/opt/stack/.venv/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1347, in _pagination
    res = self.get(path, params=params)
  File "/opt/stack/.venv/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1320, in get
    headers=headers, params=params)
  File "/opt/stack/.venv/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1297, in retry_request
    headers=headers, params=params)
  File "/opt/stack/.venv/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1240, in do_request
    content_type=self.content_type())
  File "/opt/stack/.venv/lib/python2.7/site-packages/neutronclient/client.py", line 319, in do_request
    return self.request(url, method, **kwargs)
  File "/opt/stack/.venv/lib/python2.7/site-packages/neutronclient/client.py", line 63, in request
    return self._request(url, method, body=body, headers=headers, **kwargs)
  File "/opt/stack/.venv/lib/python2.7/site-packages/neutronclient/client.py", line 314, in _request
    **kwargs)
  File "/opt/stack/.venv/lib/python2.7/site-packages/keystoneclient/utils.py", line 318, in inner
    return func(*args, **kwargs)
  File "/opt/stack/.venv/lib/python2.7/site-packages/keystoneclient/session.py", line 346, in request
    raise exceptions.from_response(resp, method, url)
BadRequest: Bad Request (HTTP 400)

At the same time UI shows list of networks correctly (contrversary to https://bugs.launchpad.net/mos/+bug/1383265)

Tags: neutron scale
Revision history for this message
Ilya Shakhat (shakhat) wrote :
tags: added: neutron
Revision history for this message
Ilya Shakhat (shakhat) wrote :

(.venv)developer@fuel:stack$ neutron subnet-list | wc -l
1153

Changed in mos:
assignee: nobody → MOS Neutron (mos-neutron)
Changed in mos:
milestone: none → 6.0
tags: added: scale
Changed in mos:
status: New → Confirmed
importance: Undecided → Medium
Revision history for this message
Ilya Shakhat (shakhat) wrote :

Steps to reproduce:
0. Deploy MOS in HA mode
1. Create 204 networks with subnets
#!/bin/bash
for i in {1..204}
do
   neutron net-create net-$i
   neutron subnet-create net-$i 10.2.$i.0/24
done
2. Run "neutron net-list". The command fails with "Bad Request (HTTP 400)"

Revision history for this message
Ilya Shakhat (shakhat) wrote :

Both neutron server and client limit request size to 8192 bytes. However haproxy does not process requests longer than 8035 bytes. It's considered to tune haproxy to be compatible with other request size limits.

Revision history for this message
OSCI Robot (oscirobot) wrote :

RPM package python-neutronclient has been built for project openstack/python-neutronclient
Package version == 2.3.9, package release == fuel6.0.mira22.git.3afe2e3.3d5c0c2

Changeset: https://review.fuel-infra.org/1070
project: openstack/python-neutronclient
branch: openstack-ci/fuel-6.0/2014.2
author: Ilya Shakhat
committer: Ilya Shakhat
subject: Limit URI length to 8000 bytes
status: patchset-created

Files placed on repository:
python-neutronclient-2.3.9-fuel6.0.mira22.git.3afe2e3.3d5c0c2.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0-stable-1070/centos

Revision history for this message
OSCI Robot (oscirobot) wrote :

DEB package python-neutronclient has been built for project openstack/python-neutronclient
Package version == 2.3.9, package release == fuel6.0~mira19+git.3afe2e3.3d5c0c2

Changeset: https://review.fuel-infra.org/1070
project: openstack/python-neutronclient
branch: openstack-ci/fuel-6.0/2014.2
author: Ilya Shakhat
committer: Ilya Shakhat
subject: Limit URI length to 8000 bytes
status: patchset-created

Files placed on repository:
python-neutronclient_2.3.9-fuel6.0~mira19+git.3afe2e3.3d5c0c2_all.deb
python-quantumclient_2.3.9-fuel6.0~mira19+git.3afe2e3.3d5c0c2_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0-stable-1070/ubuntu

Changed in mos:
importance: Medium → High
assignee: MOS Neutron (mos-neutron) → Sergey Kolekonov (skolekonov)
Changed in mos:
status: Confirmed → In Progress
Changed in mos:
status: In Progress → Fix Committed
Revision history for this message
Sergey Kolekonov (skolekonov) wrote :
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.