neutron security-group-rule-list returnes "RequestURITooLong"

[Leontii Istomin]

[root@fuel ~]# fuel --fuel-version
DEPRECATION WARNING: file /etc/fuel/client/config.yaml is found and will be used as a source for settings. However, it deprecated and will not be used by default in the ongoing version of python-fuelclient.
api: '1.0'
astute_sha: 1be5b9b827f512d740fe907c7ff72486d4030938
auth_required: true
build_id: 2015-03-02_14-00-04
build_number: '154'
feature_groups:
- mirantis
fuellib_sha: b17e3810dbca407fca2a231c26f553a46e166343
fuelmain_sha: baf24424a4e056c6753913de5f8c94851903f718
nailgun_sha: f034fbb4b68be963e4dc5b5d680061b54efbf605
ostf_sha: 103d6cf6badd57b791cfaf4310ec8bd81c7a8a46
production: docker
python-fuelclient_sha: 3ebfa9c14a192d0298ff787526bf990055a23694
release: '6.1'
release_versions:
  2014.2-6.1:
    VERSION:
      api: '1.0'
      astute_sha: 1be5b9b827f512d740fe907c7ff72486d4030938
      build_id: 2015-03-02_14-00-04
      build_number: '154'
      feature_groups:
      - mirantis
      fuellib_sha: b17e3810dbca407fca2a231c26f553a46e166343
      fuelmain_sha: baf24424a4e056c6753913de5f8c94851903f718
      nailgun_sha: f034fbb4b68be963e4dc5b5d680061b54efbf605
      ostf_sha: 103d6cf6badd57b791cfaf4310ec8bd81c7a8a46
      production: docker
      python-fuelclient_sha: 3ebfa9c14a192d0298ff787526bf990055a23694
      release: '6.1'

Centos, HA, Neutron-vlan,Ceph-all, Debug, nova-quotas, 6.1_154
Controllers:3 Computes:47

Cluster was deployed successfully.

Rally created a lot of sec groups and sec group rules. If I execute neutron security-group-rule-list, I get "An unknown exception occurred."
with "--debug" parameter I can see the following trace:
ERROR: neutronclient.shell An unknown exception occurred.
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/neutronclient/shell.py", line 691, in run_subcommand
    return run_command(cmd, cmd_parser, sub_argv)
  File "/usr/lib/python2.6/site-packages/neutronclient/shell.py", line 90, in run_command
    return cmd.run(known_args)
  File "/usr/lib/python2.6/site-packages/neutronclient/common/command.py", line 29, in run
    return super(OpenStackCommand, self).run(parsed_args)
  File "/usr/lib/python2.6/site-packages/cliff/display.py", line 91, in run
    column_names, data = self.take_action(parsed_args)
  File "/usr/lib/python2.6/site-packages/neutronclient/common/command.py", line 35, in take_action
    return self.get_data(parsed_args)
  File "/usr/lib/python2.6/site-packages/neutronclient/neutron/v2_0/__init__.py", line 670, in get_data
    self.extend_list(data, parsed_args)
  File "/usr/lib/python2.6/site-packages/neutronclient/neutron/v2_0/securitygroup.py", line 161, in extend_list
    _get_sec_group_list(sec_group_ids[i: i + chunk_size]))
  File "/usr/lib/python2.6/site-packages/neutronclient/neutron/v2_0/securitygroup.py", line 143, in _get_sec_group_list
    **search_opts).get('security_groups', [])
  File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 98, in with_params
    ret = self.function(instance, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 480, in list_security_groups
    retrieve_all, **_params)
  File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 1334, in list
    for r in self._pagination(collection, path, **params):
  File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 1347, in _pagination
    res = self.get(path, params=params)
  File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 1320, in get
    headers=headers, params=params)
  File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 1297, in retry_request
    headers=headers, params=params)
  File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 1233, in do_request
    self._check_uri_length(action)
  File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 1222, in _check_uri_length
    excess=uri_len - self.MAX_URI_LEN)
RequestURITooLong: An unknown exception occurred.
At the moment was 408 security groups
[root@node-166 ~]# mysql -e "use neutron; select * from securitygrouprules;" | wc -l
1633
The full output is attached.

[Leontii Istomin (listomin)]

here is the related bug https://bugs.launchpad.net/python-neutronclient/+bug/1271462

[OSCI Robot (oscirobot)]

package python-neutronclient has been built for project openstack/python-neutronclient
Package version == 2.3.9, package release == fuel6.1.mira25.git.82d11f8.6c754be

Changeset: https://review.fuel-infra.org/4461
project: openstack/python-neutronclient
branch: openstack-ci/fuel-6.1/2014.2
author: Ann Kamyshnikova
committer: Ann Kamyshnikova
subject: Fix for securitygrouprules listing
status: patchset-created

Files placed on repository:

NOTE: Changeset is not merged, created temporary package repository.
 repository URL: /centos-fuel-6.1-stable-4461/

[OSCI Robot (oscirobot)]

package python-neutronclient has been built for project openstack/python-neutronclient
Package version == 2.3.9, package release == fuel6.1~mira2+git.82d11f8.6c754be

Changeset: https://review.fuel-infra.org/4461
project: openstack/python-neutronclient
branch: openstack-ci/fuel-6.1/2014.2
author: Ann Kamyshnikova
committer: Ann Kamyshnikova
subject: Fix for securitygrouprules listing
status: patchset-created

Files placed on repository:

NOTE: Changeset is not merged, created temporary package repository.
 repository URL: /trusty-fuel-6.1-stable-4461/

[OSCI Robot (oscirobot)]

package python-neutronclient has been built for project openstack/python-neutronclient
Package version == 2.3.9, package release == fuel6.1.mira25.git.4c8ed4f.6c754be

Changeset: https://review.fuel-infra.org/4611
project: openstack/python-neutronclient
branch: openstack-ci/fuel-6.1/2014.2
author: Ann Kamyshnikova
committer: Ann Kamyshnikova
subject: Skip None id when getting security_group_ids
status: patchset-created

Files placed on repository:

NOTE: Changeset is not merged, created temporary package repository.
 repository URL: /centos-fuel-6.1-stable-4611/

[OSCI Robot (oscirobot)]

package python-neutronclient has been built for project openstack/python-neutronclient
Package version == 2.3.9, package release == fuel6.1.mira25.git.9f08e28.6c754be

Changeset: https://review.fuel-infra.org/4617
project: openstack/python-neutronclient
branch: openstack-ci/fuel-6.1/2014.2
author: Ann Kamyshnikova
committer: Ann Kamyshnikova
subject: Fix KeyError when filtering SG rule listing
status: patchset-created

Files placed on repository:

NOTE: Changeset is not merged, created temporary package repository.
 repository URL: /centos-fuel-6.1-stable-4617/

[OSCI Robot (oscirobot)]

package python-neutronclient has been built for project openstack/python-neutronclient
Package version == 2.3.9, package release == fuel6.1~mira2+git.9f08e28.6c754be

Changeset: https://review.fuel-infra.org/4617
project: openstack/python-neutronclient
branch: openstack-ci/fuel-6.1/2014.2
author: Ann Kamyshnikova
committer: Ann Kamyshnikova
subject: Fix KeyError when filtering SG rule listing
status: patchset-created

Files placed on repository:

NOTE: Changeset is not merged, created temporary package repository.
 repository URL: /trusty-fuel-6.1-stable-4617/

[OSCI Robot (oscirobot)]

package python-neutronclient has been built for project openstack/python-neutronclient
Package version == 2.3.9, package release == fuel6.1.mira25.git.912e408.6c754be

Changeset: https://review.fuel-infra.org/4611
project: openstack/python-neutronclient
branch: openstack-ci/fuel-6.1/2014.2
author: Ann Kamyshnikova
committer: Ann Kamyshnikova
subject: Skip None id when getting security_group_ids
status: patchset-created

Files placed on repository:

NOTE: Changeset is not merged, created temporary package repository.
 repository URL: /centos-fuel-6.1-stable-4611/

[OSCI Robot (oscirobot)]

package python-neutronclient has been built for project openstack/python-neutronclient
Package version == 2.3.9, package release == fuel6.1~mira2+git.912e408.6c754be

Changeset: https://review.fuel-infra.org/4611
project: openstack/python-neutronclient
branch: openstack-ci/fuel-6.1/2014.2
author: Ann Kamyshnikova
committer: Ann Kamyshnikova
subject: Skip None id when getting security_group_ids
status: patchset-created

Files placed on repository:

NOTE: Changeset is not merged, created temporary package repository.
 repository URL: /trusty-fuel-6.1-stable-4611/

[OSCI Robot (oscirobot)]

package python-neutronclient has been built for project openstack/python-neutronclient
Package version == 2.3.9, package release == fuel6.1.mira25

Changeset: https://review.fuel-infra.org/4617
project: openstack/python-neutronclient
branch: openstack-ci/fuel-6.1/2014.2
author: Ann Kamyshnikova
committer: Ann Kamyshnikova
subject: Fix KeyError when filtering SG rule listing
status: change-merged

Files placed on repository:

Changeset merged. Package placed on primary repository
 repository URL: /centos-fuel-6.1-stable/

[OSCI Robot (oscirobot)]

package python-neutronclient has been built for project openstack/python-neutronclient
Package version == 2.3.9, package release == fuel6.1~mira3

Changeset: https://review.fuel-infra.org/4617
project: openstack/python-neutronclient
branch: openstack-ci/fuel-6.1/2014.2
author: Ann Kamyshnikova
committer: Ann Kamyshnikova
subject: Fix KeyError when filtering SG rule listing
status: change-merged

Files placed on repository:

Changeset merged. Package placed on primary repository
 repository URL: /trusty-fuel-6.1-stable/

[OSCI Robot (oscirobot)]

package python-neutronclient has been built for project openstack/python-neutronclient
Package version == 2.3.9, package release == fuel6.1.mira26

Changeset: https://review.fuel-infra.org/4611
project: openstack/python-neutronclient
branch: openstack-ci/fuel-6.1/2014.2
author: Ann Kamyshnikova
committer: Ann Kamyshnikova
subject: Skip None id when getting security_group_ids
status: change-merged

Files placed on repository:

Changeset merged. Package placed on primary repository
 repository URL: /centos-fuel-6.1-stable/

[OSCI Robot (oscirobot)]

package python-neutronclient has been built for project openstack/python-neutronclient
Package version == 2.3.9, package release == fuel6.1~mira4

Changeset: https://review.fuel-infra.org/4611
project: openstack/python-neutronclient
branch: openstack-ci/fuel-6.1/2014.2
author: Ann Kamyshnikova
committer: Ann Kamyshnikova
subject: Skip None id when getting security_group_ids
status: change-merged

Files placed on repository:

Changeset merged. Package placed on primary repository
 repository URL: /trusty-fuel-6.1-stable/

[Ksenia Svechnikova (kdemina)]

Verified by the following steps:

1. Update neutron quotas on every controller:

vim /etc/neutron/neutron.conf

quota_security_group = 600
quota_security_group_rule = 2000

2. Run script:

#!/bin/bash

for i in {10..500}

do
  echo "Create sg_"$i
  nova secgroup-create default$i default
  nova secgroup-add-rule default$i tcp 80 80 0.0.0.0/0
  nova secgroup-add-rule default$i icmp -1 -1 0.0.0.0/0

done

3. Check neutron security-group-rule-list and neutron security-group-list

[Ksenia Svechnikova (kdemina)]

 Step 1 also contains a procedure of restarting neutron-server on every controller after updating configs

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: sridhargaddam <email address hidden>
Review: https://review.fuel-infra.org/8306

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: watanabe.isao <email address hidden>
Review: https://review.fuel-infra.org/8307

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Ann Kamyshnikova <email address hidden> on branch: openstack-ci/fuel-7.0/2015.1.0
Review: https://review.fuel-infra.org/8307
Reason: Not needed. Merged in 2.4.0

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Ann Kamyshnikova <email address hidden> on branch: openstack-ci/fuel-7.0/2015.1.0
Review: https://review.fuel-infra.org/8306
Reason: Not needed. Merged in 2.4.0.

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: openstack-ci/fuel-6.0-updates/2014.2
Change author: sridhargaddam <email address hidden>
Review: https://review.fuel-infra.org/9957

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/9957
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-6.0-updates/2014.2

Commit: cc26b18bc2764362d0359b9b9730bbc2d0374d72
Author: sridhargaddam <email address hidden>
Date: Wed Jul 29 13:53:50 2015

Fix KeyError when filtering SG rule listing

extend_list API of ListSecurityGroupRule class assumes that the data
includes security_group_id and remote_group_id fields, which may not
be the case when we filter on other fields. This patch addresses this
issue.

Related-Bug: #1429065
(cherry picked from commit 845f4618f4818d3f536eda1dde8d9b9d860f0417)

Change-Id: Id14f748957fe8eded069d444f865ac319513bdc0

[Alexey Khivin (akhivin)]

Set won't-fix for 6.0 because 6.0.1 release is officially cancelled

[Vadim Rovachev (vrovachev)]

Bug reproduced after applying fix on 6.0 Fuel version.

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: openstack-ci/fuel-6.0-updates/2014.2
Change author: Alexey Khivin <email address hidden>
Review: https://review.fuel-infra.org/10505

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/10505
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-6.0-updates/2014.2

Commit: dbb7581ee0be425d1298f59d26d352d60d428999
Author: Alexey Khivin <email address hidden>
Date: Mon Aug 17 23:18:39 2015

Fix KeyError when filtering SG rule listing (2)

Fix error in patch "Id14f748957fe8eded069d444f865ac319513bdc0"

Patch "Fix KeyError when filtering SG rule listing" does not really
fix the issue which should fix. This patch changes calculation of
value exceed of RequestURITooLong exception.

Related-Bug: #1429065
Change-Id: Ifc2e76386697b64af615dfdf1adb719aa643f398

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: openstack-ci/fuel-6.0-updates/2014.2
Change author: Alexey Khivin <email address hidden>
Review: https://review.fuel-infra.org/10538

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-6.0-updates/2014.2
Change author: watanabe.isao <email address hidden>
Review: https://review.fuel-infra.org/10547

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/10547
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-6.0-updates/2014.2

Commit: 1b3843c97f306ffbc44ee0192160e6b3b796c4de
Author: watanabe.isao <email address hidden>
Date: Tue Aug 18 17:02:58 2015

Skip None id when getting security_group_ids

When getting security_group_ids in securitygroup.py, a None id is
included into the list due to the remote group id is Null when the
security group rule direction is egress.
And this causes the calculation of the number of chunk_size is
1 more than the number it should be.
Then the request page size excess the MAX request page size
Then the RequestURITooLong exception happens again, and finally
the commend fails.

Co-Authored-By: Furukawa, Yushiro <email address hidden>
Closes-Bug: #1429065

(cherry picked from commit 59d7564202c9a5acd57f23a2708291d1d1cb6a54)

Change-Id: Ia6be2b47a46c82c819cc361e0eca5928ac299345
(cherry picked from commit 912e4084bc047c0de0292e342aa45595a668af60)

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/10538
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-6.0-updates/2014.2

Commit: 86855b574b2888f9985fb80d174e77cdfc5938c2
Author: Alexey Khivin <email address hidden>
Date: Tue Aug 18 18:14:24 2015

Fix KeyError when filtering SG rule listing (3)

Fix error in patch "Id14f748957fe8eded069d444f865ac319513bdc0" and
"Ifc2e76386697b64af615dfdf1adb719aa643f398"

After we backported URI length checks while applying
"Fix KeyError when filtering SG rule listing" we see the new issue.
URI length calculated inaccurate and part "fields=id&fields=name&"
of the url is not taken into consideration. Because of that,
RequestURITooLong may be raised

Related-Bug: #1429065

Change-Id: I7c721ff553614da7f21141fd1c5f96da0ba1b399