neutron security-group-rule-list fails with URI too long

[Vincent Untz]

When running "neutron --debug security-group-rule-list", I get:

An unknown exception occurred.
Traceback (most recent call last):
  File "/usr/lib64/python2.6/site-packages/neutronclient/shell.py", line 527, in run_subcommand
    return run_command(cmd, cmd_parser, sub_argv)
  File "/usr/lib64/python2.6/site-packages/neutronclient/shell.py", line 80, in run_command
    return cmd.run(known_args)
  File "/usr/lib64/python2.6/site-packages/neutronclient/common/command.py", line 35, in run
    return super(OpenStackCommand, self).run(parsed_args)
  File "/usr/lib64/python2.6/site-packages/cliff/display.py", line 84, in run
    column_names, data = self.take_action(parsed_args)
  File "/usr/lib64/python2.6/site-packages/neutronclient/common/command.py", line 41, in take_action
    return self.get_data(parsed_args)
  File "/usr/lib64/python2.6/site-packages/neutronclient/neutron/v2_0/__init__.py", line 612, in get_data
    self.extend_list(data, parsed_args)
  File "/usr/lib64/python2.6/site-packages/neutronclient/neutron/v2_0/securitygroup.py", line 146, in extend_list
    secgroups = neutron_client.list_security_groups(**search_opts)
  File "/usr/lib64/python2.6/site-packages/neutronclient/v2_0/client.py", line 112, in with_params
    ret = self.function(instance, *args, **kwargs)
  File "/usr/lib64/python2.6/site-packages/neutronclient/v2_0/client.py", line 475, in list_security_groups
    retrieve_all, **_params)
  File "/usr/lib64/python2.6/site-packages/neutronclient/v2_0/client.py", line 1251, in list
    for r in self._pagination(collection, path, **params):
  File "/usr/lib64/python2.6/site-packages/neutronclient/v2_0/client.py", line 1264, in _pagination
    res = self.get(path, params=params)
  File "/usr/lib64/python2.6/site-packages/neutronclient/v2_0/client.py", line 1237, in get
    headers=headers, params=params)
  File "/usr/lib64/python2.6/site-packages/neutronclient/v2_0/client.py", line 1222, in retry_request
    headers=headers, params=params)
  File "/usr/lib64/python2.6/site-packages/neutronclient/v2_0/client.py", line 1152, in do_request
    self._check_uri_length(action)
  File "/usr/lib64/python2.6/site-packages/neutronclient/v2_0/client.py", line 1141, in _check_uri_length
    excess=uri_len - self.MAX_URI_LEN)
RequestURITooLong: An unknown exception occurred.

The path is clearly too long:
/v2.0/security-groups.json?fields=id&fields=name&id=b7f0a1f8-37e5-495d-abed-345ab0460866&id=26ef80bd-8d30-4aac-9902-afa759e4af73&id=a76a326f-66c0-442e-9910-2642bd9ad521&id=5b8b8a27-c6c5-423a-8e19-ff96ac3440b3&id
=dbfa0616-e7b8-4eb3-baf0-95b7dfbcb790&id=ae2ddf91-c35e-423b-b6b2-f1803ae6aca5&id=03553aee-d9e9-41e2-a648-3e4407b7cddd&id=93041512-8b88-4365-9173-a618c252ce48&id=0c33eec5-fa3a-4009-82fc-106f9b2915d1&id=e363be01-a
95e-45af-8db5-fb9b0425d874&id=5630c0c5-1d5e-4513-b512-0efda0b4bb20&id=e6185b28-ed8f-42e7-b107-4ce3a334f182&id=4eacf6a2-5e41-4dc6-8e52-76eaf6e89b2f&id=80b4daea-32dd-4dac-ab97-91538f3561f8&id=d06b3dc8-dd76-4d2b-9c
f3-befb42863e30&id=ea64c4b5-b9ed-49ad-9074-16eb87fd2aaa&id=3054b4ce-4516-4e41-9f8b-f059e8da8383&id=b0905486-ab0e-400f-b1b7-4556dfcc2738&id=b1d528f4-5378-4fd9-ba11-b7f9ca17c8a6&id=a5803a96-8016-4186-bfd5-da39a8b5
236f&id=95165036-d83a-4e6e-b041-bc5ee15a4edd&id=ed40a816-47d1-426d-9422-05b3d6969e1e&id=1422d82b-c4ea-4b0a-b7f7-94ddeb90f98a&id=5be4e07e-75d7-4510-9957-8ee63db9522a&id=319ebd95-d171-4548-8560-c89969dd13e6&id=d63
832ac-26a5-4e5e-94b0-7f9449a854ef&id=ac67a77c-cbfb-4571-b2b7-7a64c702dcc8&id=74dca901-89c6-4f50-bf6b-2f0c6c183370&id=f1e513bd-d13d-4ac2-9728-f4bd861febce&id=c22119c5-f424-4041-8bf6-ab39091b3a3a&id=877f1823-c7fd-
49bd-8526-f703dd717602&id=3c52f00c-f0ee-4cb5-a047-6580e4cf33bc&id=2092f145-b3ff-4453-9b5d-9780f9d75b2e&id=70907959-6929-4ba9-bc0f-28ee1279afee&id=cb16926d-26cd-4f2b-b531-b2ad448f89aa&id=bc2eea80-58c1-4007-9c6e-6
fcd4c59eaec&id=0a11b0b4-b0c6-41f4-b9f8-debe74c85b5d&id=3cae06d9-34d7-4e6a-bf8e-eabd50741fb4&id=775b7520-4c02-4ad2-ae95-af8c851c63e8&id=945cd7c2-7ea8-4e40-9faf-af9811395dc4&id=834d1896-4d28-4547-bc34-6575b3c0e16e
&id=b54ec961-a7b4-40b7-b75f-5f456743deb9&id=eeb3615f-5900-48a0-8c17-4de7364d49aa&id=9a5d0923-bbbe-4a4b-89f2-4423916b4c5a&id=c6df1155-884d-4691-aba2-52972df3e7aa&id=2ecc9506-2da3-4a63-a957-7df6c1cb1be5&id=6e16c3b
2-a6be-420b-bc28-cc1a2760b509&id=280a42a7-274e-40b3-a3e8-a932a0808c23&id=7b9625df-949d-45f4-8231-46b106fd899c&id=03872a0b-b052-4a69-8fca-0c8357953aa8&id=35dfd307-5a92-4e21-97bd-77598c499732&id=7d2cab2d-5e96-4358
-a9ea-3bbbf0329bf7&id=e51f9707-211d-4fd2-9ad4-7dd7658ab907&id=67e29b95-e834-416d-af65-1538decb6368&id=7f978f46-e03c-4792-980d-ddec3acd6027&id=05319783-b03e-4672-827f-6a12422a1a04&id=436c0e2c-87c2-422c-a9ac-3a983
f5c5c11&id=498ec730-a94f-49c7-a807-5276faee7fff&id=1190fef8-9d2c-4c41-b77e-9c6a1f7547c5&id=c2802f42-f239-4aa2-82be-bf23d9029f4e&id=34a437de-8b2b-4eae-a690-967128f8b9fe&id=8bc26bf9-9af8-4544-87af-16ba94ebaabb&id=
744e4129-d88f-4289-b7e5-640894fa60fd&id=38829265-9eb6-4a47-87b4-12c0e785cbde&id=c35c4070-ace6-4bc6-96e0-edca0863047b&id=a7388841-e337-4795-95bf-97b695e570f8&id=b661b334-a50b-47e7-92fd-42b7fa151518&id=979e8417-65
94-4a5d-b176-a7a5b220237c&id=4f155eec-8e40-4ad5-98f4-aa78288c4566&id=ccd5a517-9e4c-4d0c-9413-d5141d7bdb03&id=545c59d7-6af1-4797-a23b-60f08d84631b&id=df606aa9-95dc-40b6-91b1-8a1155c19e2e&id=923e4298-77b7-4a66-8da
e-b61c5ddec573&id=1af5ea4d-0b52-4abb-ad96-c14c2b6ce0a8&id=c0593606-7a62-4766-a018-8971d1bba742&id=5eee5769-3717-4c3a-a782-7be6c1493362&id=e04ebeb2-d631-4563-baea-a7b32cbdec0e&id=f63d3453-d3c3-48ce-a665-b90aec7bc
215&id=c3186293-82f1-4467-9a0f-5d0029db797a&id=0cdd950f-8971-4b98-ac7d-3343183f6330&id=45496e16-eb0e-4a9e-a3c2-6ff411ff330f&id=553bff53-a8dc-48e1-af3f-e50f96340cb9&id=03bb5bbb-35c2-4bbe-9b14-ec37da78ff2f&id=3b159c31-36e2-4874-ac25-84aedeb4eb35&id=c444d486-73cf-437c-a8bd-19447665fdba&id=304ef462-3306-4e7e-8130-e94113a18105&id=20990c82-75b4-441b-af3f-7a1460b1c63c&id=863d854b-4ea7-43b5-b8c5-93faec51c200&id=3ce7794c-9954-4698-a737-b594e50946a2&id=1af0a752-559b-4ef9-a53b-fbc82ff60ae5&id=2df86747-e03b-4b69-a5b8-0dc781fceab5&id=93f9159b-756f-42d0-86a2-3103e5b5f390&id=916105e7-64c2-400d-96f9-65b917943977&id=cb061bf3-54d7-4031-9f85-bd9d6e37ecf1&id=3020f613-0ccf-4422-bc34-5350590762ee&id=bdd91d00-a951-4ec7-b0f5-aa0802a9687a&id=ddcc56bf-7eea-44c6-a96b-6f939940a726&id=635d10aa-5990-4c6a-94d8-a72168afbc65&id=08f396e7-baf2-48a1-bf5f-d5e7fd92652c&id=58f93cbd-9231-47c4-8d01-95540bfbbf6c&id=4845ef79-65fa-4a8c-8a94-7aaa91e0aa5c&id=fbcbd2ff-7385-4ef4-846c-1708cb45b844&id=3f5d7d7b-9cba-4f00-ab46-92377ecf2f69&id=1793e819-99eb-4999-a71d-98de59c080f4&id=0c13c1b4-b028-48e5-82aa-7d8c0c7fa3ac&id=7e4a0a8e-ce81-46a2-90e8-7c6250811a27&id=28783b24-58e2-45b0-a5c8-b2c089abc539&id=7f623db8-2d01-41bd-a29d-a091b959a04b&id=a2b323c0-f257-4013-8039-44ae46ccc72e&id=6b871479-365f-4948-a08d-aa6e8a9a9729&id=24380d6a-f041-47ae-a8b2-16e117a9d1a6&id=3d0375d8-eb75-45ae-9d61-40172da2e589&id=dbe4fe06-a9fe-4820-b6d6-7eed26435028&id=4bccaabb-0c68-4030-ae07-d90caac50820&id=ed201f6b-3fbc-41aa-b707-855808901334&id=5d3ed27b-9a5d-4c65-94c7-2f45e541c310&id=7e06ace1-0362-495e-b61e-6252a098288c&id=0fa250dd-cccf-432f-8b65-d53bd9b5eb67&id=69468f57-3429-481f-92fc-56cf0009829b&id=a9eaabc9-55ff-4deb-b815-a449fa94561a&id=79bcc6c7-83f8-4e9d-afdc-6caf6c01e399&id=6167c2a6-df99-453d-a817-40f2baacf578&id=83cda383-784f-44c6-90b1-a0b52e719425&id=f618fc30-cf53-41bf-b1ac-6fe0d8155bba&id=24d8730b-69ea-4cb8-a2af-f3f18f0b9cd5&id=86007a04-7a32-4a50-afe9-53953de3159b&id=613489d2-ac6d-4a8b-8dcb-dc176495ccb8&id=eada55ac-a915-4fa4-a1de-08423ba3f2b9&id=e7f65ad7-4667-4a8e-b9bf-c7aef48f87cc&id=d992e785-1deb-463e-99ce-9701754d386d&id=8b97768f-72d8-4796-a702-6c6eaebafecd&id=413c37bb-3c0e-4d93-9924-6d6ec0d5cbe0&id=8a7d6949-7648-4476-afbb-b96c8f2cb19e&id=c49c836f-c959-4927-9c35-e5769069af9a&id=ea5d4a83-2bd6-4e01-9cf8-a21002dfc98b&id=742f031e-1c3d-464c-8080-82fe2cd48f0c&id=52448715-a274-481b-9a0c-60d6887d3d1d&id=e090fa4d-bb2e-4d42-a87f-deb3dc3a4cdc&id=4fcd7a89-b659-46c9-8aab-1c62d7e3faf1&id=e6ea731a-866c-4185-a3d9-d7c928853877&id=d3cf0c33-bc6b-435a-afdf-9d2463bb057e&id=fe9f9432-ac6d-4e11-8abb-e9c713eaef58&id=ae6e4033-af4e-4262-b592-b87589748e15&id=10450215-da82-4824-b7ca-391b2e479cc9&id=9e92d8e0-5fb4-4089-8bb4-7945e225e990&id=47ebd341-b6b7-4455-882a-fcb304436dda&id=7cde7dc1-358c-4ace-9b5b-7acceb4d9956&id=dbfa0f84-701a-48f5-bfb0-43326bd0b943&id=9648ef82-3273-47eb-be26-53956615fee4&id=873f022e-7abe-42b6-984c-f056c6da3666&id=3bb1d796-b13a-47de-84ab-ab503013f37a&id=ca8a9c20-63fc-40f4-a642-e893d310cdbf&id=1f9a66b5-c683-4059-b826-34fa63efdda2&id=b7f30192-851a-446e-8a49-aa8f67787f2a&id=9989befc-4516-4118-b9a3-ba6fd97759fb&id=ecbd4b1f-23d8-426a-b518-8906b3d6627a&id=c3788826-4d7a-4282-8b39-4ad2deaeda37&id=333958a8-83a6-4323-900d-59a593c650e1&id=938155f5-6b06-47b7-8224-69c0b723bb23&id=e2e7b1ba-6031-419b-8fae-b83ae3b75a28&id=cbb16e31-dfe9-4d50-8ab7-6d576e8a8320&id=cd26350c-60bf-4782-b991-3f19fb21f90e&id=9dafdc9d-6d09-4b7c-ba62-c4d314c8e184&id=bb451650-75b8-41e7-8678-8bfae75df112&id=148c0c09-4b99-4cf9-a91c-783250e3256e&id=9cdea621-2e61-4e85-bf1c-163149f176af&id=6d3e9d6d-8327-4e26-9a9c-2cc67beb75df&id=07c98c79-ddc4-422f-bfc2-5997bca5158b&id=aac7d065-5e3a-4aec-aaea-5bedbf6d5145&id=0bb799ab-e766-4168-baa4-84d4da6820f4&id=c792b716-d989-441e-a9b1-1ccc89dd2f75&id=a75094d6-9b13-4906-97db-a70af2e3c9bb&id=14d76135-2820-4a00-8090-5b9e277c7dfb&id=46c729a4-3416-4958-b35e-b7d86d786fc6&id=e61fa358-a005-4c7b-b3f8-ef1c9823d6b8&id=4cbda359-70d2-4085-a43e-fcaa4ae5eac1&id=517c38a2-062e-48f2-a38f-0d3486168ad7&id=ee2b38cd-1bb6-4058-9fc5-57ee5e047464&id=5360c41e-50c5-40de-a84d-0f45889bf232&id=ffde3857-b156-4d6c-83d5-3cd9956daed0&id=d32d767d-7784-4044-9444-01bf256950a1&id=31fda34c-e6f0-4792-a063-42108cd1df21&id=86114cd4-70bd-45d6-8fac-07694f51c7aa&id=7a196652-f361-4600-b037-5bb8b370a3ef&id=4b058487-1ac1-4a2f-b7da-fadae2b07fcf&id=f4f8c736-e47f-4d42-8ff2-4713fb298819&id=32af2f3a-8811-401c-bdee-16fd1328d5af&id=35cee640-1a8c-4ffe-89a4-b2bf1b58841b&id=e22b3432-087c-4410-98d3-010c2a3843c7&id=a3b38c60-a3c1-4a7e-b267-868cad1f8eca&id=87df942c-e444-4a2b-a01a-d702c72d43b7&id=55ed9254-c040-4550-8dde-c49f244454ca&id=5c646da9-d4ac-49c5-a5ae-2c8b0d2bd870&id=f264f6da-a7ab-4e03-afcc-9eee84c96155&id=None&id=5fc53690-959f-4e2a-8851-a2194c8be2f0&id=8c006e95-1b00-4d54-88ac-fb0ab1c136b3&id=9bb76439-c4cc-4ac3-9ae2-d9ae3b4e2ded&id=8ebda0ee-5412-4a66-9cbe-109560c90627&id=cde1cfde-52c6-4a38-9f5a-fafe64be7a59&id=8952662a-afc6-4e61-bcfb-9258d5a14c41&id=53d45393-1957-4db2-a93b-6b004c2f2b4b&id=da5e6b1b-c32d-4d2d-88db-8756a2730607&id=8b401f13-f8e4-41cd-848c-ce1161ec7688&id=a4bda800-7281-4075-89e6-c03555cd6310&id=905ee98b-50ae-4be2-9f18-dca7adec838a&id=33e8e29f-741c-49df-b03e-cc25017a6366&id=917d6bc5-8bf9-4927-be2a-f989d2a225d8&id=07434142-71ef-41aa-87ee-eceace1820e3&id=18090234-7c6e-444f-8361-a022ce9bcc0d&id=4285f9e7-4ced-4a68-8fe8-7c6511d7b2aa&id=ee8492a0-86dc-4418-a505-b16a97df83cb&id=044ed585-1f98-4f71-ab58-76d5914e2c2d&id=57155d1c-5df4-4a3a-b4d4-3530205ef81b&id=6e21a7ab-7862-4c4e-bbaf-809b576b6bbe&id=bb825df8-165d-46a7-a69b-dbaf6657ef9a

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/68357

[liuweicai (liuuweicai)]

https://review.openstack.org/#/c/95990/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/95990
Committed: https://git.openstack.org/cgit/openstack/python-neutronclient/commit/?id=8f38b2e90566bbd62421edfae45cd54500a31042
Submitter: Jenkins
Branch: master

commit 8f38b2e90566bbd62421edfae45cd54500a31042
Author: liuweicai <email address hidden>
Date: Wed May 28 10:07:43 2014 +0800

    Fix listing security group rules

    It was failing when there are too many security groups, since we're
    hitting a RequestURITooLong exception.

    Co-Authored-By: Vincent Untz <email address hidden>
    Co-Authored-By: Ilya Shakhat <email address hidden>
    Change-Id: If94b6a2ed6878efad1224e056ecf43e65e85e821
    Closes-Bug: 1271462

[Mark Kirkwood (mark-kirkwood)]

Even with this fix, a long list of security groups is busting rules display in the client.

e.g
$ neutron security-group-create group0
$ neutron security-group-create group1
...
$ neutron security-group-create group499

$ neutron --debug security-group-rule-list
An unknown exception occurred.

See attached for the debug trace.

[Akihiro Motoki (amotoki)]

Revert the status to New.

[Yushiro FURUKAWA (y-furukawa-2)]

TO: Ilya
CC: Akihiro

How is the progress? If there aren't any problems, could I change the
asignee to me? I'd like to post this patch as soon as possible.

[Ilya Shakhat (shakhat)]

Yushiro, please take the issue.

[Yushiro FURUKAWA (y-furukawa-2)]

@Ilya

Thank you. I'll post the patch.

[watanabe.isao (watanabe.isao)]

I'm in the same team with Yushiro FURUKAWA. I'll take over this issue under the permission of Yushiro FURUKAWA.

[watanabe.isao (watanabe.isao)]

This issue is caused because of following reason:
1. In securitygroup.py when creating sec_group_ids, we get both security group ids and remote group ids from security group rules of security group.
2. But when the attribute of one security group rule is ingress the remote group id will be Null.
3. The Null id will be add to sec_group_ids as "id=None".
4. After changing the type of sec_group_ids from set to list, the "id=None" will be unique to 1 phrase.
5. And this causes the calculation of chunk_size unstable.
6. When "id=None" is not include in the request phrase, an RequestURITooLong will occur.

In my test environment, the "id=None" comes up as the 6th id, where as in my teammate's is 10th.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/150728

[watanabe.isao (watanabe.isao)]

Will fix this bug in bug report [1], due to the comment of Akihiro Motoki in gerrit [2].

[1] https://bugs.launchpad.net/python-neutronclient/+bug/1415692
[2] https://review.openstack.org/#/c/150728/1

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/150728
Committed: https://git.openstack.org/cgit/openstack/python-neutronclient/commit/?id=59d7564202c9a5acd57f23a2708291d1d1cb6a54
Submitter: Jenkins
Branch: master

commit 59d7564202c9a5acd57f23a2708291d1d1cb6a54
Author: watanabe.isao <email address hidden>
Date: Tue Jan 27 09:37:20 2015 +0900

    Skip None id when getting security_group_ids

    When getting security_group_ids in securitygroup.py, a None id is
    included into the list due to the remote group id is Null when the
    security group rule direction is egress.
    And this causes the calculation of the number of chunk_size is
    1 more than the number it should be.
    Then the request page size excess the MAX request page size
    Then the RequestURITooLong exception happens again, and finally
    the commend fails.

    Co-Authored-By: Furukawa, Yushiro <email address hidden>
    Change-Id: Ib671cf3ddc6827793c4a86d2d3c8a34957f5dfd3
    Related-Bug: 1271462
    Closes-Bug: 1415692