security group create errors without description

[Dean Troyer]

security group create returns an error without --description supplied. This appears to be the server rejecting the request so we should set a default value rather than sending None.

  $ openstack security group create qaz
  ERROR: openstack Security group description is not a string or unicode (HTTP 400) (Request-ID: req-dee03de3-893a-4d58-bc3d-de87d09c3fb8)

Sent body:

  {"security_group": {"name": "qaz2", "description": null}}

[Dean Troyer (dtroyer)]

The description also can not be empty ("") so the default could be the security group name repeated?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/165956

[Steve Martinelli (stevemar)]

@Dean, whats the API say? I would think null is a valid description value

[Steve Martinelli (stevemar)]

AFAICT - the API says a request with no description in the body should be allowed:
http://developer.openstack.org/api-ref-compute-v2-ext.html

In the controller, the description value is not checked but goes to validate_property...
https://github.com/openstack/nova/blob/3e5062a8ddf94351389f7e201959591b26f338a4/nova/api/openstack/compute/contrib/security_groups.py#L190-L207

At validate_property though, it errors out when any value is not a string
https://github.com/openstack/nova/blob/d33184b6f890256b332e75bd46a58823cac663ff/nova/compute/api.py#L3855-L3880

To me, this seems like an issue with nova's implementation not matching it's API. In this case the API seems correct as description should always be optional.

We can and should proceed with the OSC fix as a band-aid solution, but should mark a TODO to change it when the nova side is resolved.

[Steve Martinelli (stevemar)]

Potentially, we could change novaclient to check if description is not null, and not include the key in the body. Not sure if this will result in a different error.

[Sean Dague (sdague)]

description is optional, optional does not mean it can be null, it means it shouldn't be in the payload at all.

[Steve Martinelli (stevemar)]

@sdague, novaclient seems to always pass in a description -> https://github.com/openstack/python-novaclient/blob/0a60aae852d2688861d0b4ba097a1a00529f0611/novaclient/v2/security_groups.py#L48

[Steve Martinelli (stevemar)]

@sdague, I made a change to novaclient to ensure the body does not include the description value.

You can see in the log below that the body is just: {"security_group": {"name": "tempo"}}

But, the result remains the same (400 with description is not a string or unicode):
2015-03-25 00:22:48.445 DEBUG nova.api.openstack.wsgi [req-0c501906-c662-4afd-a3fc-3b6e5e22caf9 admin admin] Action: 'create', calling method: <bound method SecurityGroupController.create of <nova.api.openstack.compute.contrib.security_groups.SecurityGroupController object at 0x7f0768586cd0>>, body: {"security_group": {"name": "tempo"}} from (pid=14913) _process_stack /opt/stack/nova/nova/api/openstack/wsgi.py:780
2015-03-25 00:22:48.447 INFO nova.api.openstack.wsgi [req-0c501906-c662-4afd-a3fc-3b6e5e22caf9 admin admin] HTTP exception thrown: Security group description is not a string or unicode
2015-03-25 00:22:48.447 DEBUG nova.api.openstack.wsgi [req-0c501906-c662-4afd-a3fc-3b6e5e22caf9 admin admin] Returning 400 to user: Security group description is not a string or unicode from (pid=14913) __call__ /opt/stack/nova/nova/api/openstack/wsgi.py:1166
2015-03-25 00:22:48.450 INFO nova.osapi_compute.wsgi.server [req-0c501906-c662-4afd-a3fc-3b6e5e22caf9 admin admin] 10.0.2.15 "POST /v2/36c7f6452c394b44ad4ae1f2bfe07800/os-security-groups HTTP/1.1" status: 400 len: 317 time: 0.1127591

This is the change that I made to novaclient

steve:python-novaclient$ git diff
diff --git a/novaclient/v2/security_groups.py b/novaclient/v2/security_groups.py
index 40d1e7f..0cd4960 100644
--- a/novaclient/v2/security_groups.py
+++ b/novaclient/v2/security_groups.py
@@ -45,7 +45,9 @@ class SecurityGroupManager(base.ManagerWithFind):
         :param description: description of the security group
         :rtype: the security group object
         """
- body = {"security_group": {"name": name, 'description': description}}
+ body = {"security_group": {"name": name}}
+ if description:
+ body['security_group']['description'] = description
         return self._create('/os-security-groups', body, 'security_group')

Double checked that it still works for the case with a description, and it does.

2015-03-25 00:25:28.870 DEBUG nova.api.openstack.wsgi [req-e2be9899-c50a-40fc-9eaa-258db57b5cf3 admin admin] Action: 'create', calling method: <bound method SecurityGroupController.create of <nova.api.openstack.compute.contrib.security_groups.SecurityGroupController object at 0x7f0768586cd0>>, body: {"security_group": {"name": "tempo", "description": "tempo_desc"}} from (pid=14913) _process_stack /opt/stack/nova/nova/api/openstack/wsgi.py:780
2015-03-25 00:25:28.871 DEBUG oslo_db.api [req-e2be9899-c50a-40fc-9eaa-258db57b5cf3 admin admin] Loading backend 'sqlalchemy' from 'nova.db.sqlalchemy.api' from (pid=14913) _load_backend /usr/local/lib/python2.7/dist-packages/oslo_db/api.py:214
2015-03-25 00:25:28.872 WARNING oslo_config.cfg [req-e2be9899-c50a-40fc-9eaa-258db57b5cf3 admin admin] Option "sql_connection" from group "DEFAULT" is deprecated. Use option "connection" from group "database".
2015-03-25 00:25:28.892 DEBUG oslo_db.sqlalchemy.session [req-e2be9899-c50a-40fc-9eaa-258d...

[Steve Martinelli (stevemar)]

Switched it back to new, let me know if you disagree.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/167487

[Davanum Srinivas (DIMS) (dims-v)]

Steve, Dean,

Is there anything to do in Nova itself?

thanks,
dims

[Steve Martinelli (stevemar)]

@Dims, yes there is.

This request to create a new security group fails: body = {"security_group": {"name": name}}.

This is actually shown in one of nova's tests: https://github.com/openstack/nova/blob/master/nova/tests/unit/api/openstack/compute/contrib/test_security_groups.py#L171-L179

    def test_create_security_group_with_no_description(self):
        sg = security_group_template()
        del sg['description']

        req = fakes.HTTPRequest.blank('/v2/fake/os-security-groups')
        self.assertRaises(webob.exc.HTTPBadRequest, self.controller.create,
                          req, {'security_group': sg})

So not including an optional attribute results in a bad request?

[Steve Martinelli (stevemar)]

I realize this is a low priority bug, and it's RC time. But I honestly can't show how this is any less valid.

[melanie witt (melwitt)]

According to the documentation for the security groups api, the description parameter is supposed to be optional:

http://developer.openstack.org/api-ref-compute-v2-ext.html#ext-os-security-groups

[Sean Dague (sdague)]

Right, optional means you are allowed to not provide the parameter. Not that you can provide it as null. There is a python-novaclient bug here.

[melanie witt (melwitt)]

Hi Sean, in comment #8 Steve said he still got a 400 from nova after omitting the description parameter from the request body.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Steve Martinelli (<email address hidden>) on branch: master
Review: https://review.openstack.org/167487
Reason: meh

[melanie witt (melwitt)]

There is also a bug in nova. Both the documentation and the data model show "description" as an optional field, but the implementation erroneously requires it as it calls a validate_property function on "description" whether it was provided or not:

http://git.openstack.org/cgit/openstack/nova/tree/nova/api/openstack/compute/contrib/security_groups.py#n201
http://git.openstack.org/cgit/openstack/nova/tree/nova/compute/api.py#n3838

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/165956
Committed: https://git.openstack.org/cgit/openstack/python-openstackclient/commit/?id=11c39530f5f97a14d534c8d5b7160a1e74f6cdf8
Submitter: Jenkins
Branch: master

commit 11c39530f5f97a14d534c8d5b7160a1e74f6cdf8
Author: Dean Troyer <email address hidden>
Date: Thu Mar 19 11:46:05 2015 -0500

    Fix security group create description bug

    --description is optional in our CLI but the server requires it to be
    non-empty. Set a default value of the given name.

    Closes-Bug: #1434172
    Change-Id: I81507a77ad8d815000ff411784ae71e229c77f78

[melanie witt (melwitt)]

We can work around the nova bug in novaclient by passing a default value.

[Steve Martinelli (stevemar)]

@melwitt we worked around this in OSC by sending a description that was the same as the name (if no name was passed in).

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/183909

[OpenStack Infra (hudson-openstack)]

Change abandoned by Adrien Vergé (<email address hidden>) on branch: master
Review: https://review.openstack.org/183909

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/262655

[Sean Dague (sdague)]

We've updated the Nova docs that this is no longer optional.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sean Dague (<email address hidden>) on branch: master
Review: https://review.openstack.org/279552

[OpenStack Infra (hudson-openstack)]

Change abandoned by Kevin L. Mitchell (<email address hidden>) on branch: master
Review: https://review.openstack.org/262655
Reason: Not updated since December.

[Takashi Natsume (natsume-takashi)]

In python-novaclient, the "nova secgroup-create" command has been removed since https://review.openstack.org/#/c/437145/.

So it is no longer necessary to fix it in python-novaclient.