OpenStackSDK refactoring caused various OSC networking commands to fail

Bug #1652317 reported by Valeriy Ponomaryov on 2016-12-23
88
This bug affects 16 people
Affects Status Importance Assigned to Milestone
Manila
Critical
Unassigned
OpenStack SDK
New
Undecided
Unassigned
python-openstackclient
New
High
Unassigned
tacker
High
Unassigned

Bug Description

After merging upper constraint change for python-openstacksdk project (which is used by python-openstackclient) 0.9.10 -> 0.9.11
 [1] our devstack installation started failing where we try to create security group for manila.
here is logs:
http://logs.openstack.org/86/414286/2/gate/gate-manila-tempest-minimal-dsvm-lvm-ubuntu-xenial/a79dc50/logs/devstacklog.txt.gz#_2016-12-23_13_26_10_182

Where we see following error:

$ openstack security group create manila-service --description 'manila-service description'
'SecurityGroup' object has no attribute 'keys'

So, it can be said, that it is regression problem in sdk project, because it started returning class instances instead of dicts as was before.

[1] https://review.openstack.org/#/c/414366/

[UPD]
One more breakage:

$ openstack security group rule list manila-service
invalid literal for int() with base 10: '137:139'

tags: added: devstack
Changed in manila:
importance: Undecided → Critical
milestone: none → ocata-3
description: updated
description: updated
description: updated

Not an SDK bug

Changed in python-openstacksdk:
status: New → Invalid

Brian, why do you think so? It is not enough just to say it is not bug of SDK project.
Why do you think it is not "regression" bug where was changed type of objects returned by SDK?

Brian Curtin (brian.curtin) wrote :

backwards compatibility isn't guaranteed pre-1.0, this was intentionally changed as part of a refactoring.

Reviewed: https://review.openstack.org/414610
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=10b14bb50e95d2a30bba4e4c37a7de9c4b564691
Submitter: Jenkins
Branch: master

commit 10b14bb50e95d2a30bba4e4c37a7de9c4b564691
Author: vponomaryov <email address hidden>
Date: Fri Dec 23 17:07:39 2016 +0200

    [Devstack] Workaround osclient breakage

    Creation of security group using openstack client is broken. So,
    use nova client for it temporary while 'related' bug is not fixed.

    Change-Id: I51b5be5157bd49d5835b30eaa8b6b0e7185cabdf
    Related-Bug: #1652317

summary: - Creation of security group in devstack is broken using openstackclient
+ OpenStackSDK refactoring caused various OSC networking commands to fail
Changed in python-openstackclient:
importance: Undecided → High
Steve Martinelli (stevemar) wrote :

To summarize:

We're going to block SDK v0.9.11 for now, since the following commands will fail:

 - quota_set
 - quota show
 - network agent set
 - network agent show
 - network qos *
 - network qos rule *
 - port *
 - security group *
 - security group rule *

With SDK v0.9.10 installed, the following commands will fail:

 - qos_rule_type_list
 - network_service_provider_list
 - ip_availability_show

The latter seems more acceptable since it won't block any gates, and fewer impacted commands overall. We're going to re-work the commands to work with both versions of the SDK.

This was unfortunately not caught by our functional tests as they were not running (problem in the test hook), they are running now. See https://review.openstack.org/#/c/414652/.

Sridhar Ramaswamy (srics-r) wrote :

Affecting tacker gate functional tests

Changed in tacker:
importance: Undecided → High
Sridhar Ramaswamy (srics-r) wrote :

Is there an ETA for this bug? This is blocking tacker dsvm tests

Andrey Kurilin (andreykurilin) wrote :

@srics-r: after merging https://review.openstack.org/#/c/414621/ all gates should be unblocked. Can you check openstacksdk version in broken gates?

Dean Troyer (dtroyer) wrote :

https://review.openstack.org/#/c/415508/ reverts the skip for the commands that broke in SDK 0.9.11. This leaves the following still skipped for SDK 0.9.10:

 - qos_rule_type_list (https://bugs.launchpad.net/python-openstackclient/+bug/1653137)
 - network_service_provider_list (https://bugs.launchpad.net/python-openstackclient/+bug/1653138)
 - ip_availability_show (https://bugs.launchpad.net/python-openstackclient/+bug/1653139)

Moving forward, individual bugs for these have been opened (as listed above) to track their resulution.

Sridhar Ramaswamy (srics-r) wrote :

@andreykurilin: confirming the tacker gate tests are passing now, thanks!

Changed in tacker:
status: New → Fix Committed
Andrey Kurilin (andreykurilin) wrote :

Moved openstacksdk bug-report to New, since latest release 0.9.12 broke updating neutorn quotas again(it just doesn't fix issues of 0.9.11 release)

Changed in python-openstacksdk:
status: Invalid → New
Andrey Kurilin (andreykurilin) wrote :
Download full text (6.7 KiB)

Logs that can help for investigation:

openstack --debug --os-interface admin quota set --networks -1 --subnets -1 --routers -1 --floating-ips -1 --subnetpools -1 --secgroups -1 --secgroup-rules -1 --ports -1 rally-test-project-1

ESP BODY: {"NeutronError": {"message": "Unrecognized attribute(s) 'id'", "type": "HTTPBadRequest", "detail": ""}}
2017-01-09 13:34:17.292661 | 2017-01-09 13:34:17.291 |
2017-01-09 13:34:17.293790 | 2017-01-09 13:34:17.293 | Request returned failure status: 400
2017-01-09 13:34:17.294862 | 2017-01-09 13:34:17.294 | HttpException: Bad Request
2017-01-09 13:34:17.295931 | 2017-01-09 13:34:17.295 | Traceback (most recent call last):
2017-01-09 13:34:17.297009 | 2017-01-09 13:34:17.296 | File "/usr/local/lib/python2.7/dist-packages/cliff/app.py", line 400, in run_subcommand
2017-01-09 13:34:17.298138 | 2017-01-09 13:34:17.297 | result = cmd.run(parsed_args)
2017-01-09 13:34:17.299359 | 2017-01-09 13:34:17.299 | File "/usr/local/lib/python2.7/dist-packages/osc_lib/command/command.py", line 41, in run
2017-01-09 13:34:17.300660 | 2017-01-09 13:34:17.300 | return super(Command, self).run(parsed_args)
2017-01-09 13:34:17.301795 | 2017-01-09 13:34:17.301 | File "/usr/local/lib/python2.7/dist-packages/cliff/command.py", line 90, in run
2017-01-09 13:34:17.303381 | 2017-01-09 13:34:17.302 | return self.take_action(parsed_args) or 0
2017-01-09 13:34:17.304780 | 2017-01-09 13:34:17.304 | File "/usr/local/lib/python2.7/dist-packages/openstackclient/common/quota.py", line 185, in take_action
2017-01-09 13:34:17.305888 | 2017-01-09 13:34:17.305 | **network_kwargs)
2017-01-09 13:34:17.307049 | 2017-01-09 13:34:17.306 | File "/usr/local/lib/python2.7/dist-packages/openstack/network/v2/_proxy.py", line 2025, in update_quota
2017-01-09 13:34:17.308530 | 2017-01-09 13:34:17.308 | return self._update(_quota.Quota, quota, **attrs)
2017-01-09 13:34:17.309591 | 2017-01-09 13:34:17.309 | File "/usr/local/lib/python2.7/dist-packages/openstack/proxy2.py", line 36, in check
2017-01-09 13:34:17.310703 | 2017-01-09 13:34:17.310 | return method(self, expected, actual, *args, **kwargs)
2017-01-09 13:34:17.311864 | 2017-01-09 13:34:17.311 | File "/usr/local/lib/python2.7/dist-packages/openstack/proxy2.py", line 173, in _update
2017-01-09 13:34:17.313098 | 2017-01-09 13:34:17.312 | return res.update(self.session)
2017-01-09 13:34:17.314221 | 2017-01-09 13:34:17.313 | File "/usr/local/lib/python2.7/dist-packages/openstack/resource2.py", line 663, in update
2017-01-09 13:34:17.315444 | 2017-01-09 13:34:17.315 | json=request.body, headers=request.headers)
2017-01-09 13:34:17.316935 | 2017-01-09 13:34:17.316 | File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 726, in put
2017-01-09 13:34:17.318078 | 2017-01-09 13:34:17.317 | return self.request(url, 'PUT', **kwargs)
2017-01-09 13:34:17.319321 | 2017-01-09 13:34:17.318 | File "/usr/local/lib/python2.7/dist-packages/openstack/session.py", line 77, in map_exceptions_wrapper
2017-01-09 13:34:17.320508 | 2017-01-09 13:34:17.320 | http_status=e.http_status, cause=e)
2017-01-09 13:34:17.321794 | 2017-01-09 13:34:17.321 | Htt...

Read more...

Clark Boylan (cboylan) wrote :

Related to this the nodepool/shade/ansible set of jobs began to fail. They rely the demo user that devstack sets up and expect things like networking to be preconfigured. It appears that these jobs started failing when OSC was updated and have begun passing now that that version is excluded again.

It appears that they failed because the demo user did not have a valid "private" network, but did have subnets for the private networks. These subnets were then attached to private networks "owned" by the admin user. So somewhere the client/sdk/api crossed the wires on that and ended up "wiring" up the networking incorrectly for the demo user.

Doug Wilmot (dwilmot) wrote :

Also impacts openstack subnet show:

openstack subnet show dec8c058-a578-4927-abd5-19780fd9dee1 (returns 'Subnet' object has no attribute 'keys')

However the older neutron cmd still works but rather not modify our scripts

[root@OpenStack-Launcher openstack-ns]# neutron subnet-show dec8c058-a578-4927-abd5-19780fd9dee1 <<<<
+-------------------+----------------------------------------------------+
| Field | Value |
+-------------------+----------------------------------------------------+
| allocation_pools | {"start": "11.255.255.2", "end": "11.255.255.254"} |
| cidr | 11.255.255.0/24 |
| dns_nameservers | |
| enable_dhcp | False |
| gateway_ip | 11.255.255.1 |
| host_routes | |
| id | dec8c058-a578-4927-abd5-19780fd9dee1 |
| ip_version | 4 |
| ipv6_address_mode | |
| ipv6_ra_mode | |
| name | basic7-dwnew-subnet_dummy-or2jrteviwzr |
| network_id | f1c13eac-4a17-4c86-82a3-61a865195da4 |
| tenant_id | 8d33baa96b78444ab266538f6c4a23e0 |
+-------------------+----------------------------------------------------+

Reedip (reedip-banerjee) wrote :

@Doug : I verified Openstack SDK 0.9.12 and subnet show seems to work fine

Successfully installed openstacksdk-0.9.12 python-openstacksdk-0.5.2
[outofmemory@localhost devstack]$ openstack subnet show private-subnet
+-------------------+--------------------------------------+
| Field | Value |
+-------------------+--------------------------------------+
| allocation_pools | 10.0.0.2-10.0.0.62 |
| cidr | 10.0.0.0/26 |
| created_at | 2017-01-13T16:17:57Z |
| description | |
| dns_nameservers | |
| enable_dhcp | True |
| gateway_ip | 10.0.0.1 |
| host_routes | |
| id | d8136226-acc7-4ab3-b808-fba25533cba7 |
| ip_version | 4 |
| ipv6_address_mode | None |
| ipv6_ra_mode | None |
| name | private-subnet |
| network_id | f77a1585-dd37-4072-b330-503b6eb2b043 |
| project_id | 25f495a77ad1450c90cb1b1ac1442b2f |
| revision_number | 2 |
| segment_id | None |
| service_types | |
| subnetpool_id | c1b01670-1559-4494-a708-47fcf10ffc3d |
| updated_at | 2017-01-13T16:17:57Z |
+-------------------+--------------------------------------+

Reviewed: https://review.openstack.org/419911
Committed: https://git.openstack.org/cgit/openstack/python-openstacksdk/commit/?id=0a7483ed952c82d006b0c55ce1ff77212a0c7729
Submitter: Jenkins
Branch: master

commit 0a7483ed952c82d006b0c55ce1ff77212a0c7729
Author: Reedip <email address hidden>
Date: Wed Jan 18 03:49:23 2017 -0500

    Fix Setting Quotas in Neutron

    Currently Quota Set command doenst work in SDK version 0.9.12
    as the request formed for the Neutron API is not correct.
    This patch attempts to fix the same.

    Change-Id: Id58b05bcdbfee73cb9b93dd5533b4c7d93dd03aa
    Partial-Bug:#1652317
    Closes-Bug:#1655445

Reviewed: https://review.openstack.org/420301
Committed: https://git.openstack.org/cgit/openstack/python-openstacksdk/commit/?id=6d185df4983ef824d629769c11dde5dd9a6297b9
Submitter: Jenkins
Branch: master

commit 6d185df4983ef824d629769c11dde5dd9a6297b9
Author: Reedip <email address hidden>
Date: Wed Jan 18 03:52:23 2017 -0500

    Add project ID in QuotaDefault requests

    Currently QuotaDefault doesnt need ID to work with,
    but it needs the URI to have the project ID so that it
    can reference the same in the Neutron API.
    This patch fixes this issue.

    Partial-Bug:#1652317
    Change-Id: I9288b1c9fc64c6326231d535d9c8574b24afeb3a

Changed in manila:
milestone: ocata-3 → ocata-rc1

Manila project not affected anymore, fix in lib solved problem.

Changed in manila:
status: New → Invalid
Changed in manila:
milestone: ocata-rc1 → none
Changed in tacker:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers