rate_limit error when executing openstack network commands

Bug #1819020 reported by Ghada Khalil on 2019-03-07
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
StarlingX
Critical
Scott Little

Bug Description

Title
-----
rate_limit error when executing openstack network commands
This issue was reported by Shuicheng Lin

Brief Description
-----------------
When attempting to run openstack networking commands as part of system provisioning, an error is returned as follows:
controller-0:/home/wrsroot# openstack network segment range create ${PHYSNET0}-a --network-type vlan --physical-network ${PHYSNET0} --minimum 400 --maximum 499 --private --project ${ADMINID}
__init__() got an unexpected keyword argument 'rate_limit'
controller-0:/home/wrsroot# openstack network list
__init__() got an unexpected keyword argument 'rate_limit'

Severity
--------
Major: This impacts configuring networking on recent StarlingX loads; started on Mar 5.

Steps to Reproduce
------------------
run any openstack network command

Expected Behavior
------------------
Command should pass

Actual Behavior
----------------
Commands fail as per above

Reproducibility
---------------
100% reproducible

System Configuration
--------------------
Any

Branch/Pull Time/Commit
-----------------------
master as of March 5 afternoon

From Joseph Richard:
In https://github.com/openstack/openstacksdk/blame/da4ec3e70c20db3ea37cd54dec6dc1e10b309274/lower-constraints.txt, a dependency was added for keystoneauth1==3.13.0, however it looks like StarlingX is only at 3.10.0. Because we are building openstacksdk from master, we are hitting this. We will likely need to upversion keystoneauth1 to resolve this.

Timestamp/Logs
--------------
Not required; issue is reproducible

Ghada Khalil (gkhalil) wrote :

From Don Penney:
This should just be a matter of updating the RPM version being downloaded, as we aren’t rebuilding it:
stx-tools/centos-mirror-tools/rpms_3rdparties.lst:
python2-keystoneauth1-3.10.0-1.el7.noarch.rpm#https://cbs.centos.org/kojifiles/packages/python-keystoneauth1/3.10.0/1.el7/noarch/python2-keystoneauth1-3.10.0-1.el7.noarch.rpm

There should be a check of other RPM versions, as well, though, in case anything else needs to be updated.

Changed in starlingx:
importance: Undecided → Critical
description: updated
description: updated
Ghada Khalil (gkhalil) on 2019-03-07
description: updated
Changed in starlingx:
assignee: nobody → Scott Little (slittle1)
Ghada Khalil (gkhalil) on 2019-03-07
tags: added: stx.2019.05 stx.build stx.networking
tags: added: stx.distro.openstack
removed: stx.build
Scott Little (slittle1) wrote :

No pre-built rpms available for 3.13 or newer at this time. We will have to compile our own rpm from git or tarball.

This issue is reproducible using our test framework, same behaviour with neutron commands and openstack commands:

$: openstack network list
__init__() got an unexpected keyword argument 'rate_limit'

$: neutron net-list
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
Connection to neutron failed: Failed to connect Neutron server

Joseph Richard (josephrichard) wrote :

The issue does not affect neutronclient.

That error you are seeing looks like you have the wrong endpoint.
Set OS_CLOUD=openstack_helm and do not source /etc/nova/openrc when accessing neutron.

https://wiki.openstack.org/w/index.php?title=StarlingX/Containers/Installation#Verify_the_cluster_endpoints

Fix proposed to branch: master
Review: https://review.openstack.org/642036

Changed in starlingx:
status: New → In Progress
Scott Little (slittle1) wrote :

    Investigation showed that there is currently no centos rpm
providing python2-keystoneauth1-3.13.0 or better. Solution
is to build python2-keystoneauth1-3.13.1 from openstack's git.
We can revert to a downloaded rpm when one becomes available.

Code reviews:

stx-manifest: https://review.openstack.org/642040
stx-upstream: https://review.openstack.org/642036
stx-tools: https://review.openstack.org/642037

Reviewed: https://review.openstack.org/642040
Committed: https://git.openstack.org/cgit/openstack/stx-manifest/commit/?id=03eb066b2c5d934709d6440002777db94887804c
Submitter: Zuul
Branch: master

commit 03eb066b2c5d934709d6440002777db94887804c
Author: Scott Little <email address hidden>
Date: Thu Mar 7 12:38:45 2019 -0500

    Upgrade python-keystoneauth1 to 3.13.0

    Issue:
       'rate_limit' error when executing openstack network commands.
    Failure was tracked to a failure to satisfy the lower-constraints.txt
    of openstacksdk, keystoneauth1==3.13.0. We currently only provide
    python2-keystoneauth1-3.10.0-1.el7.noarch.rpm.

    Solution:
        Investigation showed that there is currently no centos rpm
    providing python2-keystoneauth1-3.13.0 or better. Solution
    is to build python2-keystoneauth1-3.13.1 from openstack's git.
    We can revert to a downloaded rpm when one becomes available.

    Closes-Bug: 1819020
    Change-Id: I715ee3f05dacaa320083dd40c2b23f0f599b910e
    Signed-off-by: Scott Little <email address hidden>

Changed in starlingx:
status: In Progress → Fix Released

Reviewed: https://review.openstack.org/642036
Committed: https://git.openstack.org/cgit/openstack/stx-upstream/commit/?id=eb6883ff55b9fa7a10dcccac685cfcebd06cfd62
Submitter: Zuul
Branch: master

commit eb6883ff55b9fa7a10dcccac685cfcebd06cfd62
Author: Scott Little <email address hidden>
Date: Thu Mar 7 13:10:06 2019 -0500

    Upgrade python-keystoneauth1 to 3.13.0

    Issue:
       'rate_limit' error when executing openstack network commands.
    Failure was tracked to a failure to satisfy the lower-constraints.txt
    of openstacksdk, keystoneauth1==3.13.0. We currently only provide
    python2-keystoneauth1-3.10.0-1.el7.noarch.rpm.

    Solution:
        Investigation showed that there is currently no centos rpm
    providing python2-keystoneauth1-3.13.0 or better. Solution
    is to build python2-keystoneauth1-3.13.1 from openstack's git.
    We can revert to a downloaded rpm when one becomes available.

    Note:
        I'm taking this opportunity to clean up stale patches
    that pre-date python2-keystoneauth1-3.10.0. None of these
    where in use.

    Closes-Bug: 1819020
    Change-Id: Ie2300f352dffc2fc87fbc84a9fc98d2a4e9484d4
    Depends-On: I715ee3f05dacaa320083dd40c2b23f0f599b910e
    Signed-off-by: Scott Little <email address hidden>

Reviewed: https://review.openstack.org/642037
Committed: https://git.openstack.org/cgit/openstack/stx-tools/commit/?id=67e57499198a700a36a5464e17b7fab30d67685b
Submitter: Zuul
Branch: master

commit 67e57499198a700a36a5464e17b7fab30d67685b
Author: Scott Little <email address hidden>
Date: Thu Mar 7 12:50:35 2019 -0500

    Upgrade python-keystoneauth1 to 3.13.0

    Issue:
       'rate_limit' error when executing openstack network commands.
    Failure was tracked to a failure to satisfy the lower-constraints.txt
    of openstacksdk, keystoneauth1==3.13.0. We currently only provide
    python2-keystoneauth1-3.10.0-1.el7.noarch.rpm.

    Solution:
        Investigation showed that there is currently no centos rpm
    providing python2-keystoneauth1-3.13.0 or better. Solution
    is to build python2-keystoneauth1-3.13.1 from openstack's git.
    We can revert to a downloaded rpm when one becomes available.

    Closes-Bug: 1819020
    Change-Id: I30332820a8f4ae877b0719148022bade1943eadb
    Depends-On: Ie2300f352dffc2fc87fbc84a9fc98d2a4e9484d4
    Signed-off-by: Scott Little <email address hidden>

Change abandoned by Saul Wold (<email address hidden>) on branch: master
Review: https://review.openstack.org/642368
Reason: Did something wrong will try again tomorrow.

Change abandoned by Saul Wold (<email address hidden>) on branch: master
Review: https://review.openstack.org/642365
Reason: Did something wrong will try again tomorrow.

Change abandoned by Saul Wold (<email address hidden>) on branch: master
Review: https://review.openstack.org/642364
Reason: Did something wrong will try again tomorrow.

Ken Young (kenyis) on 2019-04-05
tags: added: stx.2.0
removed: stx.2019.05
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers