tags and not-tags cannot work properly
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| | OpenStack Compute (nova) |
High
|
Zhenyu Zheng | ||
| | Newton |
High
|
Matt Riedemann | ||
| | Ocata |
High
|
Matt Riedemann | ||
Bug Description
When there is no tags in the system and user uses "nova list --not-tags xxx",
HTTP 500 will return:
root@SZX1000291
/usr/local/
warnings.
ERROR (ClientException): Unexpected API Error. Please report this at http://
<type 'exceptions.
nova-api Logs:
2017-04-14 09:34:29.952 111211 DEBUG oslo_db.
2017-04-14 09:34:29.955 111211 ERROR nova.api.
2017-04-14 09:34:29.955 111211 ERROR nova.api.
2017-04-14 09:34:29.955 111211 ERROR nova.api.
2017-04-14 09:34:29.955 111211 ERROR nova.api.
2017-04-14 09:34:29.955 111211 ERROR nova.api.
2017-04-14 09:34:29.955 111211 ERROR nova.api.
2017-04-14 09:34:29.955 111211 ERROR nova.api.
2017-04-14 09:34:29.955 111211 ERROR nova.api.
2017-04-14 09:34:29.955 111211 ERROR nova.api.
2017-04-14 09:34:29.955 111211 ERROR nova.api.
2017-04-14 09:34:29.955 111211 ERROR nova.api.
2017-04-14 09:34:29.955 111211 ERROR nova.api.
2017-04-14 09:34:29.955 111211 ERROR nova.api.
2017-04-14 09:34:29.955 111211 ERROR nova.api.
2017-04-14 09:34:29.955 111211 ERROR nova.api.
2017-04-14 09:34:29.955 111211 ERROR nova.api.
2017-04-14 09:34:29.955 111211 ERROR nova.api.
2017-04-14 09:34:29.955 111211 ERROR nova.api.
2017-04-14 09:34:29.955 111211 ERROR nova.api.
2017-04-14 09:34:29.955 111211 ERROR nova.api.
2017-04-14 09:34:29.955 111211 ERROR nova.api.
2017-04-14 09:34:29.955 111211 ERROR nova.api.
2017-04-14 09:34:29.955 111211 ERROR nova.api.
2017-04-14 09:34:29.955 111211 ERROR nova.api.
2017-04-14 09:34:29.955 111211 ERROR nova.api.
2017-04-14 09:34:29.955 111211 ERROR nova.api.
2017-04-14 09:34:29.955 111211 ERROR nova.api.
2017-04-14 09:34:29.955 111211 ERROR nova.api.
2017-04-14 09:34:29.955 111211 ERROR nova.api.
2017-04-14 09:34:29.955 111211 ERROR nova.api.
2017-04-14 09:34:29.955 111211 ERROR nova.api.
2017-04-14 09:34:29.955 111211 ERROR nova.api.
2017-04-14 09:34:29.955 111211 ERROR nova.api.
2017-04-14 09:34:29.955 111211 ERROR nova.api.
2017-04-14 09:34:29.955 111211 ERROR nova.api.
2017-04-14 09:34:29.955 111211 ERROR nova.api.
2017-04-14 09:34:29.958 111211 INFO nova.api.
<type 'exceptions.
2017-04-14 09:34:29.959 111211 DEBUG nova.api.
<type 'exceptions.
2017-04-14 09:34:29.960 111211 INFO nova.osapi_
| Changed in nova: | |
| assignee: | nobody → Zhenyu Zheng (zhengzhenyu) |
| tags: | added: api cells ocata-backport-potential |
| Changed in nova: | |
| importance: | Undecided → High |
| status: | New → Confirmed |
| Zhenyu Zheng (zhengzhenyu) wrote : | #1 |
| summary: |
- not-tags cannot work properly + tags and not-tags cannot work properly |
Fix proposed to branch: master
Review: https:/
| Changed in nova: | |
| status: | Confirmed → In Progress |
Related fix proposed to branch: master
Review: https:/
| tags: | added: newton-backport-potential |
| Jay Pipes (jaypipes) wrote : | #4 |
Kevin, would it be possible to post the actual error message from the logs instead of the returned 500 error? Thanks!
| Changed in nova: | |
| assignee: | Zhenyu Zheng (zhengzhenyu) → Matt Riedemann (mriedem) |
Related fix proposed to branch: stable/ocata
Review: https:/
Fix proposed to branch: stable/ocata
Review: https:/
Fix proposed to branch: stable/newton
Review: https:/
| Changed in nova: | |
| assignee: | Matt Riedemann (mriedem) → Zhenyu Zheng (zhengzhenyu) |
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: master
commit 01dd1a05a213c0c
Author: Matt Riedemann <email address hidden>
Date: Sun Apr 16 20:45:25 2017 -0400
Add regression test for server filtering by tags bug 1682693
There was a latent bug in the DB API code such that when we
process filters when listing instances, the various tags
filters have their values popped out of the filters dict and
the values (which are lists) for the filter also have the
first item in the list popped out to build the query.
This latent bug was exposed in Newton when we started listing
instances in the API from both cell0 and the main cell database,
because the query to cell0 would pop an item and then it
would not be in the 2nd query to the main cell database. If we
only had one tag in the filter list, we get an IndexError on
the 2nd pop() call.
Note that we also use the build_requests table in the API to
list instances, but there would not be any tagged servers in
that table since a server has to be ACTIVE before you can tag it,
and build_requests only exist until the instance is put into a
cell and starts building there (so it won't be ACTIVE yet).
Change-Id: I65bdfadd406368
Related-Bug: #1682693
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: master
commit c4820305d2f9ee8
Author: Kevin_Zheng <email address hidden>
Date: Fri Apr 14 11:57:59 2017 +0800
Use deepcopy when process filters in db api
In db API when we process filters, we didn't
use deepcopy. In cases of "tags" and "not-tags"
we used pop to get the first tag, filtered out
results, and then joined with other tags for
later filtering. When we did pop(), the original
value was deleted, the key "tags"/"not-tags" remains.
In the cell scenario, both single cell(we will
query cell0 and the other cell) and multicell,
as we have to query all the cells in a loop and
the tags list in the filter will keep popping,
this will lead to either a HTTP 500 error(popping
from an empty list) or incorrect result(when
number of tags in the list is larger than cell
number, no HTTP 500 will show, but the filter
results for each cell will be different as
each loop will pop one tag).
closes-bug: #1682693
Change-Id: Ia2738dd0c7d184
| Changed in nova: | |
| status: | In Progress → Fix Released |
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: stable/ocata
commit c9620d57c84378c
Author: Matt Riedemann <email address hidden>
Date: Sun Apr 16 20:45:25 2017 -0400
Add regression test for server filtering by tags bug 1682693
There was a latent bug in the DB API code such that when we
process filters when listing instances, the various tags
filters have their values popped out of the filters dict and
the values (which are lists) for the filter also have the
first item in the list popped out to build the query.
This latent bug was exposed in Newton when we started listing
instances in the API from both cell0 and the main cell database,
because the query to cell0 would pop an item and then it
would not be in the 2nd query to the main cell database. If we
only had one tag in the filter list, we get an IndexError on
the 2nd pop() call.
Note that we also use the build_requests table in the API to
list instances, but there would not be any tagged servers in
that table since a server has to be ACTIVE before you can tag it,
and build_requests only exist until the instance is put into a
cell and starts building there (so it won't be ACTIVE yet).
Change-Id: I65bdfadd406368
Related-Bug: #1682693
(cherry picked from commit 01dd1a05a213c0c
| tags: | added: in-stable-ocata |
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: stable/ocata
commit 01ffbdf8be1cec5
Author: Kevin_Zheng <email address hidden>
Date: Fri Apr 14 11:57:59 2017 +0800
Use deepcopy when process filters in db api
In db API when we process filters, we didn't
use deepcopy. In cases of "tags" and "not-tags"
we used pop to get the first tag, filtered out
results, and then joined with other tags for
later filtering. When we did pop(), the original
value was deleted, the key "tags"/"not-tags" remains.
In the cell scenario, both single cell(we will
query cell0 and the other cell) and multicell,
as we have to query all the cells in a loop and
the tags list in the filter will keep popping,
this will lead to either a HTTP 500 error(popping
from an empty list) or incorrect result(when
number of tags in the list is larger than cell
number, no HTTP 500 will show, but the filter
results for each cell will be different as
each loop will pop one tag).
closes-bug: #1682693
Change-Id: Ia2738dd0c7d184
(cherry picked from commit c4820305d2f9ee8
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: stable/newton
commit 6d1ce4cf5cfa952
Author: Kevin_Zheng <email address hidden>
Date: Fri Apr 14 11:57:59 2017 +0800
Use deepcopy when process filters in db api
In db API when we process filters, we didn't
use deepcopy. In cases of "tags" and "not-tags"
we used pop to get the first tag, filtered out
results, and then joined with other tags for
later filtering. When we did pop(), the original
value was deleted, the key "tags"/"not-tags" remains.
In the cell scenario, both single cell(we will
query cell0 and the other cell) and multicell,
as we have to query all the cells in a loop and
the tags list in the filter will keep popping,
this will lead to either a HTTP 500 error(popping
from an empty list) or incorrect result(when
number of tags in the list is larger than cell
number, no HTTP 500 will show, but the filter
results for each cell will be different as
each loop will pop one tag).
closes-bug: #1682693
Conflicts:
NOTE(mriedem): The conflict on the functional regression test
is due to I65bdfadd406368
backported to Newton. That was not backported simply because it
relies on too many fixtures to stub out a real cells environment
that trying to get it to work in Newton is not worth it. This
fix is still unit tested which is sufficient.
Change-Id: Ia2738dd0c7d184
(cherry picked from commit c4820305d2f9ee8
(cherry picked from commit 01ffbdf8be1cec5
This issue was fixed in the openstack/nova 15.0.4 release.
This issue was fixed in the openstack/nova 14.0.6 release.
This issue was fixed in the openstack/nova 16.0.0.0b2 development milestone.
In http://
we did copied the filters dict
in http://
http://
we pop the first tag in tags to do SQL query, as we didn't do deep copy, the
"tags" or "not-tags" key will be still in filters dict but the value has been removed,
as we loop through every cell, Error will happen, and also, the result is incorrect.