Quota_server_group_members parameter is not respected with min_count/max_count

Bug #1538169 reported by Dan P on 2016-01-26
22
This bug affects 4 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Medium
Slawek Kaplonski

Bug Description

When I create instances which I want to be in the same server group (--hint group=<UUID>) with parameter: --num-instances or --min-count/--max-count and I have quota_server_group_members set to 5, nova is spawning all instances regardless quota.

 Example:
nova boot --image $image --flavor small --hint group=528ff1e3-5a6d-475c-826e-c70d7a56e769 --nic net-id=<UUID> --min-count=2 --max-count=10 test-instance

All Instances has spawned correct. In next request I have an error:
ERROR (Forbidden): Quota exceeded, too many servers in group (HTTP 403) (Request-ID: req-f8aa39cd-2a14-486d-95a9-747ed9eecbfc)

Normally request which should spawn 10 instances (but quota is set to 5) should give an error with Quota exceeded.
The same situation is with --num-instances parameter.

I check it many times in Juno release and on devstack using nova master branch.

I added simple patch to forbidden creating new instances over quota, but it's should be changed and it's not working correctly with --min-count/--max-count parameter.

Dan P (dduuch-v) on 2016-01-26
summary: - Quota_server_group parameter is not respected by nova
+ Quota_server_group_members parameter is not respected by nova

Juno is EOL [1] so we aren't doing fixes for it anymore. I've closed this as invalid, but if you still have this issue in a later, supported release, please feel free to reopen this issue and provide the updated version details of your environment in the description.

References:
--------------
[1] http://docs.openstack.org/releases/

Changed in nova:
status: New → Invalid
Dan P (dduuch-v) wrote :

As I said, I check it on devstack with Mitaka realease (BRANCH=master parameter in localrc) and this problem still exist.

Dan P (dduuch-v) on 2016-01-27
Changed in nova:
status: Invalid → New
Augustina Ragwitz (auggy) wrote :

What is your output for nova quota-show for server_groups and server_group_members?

Changed in nova:
status: New → Incomplete
Augustina Ragwitz (auggy) wrote :

I attempted to reproduce this using devstack. I set my devstack local.conf set quota_server_group_members=5, which I verified in my running devstack by "nova quota-show". Below are the entries for server_groups and server_group_members:

| server_groups | 10 |
| server_group_members | 5 |

Then I created a new server_group:
nova server-group-create group1 affinity
+--------------------------------------+--------+---------------+---------+----------+
| Id | Name | Policies | Members | Metadata |
+--------------------------------------+--------+---------------+---------+----------+
| 79baa34b-da96-4533-8620-8a4843799f9f | group1 | [u'affinity'] | [] | {} |
+--------------------------------------+--------+---------------+---------+----------+

I then launched 7 instances successfully:

nova boot --image a87f8ba3-efa4-43a8-a1f9-6e6f82fa99e9 --flavor m1.nano --hint group=79baa34b-da96-4533-8620-8a4843799f9f --min-count=7 test3

nova list
+--------------------------------------+---------+--------+------------+-------------+-------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+---------+--------+------------+-------------+-------------------+
| 033c9d03-78a5-44ad-8ff0-7a32be0b557f | test3-1 | ACTIVE | - | Running | private=10.0.0.10 |
| a0e6f0ce-6ca9-4294-86f6-e3253fd5581a | test3-2 | ACTIVE | - | Running | private=10.0.0.11 |
| 824892cf-b98c-4f46-9bd3-bc90190e9460 | test3-3 | ACTIVE | - | Running | private=10.0.0.13 |
| a37bc738-719f-494a-bae2-776acf665a01 | test3-4 | ACTIVE | - | Running | private=10.0.0.9 |
| d3d96caf-3078-4d1e-9e6d-de789c6d3203 | test3-5 | ACTIVE | - | Running | private=10.0.0.15 |
| bbec959d-9d04-4fdd-bc1f-882261f6dc64 | test3-6 | ACTIVE | - | Running | private=10.0.0.12 |
| 75affa96-530f-46c3-9cd5-57864b336d6f | test3-7 | ACTIVE | - | Running | private=10.0.0.14 |
+--------------------------------------+---------+--------+------------+-------------+-------------------+

I tried launching an 8th instance and got an error:
nova boot --image a87f8ba3-efa4-43a8-a1f9-6e6f82fa99e9 --flavor m1.nano --hint group=79baa34b-da96-4533-8620-8a4843799f9f test4
ERROR (Forbidden): Quota exceeded, too many servers in group (HTTP 403) (Request-ID: req-03d87586-37c1-4807-92c1-64aa83b004f7)

I double checked test coverage and from what I can tell we're only testing creating individual instances after the quota has been met. There doesn't appear to be multiple create test coverage. We should probably make sure our test coverage reproduces this problem and ensure we have test coverage for this scenario.

Changed in nova:
status: Incomplete → Confirmed
tags: added: api compute
yuntongjin (yuntongjin) on 2016-02-24
Changed in nova:
assignee: nobody → yuntongjin (yuntongjin)
yuntongjin (yuntongjin) wrote :

the root cause is:
in instance_group object, a member is add to DB but not add in instance_group object
members = db.instance_group_members_add(context, group_uuid,
                instance_uuids)
https://github.com/openstack/nova/blob/master/nova/objects/instance_group.py#L196

so in compute.api, the resource count won't increased.
                        count = objects.Quotas.count(context,
                                             'server_group_members',
                                             instance_group,
                                             context.user_id)
https://github.com/openstack/nova/blob/master/nova/compute/api.py#L935

Fix proposed to branch: master
Review: https://review.openstack.org/286443

Changed in nova:
status: Confirmed → In Progress
yuntongjin (yuntongjin) on 2016-03-08
Changed in nova:
importance: Undecided → High

Hello,
I cherry-pick your patch and this fix doesn't solve the whole problem.
Example:

quota_server_group_members = 5

stack@watcher-launchpad:~$ nova boot --flavor c1 --image cirros-0.3.4-x86_64-uec --nic net-id=cb100c59-8a61-4740-aaca-5351d9ebff63 --hint group=ea4b6aa4-784b-4ac6-90f9-efe44728ed54 --min-count=2 --max-count=10 test1
ERROR (Forbidden): Quota exceeded, too many servers in group (HTTP 403) (Request-ID: req-3b720127-308d-4c95-a825-a79157f5eb86)
stack@watcher-launchpad:~$ nova list
+----+------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+----+------+--------+------------+-------------+----------+
+----+------+--------+------------+-------------+----------+

It should spawn 5 instances (there was no instance in group when I make request) - min. 2 instances, but max should be equal to quota_server_group_members.

Change abandoned by Michael Still (<email address hidden>) on branch: master
Review: https://review.openstack.org/286443
Reason: This code hasn't been updated in a long time, and is in merge conflict. I am going to abandon this review, but feel free to restore it if you're still working on this.

Changed in nova:
assignee: yuntongjin (yuntongjin) → Slawek Kaplonski (slaweq)

There is still one more problem with quota_server_group_members IMHO:
1. user spawns instances in server_group and provide --min-count and --max-count parameters
2. both, min-count and max-count are below quota for instances (e.g min-count=2, max-count=5)
3. max-count is above quota_server_group_members (e.g. it was set to 3 for tenant)
In such case nova will not spawn any instance and returns error "Quota exceeded" but IMO it should spawn at least 2 instances (min-count) or 3 (quota_server_group_memebers)

melanie witt (melwitt) wrote :

Being that this has been broken since Juno apparently, I think this is more appropriately Medium importance.

Changed in nova:
importance: High → Medium
melanie witt (melwitt) on 2016-09-16
summary: - Quota_server_group_members parameter is not respected by nova
+ Quota_server_group_members parameter is not respected with
+ min_count/max_count

Reviewed: https://review.openstack.org/370567
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=43826e458eefc4157b45d8a04422cbdcdec4f7ff
Submitter: Jenkins
Branch: master

commit 43826e458eefc4157b45d8a04422cbdcdec4f7ff
Author: yuntongjin <email address hidden>
Date: Tue Mar 1 16:00:16 2016 +0800

    Add members in InstanceGroup object members field

    In InstanceGroup object, add_members method doesn't add members
    into object members field, which causes
    objects.Quotas.count(server_group_members) won't increase, and
    instance group quota check invalid. This makes an issue that user
    could spawn with one call more instances in server group than
    was allowed in quota_server_group_members is it was below
    instances quota settting.

    Co-authored-by: yuntongjin <email address hidden>

    Change-Id: Icdd8aef688776f00fef6ede6e1bed01af29f9917
    Closes-bug: #1538169

Changed in nova:
status: In Progress → Fix Released

This issue was fixed in the openstack/nova 15.0.0.0b1 development milestone.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers