Quota_server_group_members parameter is not respected with min_count/max_count

[Dan P]

When I create instances which I want to be in the same server group (--hint group=<UUID>) with parameter: --num-instances or --min-count/--max-count and I have quota_server_group_members set to 5, nova is spawning all instances regardless quota.

 Example:
nova boot --image $image --flavor small --hint group=528ff1e3-5a6d-475c-826e-c70d7a56e769 --nic net-id=<UUID> --min-count=2 --max-count=10 test-instance

All Instances has spawned correct. In next request I have an error:
ERROR (Forbidden): Quota exceeded, too many servers in group (HTTP 403) (Request-ID: req-f8aa39cd-2a14-486d-95a9-747ed9eecbfc)

Normally request which should spawn 10 instances (but quota is set to 5) should give an error with Quota exceeded.
The same situation is with --num-instances parameter.

I check it many times in Juno release and on devstack using nova master branch.

I added simple patch to forbidden creating new instances over quota, but it's should be changed and it's not working correctly with --min-count/--max-count parameter.

[Augustina Ragwitz (auggy)]

Juno is EOL [1] so we aren't doing fixes for it anymore. I've closed this as invalid, but if you still have this issue in a later, supported release, please feel free to reopen this issue and provide the updated version details of your environment in the description.

References:
--------------
[1] http://docs.openstack.org/releases/

[Dan P (dduuch-v)]

As I said, I check it on devstack with Mitaka realease (BRANCH=master parameter in localrc) and this problem still exist.

[Augustina Ragwitz (auggy)]

What is your output for nova quota-show for server_groups and server_group_members?

[Augustina Ragwitz (auggy)]

I attempted to reproduce this using devstack. I set my devstack local.conf set quota_server_group_members=5, which I verified in my running devstack by "nova quota-show". Below are the entries for server_groups and server_group_members:

| server_groups | 10 |
| server_group_members | 5 |

Then I created a new server_group:
nova server-group-create group1 affinity
+--------------------------------------+--------+---------------+---------+----------+
| Id | Name | Policies | Members | Metadata |
+--------------------------------------+--------+---------------+---------+----------+
| 79baa34b-da96-4533-8620-8a4843799f9f | group1 | [u'affinity'] | [] | {} |
+--------------------------------------+--------+---------------+---------+----------+

I then launched 7 instances successfully:

nova boot --image a87f8ba3-efa4-43a8-a1f9-6e6f82fa99e9 --flavor m1.nano --hint group=79baa34b-da96-4533-8620-8a4843799f9f --min-count=7 test3

nova list
+--------------------------------------+---------+--------+------------+-------------+-------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+---------+--------+------------+-------------+-------------------+
| 033c9d03-78a5-44ad-8ff0-7a32be0b557f | test3-1 | ACTIVE | - | Running | private=10.0.0.10 |
| a0e6f0ce-6ca9-4294-86f6-e3253fd5581a | test3-2 | ACTIVE | - | Running | private=10.0.0.11 |
| 824892cf-b98c-4f46-9bd3-bc90190e9460 | test3-3 | ACTIVE | - | Running | private=10.0.0.13 |
| a37bc738-719f-494a-bae2-776acf665a01 | test3-4 | ACTIVE | - | Running | private=10.0.0.9 |
| d3d96caf-3078-4d1e-9e6d-de789c6d3203 | test3-5 | ACTIVE | - | Running | private=10.0.0.15 |
| bbec959d-9d04-4fdd-bc1f-882261f6dc64 | test3-6 | ACTIVE | - | Running | private=10.0.0.12 |
| 75affa96-530f-46c3-9cd5-57864b336d6f | test3-7 | ACTIVE | - | Running | private=10.0.0.14 |
+--------------------------------------+---------+--------+------------+-------------+-------------------+

I tried launching an 8th instance and got an error:
nova boot --image a87f8ba3-efa4-43a8-a1f9-6e6f82fa99e9 --flavor m1.nano --hint group=79baa34b-da96-4533-8620-8a4843799f9f test4
ERROR (Forbidden): Quota exceeded, too many servers in group (HTTP 403) (Request-ID: req-03d87586-37c1-4807-92c1-64aa83b004f7)

I double checked test coverage and from what I can tell we're only testing creating individual instances after the quota has been met. There doesn't appear to be multiple create test coverage. We should probably make sure our test coverage reproduces this problem and ensure we have test coverage for this scenario.

[yuntongjin (yuntongjin)]

the root cause is:
in instance_group object, a member is add to DB but not add in instance_group object
members = db.instance_group_members_add(context, group_uuid,
                instance_uuids)
https://github.com/openstack/nova/blob/master/nova/objects/instance_group.py#L196

so in compute.api, the resource count won't increased.
                        count = objects.Quotas.count(context,
                                             'server_group_members',
                                             instance_group,
                                             context.user_id)
https://github.com/openstack/nova/blob/master/nova/compute/api.py#L935

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/286443

[daniel.pawlik (daniel-pawlik)]

Hello,
I cherry-pick your patch and this fix doesn't solve the whole problem.
Example:

quota_server_group_members = 5

stack@watcher-launchpad:~$ nova boot --flavor c1 --image cirros-0.3.4-x86_64-uec --nic net-id=cb100c59-8a61-4740-aaca-5351d9ebff63 --hint group=ea4b6aa4-784b-4ac6-90f9-efe44728ed54 --min-count=2 --max-count=10 test1
ERROR (Forbidden): Quota exceeded, too many servers in group (HTTP 403) (Request-ID: req-3b720127-308d-4c95-a825-a79157f5eb86)
stack@watcher-launchpad:~$ nova list
+----+------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+----+------+--------+------------+-------------+----------+
+----+------+--------+------------+-------------+----------+

It should spawn 5 instances (there was no instance in group when I make request) - min. 2 instances, but max should be equal to quota_server_group_members.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Michael Still (<email address hidden>) on branch: master
Review: https://review.openstack.org/286443
Reason: This code hasn't been updated in a long time, and is in merge conflict. I am going to abandon this review, but feel free to restore it if you're still working on this.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/370567

[Slawek Kaplonski (slaweq)]

There is still one more problem with quota_server_group_members IMHO:
1. user spawns instances in server_group and provide --min-count and --max-count parameters
2. both, min-count and max-count are below quota for instances (e.g min-count=2, max-count=5)
3. max-count is above quota_server_group_members (e.g. it was set to 3 for tenant)
In such case nova will not spawn any instance and returns error "Quota exceeded" but IMO it should spawn at least 2 instances (min-count) or 3 (quota_server_group_memebers)

[melanie witt (melwitt)]

Being that this has been broken since Juno apparently, I think this is more appropriately Medium importance.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/370567
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=43826e458eefc4157b45d8a04422cbdcdec4f7ff
Submitter: Jenkins
Branch: master

commit 43826e458eefc4157b45d8a04422cbdcdec4f7ff
Author: yuntongjin <email address hidden>
Date: Tue Mar 1 16:00:16 2016 +0800

    Add members in InstanceGroup object members field

    In InstanceGroup object, add_members method doesn't add members
    into object members field, which causes
    objects.Quotas.count(server_group_members) won't increase, and
    instance group quota check invalid. This makes an issue that user
    could spawn with one call more instances in server group than
    was allowed in quota_server_group_members is it was below
    instances quota settting.

    Co-authored-by: yuntongjin <email address hidden>

    Change-Id: Icdd8aef688776f00fef6ede6e1bed01af29f9917
    Closes-bug: #1538169

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 15.0.0.0b1 development milestone.