Concurrent create_policy_target_group call fails

Bug #1462024 reported by AJAY KALAMBUR
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Group Based Policy
Fix Released
High
Robert Kukura

Bug Description

I see an issue when running Rally tests concurrently which creates policy target groups in parallel. I see the following error returned by the create policy target group API

5-83c6-4996-afa8-d602b48900f1 | ITER: 19 END: Error BadRequest: IP Pool 10.0.0.0/8 overlaps with one of the existing L3P for the same tenant [u'10.0.0.0/8'].

k/scenarios/gbp/gbp.py\", line 23, in test_gbp\n self._create_policy_target_group(pt_group_name)\n File \"/home/akalambu/rally/local/lib/python2.7/site-packages/rally/benchmark/scenarios/base.py\", line 262, in func_atomic_actions\n f = func(self, *args, **kwargs)\n File \"/home/akalambu/rally/local/lib/python2.7/site-packages/rally/plugins/openstack/scenarios/gbp/utils.py\", line 157, in _create_policy_target_group\n self.clients(\"gbp\").create_policy_target_group(body)\n File \"/home/akalambu/rally/local/lib/python2.7/site-packages/gbpclient/v2_0/client.py\", line 97, in with_params\n ret = self.function(instance, *args, **kwargs)\n File \"/home/akalambu/rally/local/lib/python2.7/site-packages/gbpclient/v2_0/client.py\", line 266, in create_policy_target_group\n return self.post(self.policy_target_groups_path, body=body)\n File \"/home/akalambu/rally/local/lib/python2.7/site-packages/gbpclient/v2_0/client.py\", line 794, in post\n headers=headers, params=params)\n File \"/home/akalambu/rally/local/lib/python2.7/site-packages/gbpclient/v2_0/client.py\", line 720, in do_request\n self._handle_fault_response(status_code, replybody)\n File \"/home/akalambu/rally/local/lib/python2.7/site-packages/gbpclient/v2_0/client.py\", line 691, in _handle_fault_response\n exception_handler_v20(status_code, des_error_body)\n File \"/home/akalambu/rally/local/lib/python2.7/site-packages/gbpclient/v2_0/client.py\", line 64, in exception_handler_v20\n status_code=status_code)\nBadRequest: IP Pool 10.0.0.0/8 overlaps with one of the existing L3P for the same tenant [u'10.0.0.0/8'].\n"

Note from Sumeet for reference
Hi Ajay, Overlapping IP pools are not currently supported for the same
tenant with the default Neutron rendering. So when you create multiple
PTGs in parallel without specifying the L2P/L3P, the implicit workflow
tries to create a L3P with a default IP Pool (10.0.0.0/8 in this
case). The second PTG is trying to be created while the post-commit
for the first one has not finished yet. Hence you are seeing this
issue. I would appreciate if you can file a bug for this, since we
should explore what is a better workflow in this case.

Changed in group-based-policy:
status: New → Confirmed
importance: Undecided → High
assignee: nobody → Robert Kukura (rkukura)
milestone: none → kilo-gbp-4
Changed in group-based-policy:
milestone: kilo-gbp-4 → next
Revision history for this message
Robert Kukura (rkukura) wrote :

It looks to me like this can be resolved by catching this exception and then doing another query to see if a concurrent request has created a default L3P for the tenant, and if so, using it.

This will also resolve the REVISIT regarding concurrency, as long as overlapping L3Ps are disallowed. But once that restriction is removed, this exception will no longer be raised and we'll need some other mechanism, such as the DB table mentioned in the REVISIT, to prevent multiple default L3Ps from being created for a tenant.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to group-based-policy (master)

Fix proposed to branch: master
Review: https://review.openstack.org/224293

Changed in group-based-policy:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to group-based-policy (master)

Reviewed: https://review.openstack.org/224293
Committed: https://git.openstack.org/cgit/stackforge/group-based-policy/commit/?id=7acac86b65f360b1b0dc230cc944c96f1366d9dd
Submitter: Jenkins
Branch: master

commit 7acac86b65f360b1b0dc230cc944c96f1366d9dd
Author: Robert Kukura <email address hidden>
Date: Wed Sep 16 14:50:26 2015 -0400

    Handle concurrent implicit L3P creation

    In the implicit_policy driver, when creating a default L3 policy,
    raise DefaultL3PolicyAlreadyExists if an L3 policy named 'default'
    already exists.

    If DefaultL3PolicyAlreadyExists is raised when the implicit_policy
    driver attempts to create the default L3 policy for a tenant, query
    again to see if a default L3 policy has been concurrently created, and
    if so, use that. This requires adding local_api wrappers for
    postcommit group policy resource CRUD operations called in the
    implicit_policy driver, so that clean DB sessions are used.

    Also, fix the resource_mapping driver's
    delete_policy_target_group_postcommit to gracefully handle partially
    constructed states, such as a null L2 policy or a subnet not attached
    to a router.

    Closes-bug: 1462024
    Partial-bug: 1417312

    Change-Id: I09f29eef22edb45290070aae30e97c93c77ea341

Changed in group-based-policy:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to group-based-policy (stable/kilo)

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/228607

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to group-based-policy (stable/kilo)

Reviewed: https://review.openstack.org/228607
Committed: https://git.openstack.org/cgit/stackforge/group-based-policy/commit/?id=4fcd843eff4f19d1d8c77ba30c1d9f0bac06b9c8
Submitter: Jenkins
Branch: stable/kilo

commit 4fcd843eff4f19d1d8c77ba30c1d9f0bac06b9c8
Author: Robert Kukura <email address hidden>
Date: Wed Sep 16 14:50:26 2015 -0400

    Handle concurrent implicit L3P creation

    In the implicit_policy driver, when creating a default L3 policy,
    raise DefaultL3PolicyAlreadyExists if an L3 policy named 'default'
    already exists.

    If DefaultL3PolicyAlreadyExists is raised when the implicit_policy
    driver attempts to create the default L3 policy for a tenant, query
    again to see if a default L3 policy has been concurrently created, and
    if so, use that. This requires adding local_api wrappers for
    postcommit group policy resource CRUD operations called in the
    implicit_policy driver, so that clean DB sessions are used.

    Also, fix the resource_mapping driver's
    delete_policy_target_group_postcommit to gracefully handle partially
    constructed states, such as a null L2 policy or a subnet not attached
    to a router.

    Closes-bug: 1462024
    Partial-bug: 1417312

    Conflicts:
     gbpservice/network/neutronv2/local_api.py
     gbpservice/neutron/services/grouppolicy/drivers/resource_mapping.py

    Change-Id: I09f29eef22edb45290070aae30e97c93c77ea341
    (cherry picked from commit 7acac86b65f360b1b0dc230cc944c96f1366d9dd)

tags: added: in-stable-kilo
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to group-based-policy (stable/juno)

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/228684

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to group-based-policy (stable/juno)

Reviewed: https://review.openstack.org/228684
Committed: https://git.openstack.org/cgit/stackforge/group-based-policy/commit/?id=9f3bef975ed2f0f56edd99f36121bbd9f1c58c72
Submitter: Jenkins
Branch: stable/juno

commit 9f3bef975ed2f0f56edd99f36121bbd9f1c58c72
Author: Robert Kukura <email address hidden>
Date: Wed Sep 16 14:50:26 2015 -0400

    Handle concurrent implicit L3P creation

    In the implicit_policy driver, when creating a default L3 policy,
    raise DefaultL3PolicyAlreadyExists if an L3 policy named 'default'
    already exists.

    If DefaultL3PolicyAlreadyExists is raised when the implicit_policy
    driver attempts to create the default L3 policy for a tenant, query
    again to see if a default L3 policy has been concurrently created, and
    if so, use that. This requires adding local_api wrappers for
    postcommit group policy resource CRUD operations called in the
    implicit_policy driver, so that clean DB sessions are used.

    Also, fix the resource_mapping driver's
    delete_policy_target_group_postcommit to gracefully handle partially
    constructed states, such as a null L2 policy or a subnet not attached
    to a router.

    Closes-bug: 1462024
    Partial-bug: 1417312

    Conflicts:
     gbpservice/network/neutronv2/local_api.py
     gbpservice/neutron/services/grouppolicy/drivers/implicit_policy.py

    Change-Id: I09f29eef22edb45290070aae30e97c93c77ea341
    (cherry picked from commit 7acac86b65f360b1b0dc230cc944c96f1366d9dd)
    (cherry picked from commit 4fcd843eff4f19d1d8c77ba30c1d9f0bac06b9c8)

tags: added: in-stable-juno
Changed in group-based-policy:
milestone: next → liberty-1
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.