Periodic job * neutron-dynamic-routing-dsvm-tempest-with-ryu-master-scenario-ipv4 fails

Bug #1766702 reported by Slawek Kaplonski
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
High
Federico Ressi

Bug Description

Periodic job neutron-dynamic-routing-dsvm-tempest-with-ryu-master-scenario-ipv4 is failing all the time since about 21.04.2018.
Example log from failed run: http://logs.openstack.org/periodic/git.openstack.org/openstack/neutron-dynamic-routing/master/neutron-dynamic-routing-dsvm-tempest-with-ryu-master-scenario-ipv4/708499c/job-output.txt.gz

Possible culprit: https://review.openstack.org/#/c/560465/ but I didn’t check it deeply so it might be only trigger for some other failure in fact.

Revision history for this message
Brian Haley (brian-haley) wrote :

I proposed https://review.openstack.org/564304 to test a theory on this, we'll see...

Revision history for this message
Federico Ressi (fressi-redhat) wrote :

I was looking at one of the failures. Please see how create_subnet is used here:

https://github.com/openstack/neutron-dynamic-routing/blob/master/neutron_dynamic_routing/tests/tempest/scenario/base.py#L192-L198

        self.create_subnet(
            {'id': ext_net_id},
            cidr=netaddr.IPNetwork(exsubnet.cidr),
            mask_bits=exsubnet.mask,
            ip_version=ip_version,
            client=self.admin_client,
            subnetpool_id=ext_subnetpool['id'])

As create_subnet is now able to look at tempest.conf and give you unused subnets CIDRs according to parameters specified in tempest.com I believe that some of the logic I saw here in this test infrastructure I suspect it is now redundant, complex, error prone and shouldn't be required any more. What I suggest is to call create_subnet without any given cidr and mask_bits and leave it giving you an available CIDR from the pool for your test class.

self.create_subnet(
     {'id': ext_net_id},
     ip_version=ip_version,
     client=self.admin_client,
     subnetpool_id=ext_subnetpool['id'])

The pool of CIDRs is configured in tempest.conf by user and it should not be required any more to hardcode any more any CIDR in your code.

Please configure these options of tempest.conf to be sure you have enough subnet CIDRs for every one of your test classes:

https://github.com/openstack/tempest/blob/master/tempest/config.py#L581-L592

Revision history for this message
Brian Haley (brian-haley) wrote :

Federico - I though I tried that in the linked patch, but it led to failures. Let me update it again by just modifying the first create_subnet() and see what happens.

Changed in neutron:
assignee: nobody → Slawek Kaplonski (slaweq)
Revision history for this message
Slawek Kaplonski (slaweq) wrote :

I am able to reproduce this issue locally. It happens when more than one test is run at once and I think that there is some issue with subnet cidrs and subnetpools configuration.

Revision history for this message
Slawek Kaplonski (slaweq) wrote :

So basically problem here is that in neutron-dynamic-routing tempest tests, there is first created subnetpool and then subnet from this pool. And cidrs are always hardcoded there so it can't allocate same cidr for two tests.
Allowing to allocate cidr automatically is also not a solution here as cidr needs to be from specific subnetpool.
I will investigate how to fix that in neutron-dynamic-routing tests without changing too much logic there

Revision history for this message
Federico Ressi (fressi-redhat) wrote :

How is the state of this bug? Have you discovered something about it?

Revision history for this message
Slawek Kaplonski (slaweq) wrote :

Hi Federico: I didn't have time to fix it after my last comment. I hope I will be able to work on it today

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron-dynamic-routing (master)

Fix proposed to branch: master
Review: https://review.openstack.org/567552

Changed in neutron:
status: Confirmed → In Progress
Revision history for this message
Federico Ressi (fressi-redhat) wrote :

Hi, I am creating a patch to create_subnet to allow skipping subnet CIDRs for cases like this when it is OK reusing the same CIDR for two subnets and allow manual CIDRs management.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/567742

Changed in neutron:
assignee: Slawek Kaplonski (slaweq) → Federico Ressi (fressi-redhat)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron-dynamic-routing (master)

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: master
Review: https://review.openstack.org/567552
Reason: Federico's approach is better: https://review.openstack.org/#/c/567742/ :) Thx

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron-dynamic-routing (master)

Reviewed: https://review.openstack.org/567742
Committed: https://git.openstack.org/cgit/openstack/neutron-dynamic-routing/commit/?id=26bfa08b23060800a5daf3ab9aa3b8f3019f7f0c
Submitter: Zuul
Branch: master

commit 26bfa08b23060800a5daf3ab9aa3b8f3019f7f0c
Author: Federico Ressi <email address hidden>
Date: Fri May 11 06:47:34 2018 +0200

    Disable subnet CIDR reservation

    Reserving subnet CIDR make impossible to reuse the same
    CIDR for a subnet twice for the test case class.

    Change-Id: Ia32f0e3fea509d1eb0074f5d4bdaf5b02b6dcae0
    Depends-On: I421dd28999086ee0af9246121f4f033291e341b4
    Closes-Bug: #1766702

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron-dynamic-routing 13.0.0.0b2

This issue was fixed in the openstack/neutron-dynamic-routing 13.0.0.0b2 development milestone.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers