Periodic job * neutron-dynamic-routing-dsvm-tempest-with-ryu-master-scenario-ipv4 fails

[Slawek Kaplonski]

Periodic job neutron-dynamic-routing-dsvm-tempest-with-ryu-master-scenario-ipv4 is failing all the time since about 21.04.2018.
Example log from failed run: http://logs.openstack.org/periodic/git.openstack.org/openstack/neutron-dynamic-routing/master/neutron-dynamic-routing-dsvm-tempest-with-ryu-master-scenario-ipv4/708499c/job-output.txt.gz

Possible culprit: https://review.openstack.org/#/c/560465/ but I didn’t check it deeply so it might be only trigger for some other failure in fact.

[Brian Haley (brian-haley)]

I proposed https://review.openstack.org/564304 to test a theory on this, we'll see...

[Federico Ressi (fressi-redhat)]

I was looking at one of the failures. Please see how create_subnet is used here:

https://github.com/openstack/neutron-dynamic-routing/blob/master/neutron_dynamic_routing/tests/tempest/scenario/base.py#L192-L198

        self.create_subnet(
            {'id': ext_net_id},
            cidr=netaddr.IPNetwork(exsubnet.cidr),
            mask_bits=exsubnet.mask,
            ip_version=ip_version,
            client=self.admin_client,
            subnetpool_id=ext_subnetpool['id'])

As create_subnet is now able to look at tempest.conf and give you unused subnets CIDRs according to parameters specified in tempest.com I believe that some of the logic I saw here in this test infrastructure I suspect it is now redundant, complex, error prone and shouldn't be required any more. What I suggest is to call create_subnet without any given cidr and mask_bits and leave it giving you an available CIDR from the pool for your test class.

self.create_subnet(
     {'id': ext_net_id},
     ip_version=ip_version,
     client=self.admin_client,
     subnetpool_id=ext_subnetpool['id'])

The pool of CIDRs is configured in tempest.conf by user and it should not be required any more to hardcode any more any CIDR in your code.

Please configure these options of tempest.conf to be sure you have enough subnet CIDRs for every one of your test classes:

https://github.com/openstack/tempest/blob/master/tempest/config.py#L581-L592

[Brian Haley (brian-haley)]

Federico - I though I tried that in the linked patch, but it led to failures. Let me update it again by just modifying the first create_subnet() and see what happens.

[Slawek Kaplonski (slaweq)]

I am able to reproduce this issue locally. It happens when more than one test is run at once and I think that there is some issue with subnet cidrs and subnetpools configuration.

[Slawek Kaplonski (slaweq)]

So basically problem here is that in neutron-dynamic-routing tempest tests, there is first created subnetpool and then subnet from this pool. And cidrs are always hardcoded there so it can't allocate same cidr for two tests.
Allowing to allocate cidr automatically is also not a solution here as cidr needs to be from specific subnetpool.
I will investigate how to fix that in neutron-dynamic-routing tests without changing too much logic there

[Federico Ressi (fressi-redhat)]

How is the state of this bug? Have you discovered something about it?

[Slawek Kaplonski (slaweq)]

Hi Federico: I didn't have time to fix it after my last comment. I hope I will be able to work on it today

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/567552

[Federico Ressi (fressi-redhat)]

Hi, I am creating a patch to create_subnet to allow skipping subnet CIDRs for cases like this when it is OK reusing the same CIDR for two subnets and allow manual CIDRs management.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/567742

[OpenStack Infra (hudson-openstack)]

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: master
Review: https://review.openstack.org/567552
Reason: Federico's approach is better: https://review.openstack.org/#/c/567742/ :) Thx

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/567742
Committed: https://git.openstack.org/cgit/openstack/neutron-dynamic-routing/commit/?id=26bfa08b23060800a5daf3ab9aa3b8f3019f7f0c
Submitter: Zuul
Branch: master

commit 26bfa08b23060800a5daf3ab9aa3b8f3019f7f0c
Author: Federico Ressi <email address hidden>
Date: Fri May 11 06:47:34 2018 +0200

    Disable subnet CIDR reservation

    Reserving subnet CIDR make impossible to reuse the same
    CIDR for a subnet twice for the test case class.

    Change-Id: Ia32f0e3fea509d1eb0074f5d4bdaf5b02b6dcae0
    Depends-On: I421dd28999086ee0af9246121f4f033291e341b4
    Closes-Bug: #1766702

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron-dynamic-routing 13.0.0.0b2 development milestone.