IpAddressGenerationFailure during rally test execution

[Ann Taraday]

Environment 3 controllers, 46 computes, liberty. L3 HA During execution NeutronNetworks.create_and_delete_routers in logs there is hundreds of traces in dhcp-agent logs http://paste.openstack.org/show/491423/, http://paste.openstack.org/show/491408/. Tests passes but having these traces in logs is ugly.

[LIU Yulong (dragon889)]

Kevin's patch is merged, this bug should be invalid.

[Ann Taraday (akamyshnikova)]

@LIU Yulong
Kevin's patch was applied for env.

[LIU Yulong (dragon889)]

oh, I see, this trace
http://paste.openstack.org/show/491423/
is not related to https://bugs.launchpad.net/neutron/+bug/1533440 and l3-ha.
I encountered this exception in rally test every time.
This is a Rally bug I think, because you have no way to set the test network's subnet range.
This line cause the bug:
https://github.com/openstack/rally/blob/master/rally/plugins/openstack/scenarios/neutron/utils.py#L141

[Piyush (pirsriva)]

I am not sure whether it is a Rally bug. I doubt the JSON has been configured correctly based on requirements.

@Ann- Could you kindly put up the JSON you used for running?

@LIU Yulong- Rally DOES give an option of specifying the starting CIDR as well as the exact subnet CIDR for a subnets created.
You can either specify start_cidr for subnets creation (https://github.com/openstack/rally/blob/master/samples/tasks/scenarios/neutron/create_and_delete_routers.json#L7) or mention the exact CIDR for subnet in subnet_create_args (https://github.com/openstack/rally/blob/master/samples/tasks/scenarios/neutron/create_and_delete_routers.json#L6)

[Ann Taraday (akamyshnikova)]

@Piyush Raman Srivastava

Thanks for response! The input task was configured like this http://akamyshnikova.github.io/neutron-benchmark-results/rally/create_delete_92_40.html#/NeutronNetworks.create_and_delete_routers/task

[Piyush (pirsriva)]

Hi,

I am not able to reproduce the issue on my setup as of now, I ran your JSON on my setup and it worked ( http://paste.openstack.org/show/492655/). I further checked DHCP logs and did NOT get any such error.

We use /30 subnet so IP range for lets say subnet with CID 1.1.1.0/30 is 1.1.1.1 -> 1.1.1.1.2 (all 0's and all 1'h host bit is reserved ). Now DHCP needs 1 IP (1.1.0.2) and router will take IP of gateway (1.1.0.1). So i dont understand how can we get an error saying no more IP since each subnet needs 2 ports only ( DHCP + Subnet Gateway )

Also as you mentioned that Rally task was a success which means API was a success and hence port should have been CREATED without any issue by Neutron.

All the generated CIDR also look good and comply /30 subnet ( http://paste.openstack.org/show/492656/ )

Let me think more on how to reproduce the issue.

[LIU Yulong (dragon889)]

@ Piyush Raman Srivastava (pirsriva)
Hi, my JSON input is:

{
    "NeutronNetworks.create_and_delete_routers": [
        {
            "args": {
                "network_create_args": {},
                "subnet_create_args": {},
                "subnet_cidr_start": "1.1.0.0/30",
                "subnets_per_network": 2,
                "router_create_args": {}
            },
            "runner": {
                "type": "constant",
                "times": 200,
                "concurrency": 40
            },
            "context": {
                "network": {},
                "users": {
                    "tenants": 40,
                    "users_per_tenant": 10
                },
                "quotas": {
                    "neutron": {
                        "network": -1,
                        "subnet": -1,
                        "router": -1
                    }
                }
            }
        }
    ]
}

You will find there has some subnets in the attachment image like:
"c_rally_8b1beff2_QZK6pEy1 10.2.11.0/24" even I set the "subnet_cidr_start": "1.1.0.0/30".
And the trace http://paste.openstack.org/show/491423/ was always raised by this kind of subnets.

[Andriy Kurilin (andreykurilin)]

@LIU Yulong (dragon889) :
sorry for late reply. Could you verify this issue on rally/master ?

[LIU Yulong (dragon889)]

@Andrey Kurilin (andreykurilin)
The current rally task input is a little confused to me.
My task input JSON is:
{
    "NeutronNetworks.create_and_delete_routers": [
        {
            "args": {
                "network_create_args": {},
                "subnet_create_args": {},
                "subnet_cidr_start": "1.1.0.0/30",
                "subnets_per_network": 2,
                "router_create_args": {}
            },
            "runner": {
                "type": "constant",
                "times": 30,
                "concurrency": 10
            },
            "context": {
                "network": {
                    "network_create_args": {},
                    "subnets_per_network": 1,
                    "start_cidr": "10.0.0.0/16",
                    "networks_per_tenant": 1
                },
                "users": {
                    "tenants": 3,
                    "users_per_tenant": 3
                },
                "quotas": {
                    "neutron": {
                        "network": -1,
                        "subnet": -1,
                        "router": -1
                    }
                }
            }
        }
    ]
}

And during the rally execution, you will find some subnets like "10.x.0.0/16" in the attachment image, and mostly the subnet CIDR is "1.1.0.xx/30".
And if I don't set the "context: networks: start_cidr" the result is same as my previous comment: https://bugs.launchpad.net/neutron/+bug/1562887/comments/7
You will get some subnet with CIDR "10.2.xx.0/24"

So I have a question, what's the difference between "args: subnet_cidr_start" and "context: networks: start_cidr" ?
So, during the rally create_and_delete_routers task which subnet will be used to create router interface and which subnet will start a DHCP agent?

[Ann Taraday (akamyshnikova)]

I still see this errors in logs