tempest failure: No more IP addresses available on network

[Akihiro Motoki]

From yesterday, this error suddenly starts to occur.

IpAddressGenerationFailureClient: No more IP addresses available on network 438c3277-c139-4cdd-b675-3bd52070da89.

logstash query: "IpAddressGenerationFailureClient: No more IP addresses available on network "

http://logstash.openstack.org/index.html#eyJzZWFyY2giOiJcIklwQWRkcmVzc0dlbmVyYXRpb25GYWlsdXJlQ2xpZW50OiBObyBtb3JlIElQIGFkZHJlc3NlcyBhdmFpbGFibGUgb24gbmV0d29yayBcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTM4MjUzNzk0OTgxMH0=

[dkehn (dekehn)]

I'm seeing it as well, usually a recheck no bug would get me through it, but 2 failures in a row.

https://review.openstack.org/#/c/45927/

http://logs.openstack.org/27/45927/12/check/check-tempest-devstack-vm-neutron-isolated/3797285/console.html

[Justin Hammond (justin-hammond)]

had a failure with my change: https://review.openstack.org/#/c/44942/

[Nachi Ueno (nati-ueno)]

I'm reading logs.
I
https://etherpad.openstack.org/p/debug1243726

[Aaron Rosen (arosen)]

It seems to be failing on allocating an IPaddress out of the floatingip pool. By default the floatingip pool is only has 16 ip addresses free. I wonder since we are running the tests in parallel if we are sometimes actually running out of ip_addresses here. Just to confirm I've got a test here running https://review.openstack.org/#/c/53474/ which bumps it from a /28 to /24

[Nachi Ueno (nati-ueno)]

> Aaron
Nice findings!

I think this one is also one of issue too
https://review.openstack.org/#/c/53478/

[Matthew Treinish (treinish)]

Just a clarification tempest does not run in parallel with neutron enabled. (the other tempest runs with nova-network are running in parallel) We can't turn parallel on with neutron until tenant isolation is enabled, which is what I was actually working on when this bug appeared, see https://review.openstack.org/#/c/53459/ which is the revert of the revert. (when the original was merged it seems like it triggered this bug) So, this bug is appearing when running things serially.

[Aaron Rosen (arosen)]

Thanks for the clarification matt

[Armando Migliaccio (armando-migliaccio)]

Hopefully this one might solve this as well:

https://review.openstack.org/#/c/54850/

[Armando Migliaccio (armando-migliaccio)]

Hopefully these:

https://review.openstack.org/#/c/57775/
https://review.openstack.org/#/c/57812/

Will address the obvious ones and leave out the real ones.

[Armando Migliaccio (armando-migliaccio)]

Relevant bug: https://bugs.launchpad.net/neutron/+bug/1253344

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/59664

[Armando Migliaccio (armando-migliaccio)]

Related bug: https://bugs.launchpad.net/neutron/+bug/1257235

[Armando Migliaccio (armando-migliaccio)]

Related bug: https://bugs.launchpad.net/neutron/+bug/1257237

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/57812
Committed: http://github.com/openstack/neutron/commit/bff120a4775a1b1f3846a27c38d8eff4a678fd24
Submitter: Jenkins
Branch: master

commit bff120a4775a1b1f3846a27c38d8eff4a678fd24
Author: armando-migliaccio <email address hidden>
Date: Thu Nov 21 18:42:08 2013 -0800

    Handle exceptions on create_dhcp_port

    If a network/subnet is deleted while creating the dhcp
    port, the agent will detect a conflict on state of the
    network and deal with it accordingly.

    A concurrent delete may manifest itself via a number
    of exceptions, IPAddressGenerationFailure amongst others,
    hence the refactoring of the error handling logic into its
    own utility method.

    Partial-bug: #1253344
    Related-bug: #1243726

    Change-Id: I442beb5f82f3db8786eea53926903ef0ba0efbf1

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/59664
Committed: http://github.com/openstack/neutron/commit/b48ddb648b4fe1b428328a91d0a045b837119f29
Submitter: Jenkins
Branch: master

commit b48ddb648b4fe1b428328a91d0a045b837119f29
Author: armando-migliaccio <email address hidden>
Date: Tue Nov 26 14:45:24 2013 -0800

    Handle failures on update_dhcp_port

    Ensure exceptions due to conflicting
    state of network or subnet resources
    are dealt with by the dhcp agent.

    Closes-bug: #1253344
    Related-bug: #1243726

    Change-Id: I4fd51442c034fabc91d5a3f065f4df98f5fad35b

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/havana
Review: https://review.openstack.org/60571

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/havana
Review: https://review.openstack.org/60572

[Armando Migliaccio (armando-migliaccio)]

I think it's safe to say that this can be closed. If I run the following query:

"IpAddressGenerationFailureClient: No more IP addresses available on network " AND NOT build_queue:"experimental" AND NOT build_change:"59787" (https://review.openstack.org/#/c/59787/ - Bump api_workers from 0 to 4 -- which is a WIP and has the same nature of increasing the parallelism as the experimental queue job)

No occurrences come up in the last 7 days.