[nailgun] Not enough free IP addresses in pool

[Denis Ipatov]

Everything worked perfectly for a 2 node system, but when I tried to create another environment with 4 nodes (1 controller, 3 compute), I got an error about not enough IP addresses.

2014-01-03 14:21:13 INFO
[7fabf6bfd700] (helpers) Task bfcab88b-1862-4132-a501-1fd8a701902c (provision) message is set to Not enough free IP addresses in pool
2014-01-03 14:21:13 INFO
[7fabf6bfd700] (helpers) Task bfcab88b-1862-4132-a501-1fd8a701902c (provision) progress is set to 100
2014-01-03 14:21:13 INFO
[7fabf6bfd700] (helpers) Task bfcab88b-1862-4132-a501-1fd8a701902c (provision) status is set to error
2014-01-03 14:21:13 INFO
[7fabf6bfd700] (manager) 8
2014-01-03 14:21:12 INFO
[7fabf6bfd700] (checker) Untagged networks found, checking intersection between them...

My admin network has pool with 30 IP.

Also from docs:
«It is possible to run "fuelmenu" from a root shell on Fuel Master node after deployment to make minor changes to network interfaces, DNS, and gateway. The PXE settings, however, cannot be changed after deployment as it will lead to deployment failure.»
«Warning
Once IP settings are set at the boot time for Fuel Master node, they should not be changed during the whole lifecycle of Fuel.»
This means that it only can be done by redeploying. How to can I change IP pool without redeploying?

[Ryan Moe (rmoe)]

How many interfaces do you have on each machine? Fuel will allocate one admin IP per interface.

[Denis Ipatov (dipatov)]

Each node has 4 NIC. I know it. I'd like to change IP pool without redeploy environment.

[Vladimir Kuklin (vkuklin)]

As a workaround, you can simply update nailgun database information along with network configuration of FUEL node. Contact #fuel-dev IRC channel on freenode to figure out the details.

[Andrey Danin (gcon-monolake)]

Here is the solution: http://paste.openstack.org/show/61813/

[Andrey Danin (gcon-monolake)]

The blueprint about documenting this is created: https://blueprints.launchpad.net/fuel/+spec/doc-change-fueladmin-net-on-the-fly

[Matthew Mosesohn (raytrac3r)]

A short term fix to reduce the number of admin network IPs per node (to 1 each) is committed here: https://review.openstack.org/#/c/69617/

[Roman Alekseenkov (ralekseenkov)]

The issue was hit by 5+ external users, so I'm targeting this to 4.1

[Andrey Danin (gcon-monolake)]

Related bug https://bugs.launchpad.net/fuel/+bug/1269726
We still need to expand DHCP pool to the whole Fueladmin network.

[Matthew Mosesohn (raytrac3r)]

We could short circuit this bug another way by defaulting to a /23 network for PXE? 10.20.0.0/23 instead of 10.20.0.0/24 It would be a couple lines changed in Fuel Menu and in our ISO build script

[Andrew Woodward (xarses)]

Both this issues of managing fuelweb_admin network and consuming the entire subnet for DHCP assignment will be solved by the multiple cluster networks (multi-l3) code changes https://blueprints.launchpad.net/fuel/+spec/multiple-cluster-networks

[Matthew Mosesohn (raytrac3r)]

Andrew, this was marked "High" priority for 4.1 release, not 5.0. multi-l3 won't make it to 4.1.

[Matthew Mosesohn (raytrac3r)]

Andrey Danin, where are we with dropping the static/dynamic range split, and just use 1 combined pool for admin net?

[Andrey Danin (gcon-monolake)]

Blueprint has bee created https://blueprints.launchpad.net/fuel/+spec/get-gid-of-splitted-fueladmin-network

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/78406

[Meg McRoberts (dreidellhasa)]

Documented as a Known Issue in the 5.0 Release Notes.

[Meg McRoberts (dreidellhasa)]

Listed as "Known Issue" in 5.0.1 Release Notes.

[Dima Shulyak (dshulyak)]

I see only one way how we can properly solve this issue in 5.1:

1. Use network handlers part from @rmoe patch
   https://review.openstack.org/#/c/99179/1/nailgun/nailgun/api/v1/handlers/network.py,unified
Also it should be under v2 api.

2. Add additional action in fuel cli, which will allow to list all networks and update it.

such approach feels solid and safe to me.

So, should we do this? Or it can be postponed?

Patch with reusing dhcp address helps, but dont solves root issue.

[Aleksey Kasatkin (alekseyk-ru)]

Dmitry,

I've some concerns on this API:

https://review.openstack.org/#/c/83204/16/nailgun/nailgun/api/handlers/network.py

What about to change corresponding settings in settings.yaml and make changes in DB on nailgun restart instead?

It seems that some reconfiguration of master node networking is required also (e.g. when CIDR is changed).
That all can be done with additional command in manage.py or a script (but DB changes should be done by nailgun).

How about this?

[Dmitry Pyzhov (dpyzhov)]

This issue will affect users with more then 100 nodes. Setting lower priority and moving to 6.0

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/132654

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/132672

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/132674

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/132654
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=2c24fca5ad39c1e9e36e19f4ca95c0cb07946c4a
Submitter: Jenkins
Branch: master

commit 2c24fca5ad39c1e9e36e19f4ca95c0cb07946c4a
Author: Dima Shulyak <email address hidden>
Date: Tue Nov 4 13:12:57 2014 +0200

    Reuse addresses from dhcp pool when possible

    We can get rid from static pool, by reusing dhcp
    address, assigned to node initially.

    General procedure for assignment of ip addresses
    is not changed, but i've added one more condition
    to check if ip address is in range of admin_net.ip_range
    if it is - it will be reused when creating ip address
    if not - address will be used from pool, it will guarantee
    that patch works on upgraded environments

    Also made refactoring of old code

    DocImpact
    Closes-Bug: 1271571
    Implments: blueprint get-gid-of-splitted-fueladmin-network

    Change-Id: I1fcc4532faa6b255674b589ebd44534082e29c96

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/132672
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=32fc05f7a204205bcf68581445195adf1f277ece
Submitter: Jenkins
Branch: master

commit 32fc05f7a204205bcf68581445195adf1f277ece
Author: Dima Shulyak <email address hidden>
Date: Tue Nov 4 15:21:30 2014 +0200

    Use dhcp pool for nailgun default admin range

    As default range for nailgun dhcp pool will be used

    Change-Id: I9f31b7d6b7398fcf9b1ade55b95efc2ffc9194d8
    Related-Bug: 1271571
    Implements: blueprint get-gid-of-splitted-fueladmin-network

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/132674
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=681b28de0300294cb86bd54aa0b89ca5a924af73
Submitter: Jenkins
Branch: master

commit 681b28de0300294cb86bd54aa0b89ca5a924af73
Author: Dima Shulyak <email address hidden>
Date: Tue Nov 4 15:23:15 2014 +0200

    Remove static pool completely from fuelmenu

    Ips from dhcp pool will be reused by nailgun and
    assigned to cobbler system during provisioning

    Change-Id: Ifef74b334ec0173c4cb1d930f0a515b1c17962af
    Related-Bug: 1271571
    Implments: blueprint get-gid-of-splitted-fueladmin-network

[OpenStack Infra (hudson-openstack)]

Change abandoned by Andrew Woodward (<email address hidden>) on branch: master
Review: https://review.openstack.org/78406
Reason: was implemented in another patch