[nailgun] Not enough free IP addresses in pool

Bug #1271571 reported by Denis Ipatov on 2014-01-22
30
This bug affects 4 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Medium
Dima Shulyak
4.1.x
High
Fuel Python (Deprecated)
5.0.x
High
Fuel Python (Deprecated)

Bug Description

Everything worked perfectly for a 2 node system, but when I tried to create another environment with 4 nodes (1 controller, 3 compute), I got an error about not enough IP addresses.

2014-01-03 14:21:13 INFO
[7fabf6bfd700] (helpers) Task bfcab88b-1862-4132-a501-1fd8a701902c (provision) message is set to Not enough free IP addresses in pool
2014-01-03 14:21:13 INFO
[7fabf6bfd700] (helpers) Task bfcab88b-1862-4132-a501-1fd8a701902c (provision) progress is set to 100
2014-01-03 14:21:13 INFO
[7fabf6bfd700] (helpers) Task bfcab88b-1862-4132-a501-1fd8a701902c (provision) status is set to error
2014-01-03 14:21:13 INFO
[7fabf6bfd700] (manager) 8
2014-01-03 14:21:12 INFO
[7fabf6bfd700] (checker) Untagged networks found, checking intersection between them...

My admin network has pool with 30 IP.

Also from docs:
«It is possible to run "fuelmenu" from a root shell on Fuel Master node after deployment to make minor changes to network interfaces, DNS, and gateway. The PXE settings, however, cannot be changed after deployment as it will lead to deployment failure.»
«Warning
Once IP settings are set at the boot time for Fuel Master node, they should not be changed during the whole lifecycle of Fuel.»
This means that it only can be done by redeploying. How to can I change IP pool without redeploying?

Denis Ipatov (dipatov) on 2014-01-22
description: updated
Ryan Moe (rmoe) wrote :

How many interfaces do you have on each machine? Fuel will allocate one admin IP per interface.

Changed in fuel:
status: New → Confirmed
Denis Ipatov (dipatov) wrote :

Each node has 4 NIC. I know it. I'd like to change IP pool without redeploy environment.

Vladimir Kuklin (vkuklin) wrote :

As a workaround, you can simply update nailgun database information along with network configuration of FUEL node. Contact #fuel-dev IRC channel on freenode to figure out the details.

Changed in fuel:
importance: Undecided → Medium
Andrey Danin (gcon-monolake) wrote :
Andrey Danin (gcon-monolake) wrote :
Changed in fuel:
assignee: nobody → Andrey Danin (gcon-monolake)
status: Confirmed → Triaged
milestone: none → 4.1
Mike Scherbakov (mihgen) on 2014-02-05
Changed in fuel:
milestone: 4.1 → 5.0
Matthew Mosesohn (raytrac3r) wrote :

A short term fix to reduce the number of admin network IPs per node (to 1 each) is committed here: https://review.openstack.org/#/c/69617/

The issue was hit by 5+ external users, so I'm targeting this to 4.1

tags: added: customer-found
Changed in fuel:
milestone: 5.0 → 4.1
importance: Medium → High
Andrey Danin (gcon-monolake) wrote :

Related bug https://bugs.launchpad.net/fuel/+bug/1269726
We still need to expand DHCP pool to the whole Fueladmin network.

Matthew Mosesohn (raytrac3r) wrote :

We could short circuit this bug another way by defaulting to a /23 network for PXE? 10.20.0.0/23 instead of 10.20.0.0/24 It would be a couple lines changed in Fuel Menu and in our ISO build script

Andrew Woodward (xarses) wrote :

Both this issues of managing fuelweb_admin network and consuming the entire subnet for DHCP assignment will be solved by the multiple cluster networks (multi-l3) code changes https://blueprints.launchpad.net/fuel/+spec/multiple-cluster-networks

Matthew Mosesohn (raytrac3r) wrote :

Andrew, this was marked "High" priority for 4.1 release, not 5.0. multi-l3 won't make it to 4.1.

tags: added: 4.1-postponed
Changed in fuel:
milestone: 4.1 → 5.0
Matthew Mosesohn (raytrac3r) wrote :

Andrey Danin, where are we with dropping the static/dynamic range split, and just use 1 combined pool for admin net?

Fix proposed to branch: master
Review: https://review.openstack.org/78406

Changed in fuel:
assignee: Andrey Danin (gcon-monolake) → Andrew Woodward (xarses)
status: Triaged → In Progress
tags: added: backports-4.1.1
tags: removed: 4.1-postponed backports-4.1.1
Andrew Woodward (xarses) on 2014-04-22
Changed in fuel:
milestone: 5.0 → 5.1
Mike Scherbakov (mihgen) on 2014-05-08
tags: added: release-notes
Changed in fuel:
status: In Progress → Confirmed

Documented as a Known Issue in the 5.0 Release Notes.

Meg McRoberts (dreidellhasa) wrote :

Listed as "Known Issue" in 5.0.1 Release Notes.

Dmitry Ilyin (idv1985) on 2014-07-15
summary: - Not enough free IP addresses in pool
+ [nailgun] Not enough free IP addresses in pool
Dmitry Pyzhov (dpyzhov) on 2014-07-15
no longer affects: fuel/5.1.x
Dima Shulyak (dshulyak) wrote :

I see only one way how we can properly solve this issue in 5.1:

1. Use network handlers part from @rmoe patch
   https://review.openstack.org/#/c/99179/1/nailgun/nailgun/api/v1/handlers/network.py,unified
Also it should be under v2 api.

2. Add additional action in fuel cli, which will allow to list all networks and update it.

such approach feels solid and safe to me.

So, should we do this? Or it can be postponed?

Patch with reusing dhcp address helps, but dont solves root issue.

Aleksey Kasatkin (alekseyk-ru) wrote :

Dmitry,

I've some concerns on this API:

https://review.openstack.org/#/c/83204/16/nailgun/nailgun/api/handlers/network.py

What about to change corresponding settings in settings.yaml and make changes in DB on nailgun restart instead?

It seems that some reconfiguration of master node networking is required also (e.g. when CIDR is changed).
That all can be done with additional command in manage.py or a script (but DB changes should be done by nailgun).

How about this?

Dmitry Pyzhov (dpyzhov) wrote :

This issue will affect users with more then 100 nodes. Setting lower priority and moving to 6.0

Changed in fuel:
milestone: 5.1 → 6.0
importance: High → Medium

Fix proposed to branch: master
Review: https://review.openstack.org/132654

Changed in fuel:
assignee: Fuel Python Team (fuel-python) → Dima Shulyak (dshulyak)
status: Confirmed → In Progress

Reviewed: https://review.openstack.org/132654
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=2c24fca5ad39c1e9e36e19f4ca95c0cb07946c4a
Submitter: Jenkins
Branch: master

commit 2c24fca5ad39c1e9e36e19f4ca95c0cb07946c4a
Author: Dima Shulyak <email address hidden>
Date: Tue Nov 4 13:12:57 2014 +0200

    Reuse addresses from dhcp pool when possible

    We can get rid from static pool, by reusing dhcp
    address, assigned to node initially.

    General procedure for assignment of ip addresses
    is not changed, but i've added one more condition
    to check if ip address is in range of admin_net.ip_range
    if it is - it will be reused when creating ip address
    if not - address will be used from pool, it will guarantee
    that patch works on upgraded environments

    Also made refactoring of old code

    DocImpact
    Closes-Bug: 1271571
    Implments: blueprint get-gid-of-splitted-fueladmin-network

    Change-Id: I1fcc4532faa6b255674b589ebd44534082e29c96

Changed in fuel:
status: In Progress → Fix Committed

Reviewed: https://review.openstack.org/132672
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=32fc05f7a204205bcf68581445195adf1f277ece
Submitter: Jenkins
Branch: master

commit 32fc05f7a204205bcf68581445195adf1f277ece
Author: Dima Shulyak <email address hidden>
Date: Tue Nov 4 15:21:30 2014 +0200

    Use dhcp pool for nailgun default admin range

    As default range for nailgun dhcp pool will be used

    Change-Id: I9f31b7d6b7398fcf9b1ade55b95efc2ffc9194d8
    Related-Bug: 1271571
    Implements: blueprint get-gid-of-splitted-fueladmin-network

Reviewed: https://review.openstack.org/132674
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=681b28de0300294cb86bd54aa0b89ca5a924af73
Submitter: Jenkins
Branch: master

commit 681b28de0300294cb86bd54aa0b89ca5a924af73
Author: Dima Shulyak <email address hidden>
Date: Tue Nov 4 15:23:15 2014 +0200

    Remove static pool completely from fuelmenu

    Ips from dhcp pool will be reused by nailgun and
    assigned to cobbler system during provisioning

    Change-Id: Ifef74b334ec0173c4cb1d930f0a515b1c17962af
    Related-Bug: 1271571
    Implments: blueprint get-gid-of-splitted-fueladmin-network

Change abandoned by Andrew Woodward (<email address hidden>) on branch: master
Review: https://review.openstack.org/78406
Reason: was implemented in another patch

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers