[2.0 RC2] Static IP address are allowed to be created in a dynamic range

Bug #1604461 reported by Larry Michel on 2016-07-19
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
MAAS
Critical
Mike Pontillo
2.0
Critical
Mike Pontillo
Trunk
Critical
Mike Pontillo

Bug Description

The regiond logs full of these exception below. It's not clear as to how we got to this point, but the IP address in question is from the dynamic range.

i/2.0/users/?op=whoami HTTP/1.1" 200 80 "-" "Go-http-client/1.1"
2016-07-17 08:15:47 [-] Unhandled failure in updating lease.
 Traceback (most recent call last):
   File "/usr/lib/python3.5/threading.py", line 862, in run
     self._target(*self._args, **self._kwargs)
   File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 904, in worker
     return target()
   File "/usr/lib/python3/dist-packages/twisted/_threads/_threadworker.py", line 46, in work
     task()
   File "/usr/lib/python3/dist-packages/twisted/_threads/_team.py", line 190, in doWork
     task()
 --- <exception caught here> ---
   File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 246, in inContext
     result = inContext.theWork()
   File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 262, in <lambda>
     inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
   File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 118, in callWithContext
     return self.currentContext().callWithContext(ctx, func, *args, **kw)
   File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 81, in callWithContext
     return func(*args,**kw)
   File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 937, in callInContext
     return func(*args, **kwargs)
   File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 205, in wrapper
     return func(*args, **kwargs)
   File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 602, in call_within_transaction
     return func_outside_txn(*args, **kwargs)
   File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 433, in retrier
     return func(*args, **kwargs)
   File "/usr/lib/python3.5/contextlib.py", line 30, in inner
     return func(*args, **kwds)
   File "/usr/lib/python3/dist-packages/maasserver/rpc/leases.py", line 134, in update_lease
     lease_time=lease_time, created=created, updated=created)
   File "/usr/lib/python3/dist-packages/django/db/models/manager.py", line 127, in manager_method
     return getattr(self.get_queryset(), name)(*args, **kwargs)
   File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 348, in create
     obj.save(force_insert=True, using=self.db)
   File "/usr/lib/python3/dist-packages/maasserver/models/cleansave.py", line 29, in save
     return super(CleanSave, self).save(*args, **kwargs)
   File "/usr/lib/python3/dist-packages/maasserver/models/timestampedmodel.py", line 64, in save
     return super(TimestampedModel, self).save(*args, **kwargs)
   File "/usr/lib/python3/dist-packages/django/db/models/base.py", line 734, in save
     force_update=force_update, update_fields=update_fields)
   File "/usr/lib/python3/dist-packages/django/db/models/base.py", line 762, in save_base
     updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
   File "/usr/lib/python3/dist-packages/django/db/models/base.py", line 846, in _save_table
     result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
   File "/usr/lib/python3/dist-packages/django/db/models/base.py", line 885, in _do_insert
     using=using, raw=raw)
   File "/usr/lib/python3/dist-packages/django/db/models/manager.py", line 127, in manager_method
     return getattr(self.get_queryset(), name)(*args, **kwargs)
   File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 920, in _insert
     return query.get_compiler(using=using).execute_sql(return_id)
   File "/usr/lib/python3/dist-packages/django/db/models/sql/compiler.py", line 974, in execute_sql
     cursor.execute(sql, params)
   File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 64, in execute
     return self.cursor.execute(sql, params)
   File "/usr/lib/python3/dist-packages/django/db/utils.py", line 98, in __exit__
     six.reraise(dj_exc_type, dj_exc_value, traceback)
   File "/usr/lib/python3/dist-packages/django/utils/six.py", line 685, in reraise
     raise value.with_traceback(tb)
   File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 64, in execute
     return self.cursor.execute(sql, params)
 django.db.utils.IntegrityError: duplicate key value violates unique constraint "maasserver_staticipaddress_ip_key"
 DETAIL: Key (ip)=(**.***.**.**) already exists.

2016-07-17 08:15:50 [-] 127.0.0.1 - - [17/Jul/2016:08:15:49 +0000] "GET /MAAS/api/2.0/version/ HTTP/1.1" 200 151 "-" "Go-http-client/1.1"

ubuntu@maas2-production:/var/log/maas$ dpkg -l '*maas*'|cat
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-===============================-==================================-============-=================================================
ii maas 2.0.0~rc2+bzr5156-0ubuntu1~xenial1 all "Metal as a Service" is a physical cloud and IPAM
ii maas-cli 2.0.0~rc2+bzr5156-0ubuntu1~xenial1 all MAAS client and command-line interface
un maas-cluster-controller <none> <none> (no description available)
ii maas-common 2.0.0~rc2+bzr5156-0ubuntu1~xenial1 all MAAS server common files
ii maas-dhcp 2.0.0~rc2+bzr5156-0ubuntu1~xenial1 all MAAS DHCP server
ii maas-dns 2.0.0~rc2+bzr5156-0ubuntu1~xenial1 all MAAS DNS server
ii maas-proxy 2.0.0~rc2+bzr5156-0ubuntu1~xenial1 all MAAS Caching Proxy
ii maas-rack-controller 2.0.0~rc2+bzr5156-0ubuntu1~xenial1 all Rack Controller for MAAS
ii maas-region-api 2.0.0~rc2+bzr5156-0ubuntu1~xenial1 all Region controller API service for MAAS
ii maas-region-controller 2.0.0~rc2+bzr5156-0ubuntu1~xenial1 all Region Controller for MAAS
un maas-region-controller-min <none> <none> (no description available)
un python-django-maas <none> <none> (no description available)
un python-maas-client <none> <none> (no description available)
un python-maas-provisioningserver <none> <none> (no description available)
ii python3-django-maas 2.0.0~rc2+bzr5156-0ubuntu1~xenial1 all MAAS server Django web framework (Python 3)
ii python3-maas-client 2.0.0~rc2+bzr5156-0ubuntu1~xenial1 all MAAS python API client (Python 3)
ii python3-maas-provisioningserver 2.0.0~rc2+bzr5156-0ubuntu1~xenial1 all MAAS server provisioning libraries (Python 3)

I am attaching the logs and dhcp lease files.

Tags: oil Edit Tag help

Related branches

Larry Michel (lmic) wrote :
Larry Michel (lmic) wrote :

Seeing more duplicate and they all seem to be in the dynamic range.

Blake Rouse (blake-rouse) wrote :

The issue is that you reserved static IP addresses in the dynamic range. This should not be allowed, a dynamic range should also not be allowed to be created over any static IP addresses.

summary: - [2.0 RC2] regiond log errors: duplicate key value violates unique
- constraint "maasserver_staticipaddress_ip_key" DETAIL: Key
- (ip)=(**.***.**.**) already exist
+ [2.0 RC2] Static IP address are allowed to be created in a dynamic
+ range, and the reverse is true
summary: [2.0 RC2] Static IP address are allowed to be created in a dynamic
- range, and the reverse is true
+ range and the reverse is true
Changed in maas:
status: New → Triaged
importance: Undecided → Critical
milestone: none → 2.0.0

There is already code that checks if an IP address has been allocated inside the dynamic range (and if so, prevents creation of the range).

Currently, dynamic ranges cannot overlap other ranges, or reserved IP addresses. Reserved ranges can overlap reserved IP addresses, but not other ranges.

So I think the only thing to fix here is to verify that a user-reserved IP address is not within a dynamic range when reserving it.

(As an aside, if you wish to reserve an IP addresses inside a dynamic range, create a device first so that MAAS can add a DHCP mapping for the MAC to the reserved IP.)

summary: [2.0 RC2] Static IP address are allowed to be created in a dynamic
- range and the reverse is true
+ range
Changed in maas:
status: Triaged → Fix Committed

Any chance of escalating this into Beta 8 and releasing it. It basically renders MAAS unable to commission nodes.

Hi Trevor,

Why are you still in beta8? MAAS 2.0 rc3 has been released a few weeks
ago...

PPA:Maas/next

On Monday, August 8, 2016, Trevor G Francis <trevor.francis@46labs.com>
wrote:

> Any chance of escalating this into Beta 8 and releasing it. It basically
> renders MAAS unable to commission nodes.
>
> --
> You received this bug notification because you are subscribed to MAAS.
> https://bugs.launchpad.net/bugs/1604461
>
> Title:
> [2.0 RC2] Static IP address are allowed to be created in a dynamic
> range
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/maas/+bug/1604461/+subscriptions
>

--
Andres Rodriguez (RoAkSoAx)
Ubuntu Server Developer
MSc. Telecom & Networking
Systems Engineer

Download full text (7.0 KiB)

Sorry about that. Updated today....issue still exists:

MAAS Version 2.0.0 (rc4+bzr5187)

Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: 2016-08-11 00:38:03 [-] Unhandled failure in updating lease.
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011Traceback (most recent call last):
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011 File "/usr/lib/python3.5/threading.py", line 862, in run
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011 self._target(*self._args, **self._kwargs)
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011 File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 904, in worker
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011 return target()
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011 File "/usr/lib/python3/dist-packages/twisted/_threads/_threadworker.py", line 46, in work
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011 task()
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011 File "/usr/lib/python3/dist-packages/twisted/_threads/_team.py", line 190, in doWork
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011 task()
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011--- <exception caught here> ---
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011 File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 246, in inContext
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011 result = inContext.theWork()
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011 File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 262, in <lambda>
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011 inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011 File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 118, in callWithContext
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011 return self.currentContext().callWithContext(ctx, func, *args, **kw)
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011 File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 81, in callWithContext
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011 return func(*args,**kw)
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011 File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 937, in callInContext
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011 return func(*args, **kwargs)
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011 File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 205, in wrapper
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011 return func(*args, **kwargs)
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011 File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 602, in call_within_transaction
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011 return func_outside_txn(*args, **kwargs)
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011 File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 433, in retrier
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011 return func(*args, **kwargs)
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011 File "/usr/lib/python3.5/contextlib.py", line 30, in inner
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011 return func(*args, **kwds)
Aug 11 00:38:03 r6-c1-n1-maas sh[5475]: #011 File "/usr/lib/python3/dist-packages...

Read more...

Andres Rodriguez (andreserl) wrote :

Hi Trevor,

The bugfix will prevent any *future* creation of IP reservations inside a dynamic rnage, but it won't clean up any reservation already made. You'll need to do that by unreserving the IP address inside the dynamic range.

Hope this helps.

Can you suggest how to do that?

Changed in maas:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Bug attachments