MAAS

[2.0a1/2.0pre-a2] Fail to commission when Fabric on Machine Interface and Rack Interface dont match

Bug #1554566 reported by Andres Rodriguez
14
This bug affects 3 people
Affects Status Importance Assigned to Milestone
MAAS
Fix Released
Critical
Blake Rouse
MAAS 2.0.0

Bug Description

==> /var/log/maas/regiond.log <==
2016-03-12 00:16:59 [maasserver] ERROR: ################################ Exception: 'NoneType' object has no attribute 'url' ################################
2016-03-12 00:16:59 [maasserver] ERROR: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/django/core/handlers/base.py", line 132, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/python3/dist-packages/maasserver/utils/views.py", line 180, in view_atomic_with_post_commit_savepoint
    return view_atomic(*args, **kwargs)
  File "/usr/lib/python3.5/contextlib.py", line 30, in inner
    return func(*args, **kwds)
  File "/usr/lib/python3/dist-packages/maasserver/api/pxeconfig.py", line 202, in pxeconfig
    preseed_url = compose_preseed_url(node)
  File "/usr/lib/python3/dist-packages/maasserver/preseed.py", line 704, in compose_preseed_url
    base_url = node.get_boot_primary_rack_controller().url
AttributeError: 'NoneType' object has no attribute 'url'

2016-03-12 00:16:59 [-] 127.0.0.1 - - [11/Mar/2016:16:16:58 +0000] "GET /MAAS/api/2.0/pxeconfig/?local=192.168.10.27&bios_boot_method=pxe&mac=74-d4-35-89-bd-2b&ra
ckcontroller_id=4y3h7n&remote=192.168.10.59 HTTP/1.0" 500 40 "-" "provisioningserver.pserv_services.tftp.TFTPBackend"

See original description

Related branches

lp:~blake-rouse/maas/tftp-rpc-boot-config
Mike Pontillo (community): Approve
Diff: 1581 lines (+571/-212)
11 files modified
src/maasserver/rpc/boot.py (+16/-2)
src/maasserver/rpc/regionservice.py (+3/-3)
src/maasserver/rpc/tests/test_boot.py (+87/-34)
src/maasserver/rpc/tests/test_regionservice.py (+2/-0)
src/provisioningserver/plugin.py (+13/-10)
src/provisioningserver/pserv_services/tests/test_tftp.py (+209/-84)
src/provisioningserver/pserv_services/tftp.py (+94/-73)
src/provisioningserver/rpc/region.py (+2/-0)
src/provisioningserver/tests/test_plugin.py (+1/-5)
src/provisioningserver/utils/tests/test_twisted.py (+95/-0)
src/provisioningserver/utils/twisted.py (+49/-1)
Andres Rodriguez (andreserl)
Changed in maas:
milestone: none → 2.0.0
importance: Undecided → Critical
importance: Critical → High
Revision history for this message
Andres Rodriguez (andreserl) wrote : Re: [2.0a1] PXE interface is incorrectly determined or not determined at all

from another bug:

2016-03-11 14:57:52 [maasserver] ERROR: ################################ Exception: 'NoneType' object has no attribute 'url' ################################
2016-03-11 14:57:52 [maasserver] ERROR: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/django/core/handlers/base.py", line 132, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/python3/dist-packages/maasserver/utils/views.py", line 180, in view_atomic_with_post_commit_savepoint
    return view_atomic(*args, **kwargs)
  File "/usr/lib/python3.5/contextlib.py", line 30, in inner
    return func(*args, **kwds)
  File "/usr/lib/python3/dist-packages/maasserver/api/pxeconfig.py", line 202, in pxeconfig
    preseed_url = compose_preseed_url(node)
  File "/usr/lib/python3/dist-packages/maasserver/preseed.py", line 704, in compose_preseed_url
    base_url = node.get_boot_primary_rack_controller().url
AttributeError: 'NoneType' object has no attribute 'url'

summary: - [2.0a1] PXE interface is incorrectly determined
+ [2.0a1] PXE interface is incorrectly determined or not determined at all
Revision history for this message
Andres Rodriguez (andreserl) wrote :

https://bugs.launchpad.net/maas/+bug/1555570

Revision history for this message
Andres Rodriguez (andreserl) wrote :

In my lab:

{
    "address_ttl": null,
    "hostname": "advective-simonne",
    "boot_interface": {
        "mac_address": "74:d4:35:89:bd:2b"
    },
    "virtualblockdevice_set": [],
    "status_message": null,
    "architecture": "amd64/generic",
    "status": 0,
    "status_action": null,
    "power_state": "unknown",
    "power_type": "ipmi",
    "blockdevice_set": [],
    "storage": 0.0,
    "physicalblockdevice_set": [],
    "cpu_count": 0,
    "system_id": "4y3haq",
    "osystem": "",
    "macaddress_set": [
        {
            "mac_address": "74:d4:35:89:bd:2b"
        },
        {
            "mac_address": "74:d4:35:89:bd:2c"
        }
    ],
    "min_hwe_kernel": "",
    "owner": null,
    "interface_set": [
        {
            "discovered": null,
            "id": 136,
            "name": "eth0",
            "mac_address": "74:d4:35:89:bd:2b",
            "parents": [],
            "enabled": true,
            "type": "physical",
            "links": [],
            "vlan": {
                "vid": 0,
                "resource_uri": "/MAAS/api/2.0/vlans/1/",
                "secondary_rack": null,
                "id": 1,
                "fabric": "fabric-0",
                "mtu": 1500,
                "name": "untagged",
                "primary_rack": null,
                "dhcp_on": false
            },
            "resource_uri": "/MAAS/api/2.0/nodes/4y3haq/interfaces/136/",
            "params": "",
            "tags": [],
            "children": [],
            "effective_mtu": 1500
        },
        {
            "discovered": null,
            "id": 137,
            "name": "eth1",
            "mac_address": "74:d4:35:89:bd:2c",
            "parents": [],
            "enabled": true,
            "type": "physical",
            "links": [],
            "vlan": {
                "vid": 0,
                "resource_uri": "/MAAS/api/2.0/vlans/1/",
                "secondary_rack": null,
                "id": 1,
                "fabric": "fabric-0",
                "mtu": 1500,
                "name": "untagged",
                "primary_rack": null,
                "dhcp_on": false
            },
            "resource_uri": "/MAAS/api/2.0/nodes/4y3haq/interfaces/137/",
            "params": "",
            "tags": [],
            "children": [],
            "effective_mtu": 1500
        }
    ],
    "boot_disk": null,
    "ip_addresses": [],
    "memory": 0,
    "fqdn": "advective-simonne.maas",
    "tag_names": [],
    "disable_ipv4": false,
    "node_type_name": "Machine",
    "zone": {
        "description": "",
        "resource_uri": "/MAAS/api/2.0/zones/default/",
        "name": "default"
    },
    "domain": {
        "resource_record_count": 18,
        "resource_uri": "/MAAS/api/2.0/domains/0/",
        "id": 0,
        "name": "maas",
        "authoritative": true,
        "ttl": null
    },
    "distro_series": "",
    "node_type": 0,
    "resource_uri": "/MAAS/api/2.0/machines/4y3haq/",
    "hwe_kernel": "",
    "netboot": true,
    "swap_size": null,
    "status_name": "New"
}

description: updated
description: updated
Andres Rodriguez (andreserl)
Changed in maas:
importance: High → Critical
summary: - [2.0a1] PXE interface is incorrectly determined or not determined at all
+ [2.0a1] PXE interface is incorrectly determined
Andres Rodriguez (andreserl)
summary: - [2.0a1] PXE interface is incorrectly determined
+ [2.0a1] PXE interface & fabric are incorrectly determined
summary: - [2.0a1] PXE interface & fabric are incorrectly determined
+ [2.0a1] Fabric are incorrectly determined, which may cause PXE interface
+ incorrectly showed in the UI
description: updated
Revision history for this message
Andres Rodriguez (andreserl) wrote : Re: [2.0a1] Fabric are incorrectly determined, which may cause PXE interface incorrectly showed in the UI

Ok, I dunno if this is related, but can't commission. It seems it boots into the enlistment process instead of commissioning. I'm using manual power type, and I see the error in:

2016-03-12 00:16:59+0800 [-] Stopping protocol <tftp.bootstrap.RemoteOriginReadSession object at 0x7f077cc1ee10>
2016-03-12 00:16:59+0800 [TFTP (UDP)] Datagram received from ('192.168.10.59', 49159): <RRQDatagram(filename=b'pxelinux.cfg/2bbd8935-d474-8001-e311-f0a10054a09d',
 mode=b'octet', options=OrderedDict([(b'tsize', b'0'), (b'blksize', b'1408')]))>
2016-03-12 00:16:59+0800 [TFTP (UDP)] Datagram received from ('192.168.10.59', 49160): <RRQDatagram(filename=b'pxelinux.cfg/01-74-d4-35-89-bd-2b', mode=b'octet',
options=OrderedDict([(b'tsize', b'0'), (b'blksize', b'1408')]))>

==> /var/log/maas/regiond.log <==
2016-03-12 00:16:59 [maasserver] ERROR: ################################ Exception: 'NoneType' object has no attribute 'url' ################################
2016-03-12 00:16:59 [maasserver] ERROR: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/django/core/handlers/base.py", line 132, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/python3/dist-packages/maasserver/utils/views.py", line 180, in view_atomic_with_post_commit_savepoint
    return view_atomic(*args, **kwargs)
  File "/usr/lib/python3.5/contextlib.py", line 30, in inner
    return func(*args, **kwds)
  File "/usr/lib/python3/dist-packages/maasserver/api/pxeconfig.py", line 202, in pxeconfig
    preseed_url = compose_preseed_url(node)
  File "/usr/lib/python3/dist-packages/maasserver/preseed.py", line 704, in compose_preseed_url
    base_url = node.get_boot_primary_rack_controller().url
AttributeError: 'NoneType' object has no attribute 'url'

2016-03-12 00:16:59 [-] 127.0.0.1 - - [11/Mar/2016:16:16:58 +0000] "GET /MAAS/api/2.0/pxeconfig/?local=192.168.10.27&bios_boot_method=pxe&mac=74-d4-35-89-bd-2b&ra
ckcontroller_id=4y3h7n&remote=192.168.10.59 HTTP/1.0" 500 40 "-" "provisioningserver.pserv_services.tftp.TFTPBackend"

Andres Rodriguez (andreserl)
summary: - [2.0a1] Fabric are incorrectly determined, which may cause PXE interface
- incorrectly showed in the UI
+ [2.0a1] Fabric are incorrectly determined during enlistment, which may
+ cause PXE interface incorrectly showed in the UI
Revision history for this message
Andres Rodriguez (andreserl) wrote : Re: [2.0a1] Fabric are incorrectly determined during enlistment, which may cause PXE interface incorrectly showed in the UI

Ok, so I think this relates to: https://bugs.launchpad.net/maas/+bug/1555570 as well.

So I re-install to re-test and I had this configuration

1. a maas server with eth0, eth1, eth2, where eth0 is the network MAAS is managing. This, by default was in fabric-2.
2. maas node with eth0, eth1, both NIC's were auto placed on fabric-0

On commissioning, I'd see the failure: AttributeError: 'NoneType' object has no attribute 'url' so, what I did is to ensure that eth0, the interface where I was managing DHCP was connected to fabric-0 instead of fabric-0. After I did that, the machine commissioned successfully.

Andres Rodriguez (andreserl)
summary: - [2.0a1] Fabric are incorrectly determined during enlistment, which may
- cause PXE interface incorrectly showed in the UI
+ [2.0a1/2.0pre-a2] Fabric are incorrectly determined during enlistment,
+ which may cause PXE interface incorrectly showed in the UI
Andres Rodriguez (andreserl)
summary: - [2.0a1/2.0pre-a2] Fabric are incorrectly determined during enlistment,
- which may cause PXE interface incorrectly showed in the UI
+ [2.0a1/2.0pre-a2] Incorrect fabric on machine interface, causes
+ commissioning to fail
Andres Rodriguez (andreserl)
description: updated
summary: - [2.0a1/2.0pre-a2] Incorrect fabric on machine interface, causes
- commissioning to fail
+ [2.0a1/2.0pre-a2] Outdated Fabric may cause commissioning to fail
summary: - [2.0a1/2.0pre-a2] Outdated Fabric may cause commissioning to fail
+ [2.0a1/2.0pre-a2] Fail to commission when Fabric on Machine Interface
+ and Rack Interface dont match
Revision history for this message
Andres Rodriguez (andreserl) wrote :

Work around:

The Rack Controller interface that's on the same subnet as the subnet where DHCP has been enabled, needs to always be in Fabric-0. This can be done in the Rack Controller detail page, under the Interfaces section.

Revision history for this message
Michael Foord (mfoord) wrote :

This workaround works for me. Thanks.

Revision history for this message
Michael Foord (mfoord) wrote :

Except now enlisting doesn't work (no pxe boot).

Revision history for this message
Andres Rodriguez (andreserl) wrote : Re: [Bug 1554566] Re: [2.0a1/2.0pre-a2] Fail to commission when Fabric on Machine Interface and Rack Interface dont match

Michael,

Are you only importing Xenial?

On Monday, March 14, 2016, Michael Foord <email address hidden> wrote:

> Except now enlisting doesn't work (no pxe boot).
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1554566
>
> Title:
> [2.0a1/2.0pre-a2] Fail to commission when Fabric on Machine Interface
> and Rack Interface dont match
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/maas/+bug/1554566/+subscriptions
>

--
Andres Rodriguez (RoAkSoAx)
Ubuntu Server Developer
MSc. Telecom & Networking
Systems Engineer

Revision history for this message
John George (jog) wrote :

CDOQA tests also ran into what appears to be this same issue, while commissioning a Node into 2.0a2.
I was also able to work around it by setting the Nodes network interface, that's used to PXE boot, to match the fabric for that managed subnet (fabric-3 in this case).

2016-03-15 00:18:24 [maasserver] ERROR: ################################ Exception: 'NoneType' object has no attribute 'url' ################################
2016-03-15 00:18:24 [maasserver] ERROR: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/django/core/handlers/base.py", line 132, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/python3/dist-packages/maasserver/utils/views.py", line 180, in view_atomic_with_post_commit_savepoint
    return view_atomic(*args, **kwargs)
  File "/usr/lib/python3.5/contextlib.py", line 30, in inner
    return func(*args, **kwds)
  File "/usr/lib/python3/dist-packages/maasserver/api/support.py", line 54, in __call__
    response = upcall(request, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/django/views/decorators/vary.py", line 21, in inner_func
    response = func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/piston3/resource.py", line 190, in __call__
    result = self.error_handler(e, request, meth, em_format)
  File "/usr/lib/python3/dist-packages/piston3/resource.py", line 188, in __call__
    result = meth(request, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/maasserver/api/support.py", line 210, in dispatch
    return function(self, request, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/metadataserver/api.py", line 768, in get_preseed
    return HttpResponse(get_preseed(node), content_type="text/plain")
  File "/usr/lib/python3/dist-packages/maasserver/preseed.py", line 393, in get_preseed
    release=Config.objects.get_config('commissioning_distro_series'))
  File "/usr/lib/python3/dist-packages/maasserver/preseed.py", line 680, in render_preseed
    node, osystem, release, rack_controller=rack_controller))
  File "/usr/lib/python3/dist-packages/maasserver/preseed.py", line 628, in get_node_preseed_context
    base_url=rack_controller.url)
AttributeError: 'NoneType' object has no attribute 'url'

Revision history for this message
Michael Foord (mfoord) wrote :

Andres: I have 14.0.4 and 15.10 images.

Gavin Panella (allenap)
Changed in maas:
status: New → Triaged
Blake Rouse (blake-rouse)
Changed in maas:
status: Triaged → In Progress
assignee: nobody → Blake Rouse (blake-rouse)
MAAS Lander (maas-lander)
Changed in maas:
status: In Progress → Fix Committed
Andres Rodriguez (andreserl)
Changed in maas:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.