WebUI overrides proxy values that have not changed (was GET metadata with MAAS built-in proxy enabled fails with "500 Internal Server Error")

Bug #1787381 reported by Stefan Fleischmann on 2018-08-16
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MAAS
Critical
Andres Rodriguez
2.3
Critical
Andres Rodriguez
2.4
Critical
Andres Rodriguez

Bug Description

I upgraded our MAAS server from 2.3.3 to 2.4.1 and cannot commission nodes anymore. On the commissioned node I can see that the retrieval of 'http://172.17.100.3:5240/MAAS/metadata/latest/by-id/asfery/?op=get_preseed' fails with status "500 Internal Server Error". In the log file /var/log/maas/regiond.log I see this:

2018-08-16 05:20:49 maasserver: [error] ################################ Exception: %d format: a number is required, not NoneType ################################
2018-08-16 05:20:49 maasserver: [error] Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/python3/dist-packages/maasserver/utils/views.py", line 277, in view_atomic_with_post_commit_savepoint
    return view_atomic(*args, **kwargs)
  File "/usr/lib/python3.6/contextlib.py", line 52, in inner
    return func(*args, **kwds)
  File "/usr/lib/python3/dist-packages/maasserver/api/support.py", line 70, in __call__
    response = upcall(request, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/django/views/decorators/vary.py", line 21, in inner_func
    response = func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/piston3/resource.py", line 190, in __call__
    result = self.error_handler(e, request, meth, em_format)
  File "/usr/lib/python3/dist-packages/piston3/resource.py", line 188, in __call__
    result = meth(request, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/maasserver/api/support.py", line 310, in dispatch
    return function(self, request, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/metadataserver/api.py", line 1149, in get_preseed
    preseed = get_preseed(node, region_ip)
  File "/usr/lib/python3/dist-packages/maasserver/preseed.py", line 616, in get_preseed
    default_region_ip=default_region_ip)
  File "/usr/lib/python3/dist-packages/maasserver/preseed.py", line 936, in render_preseed
    default_region_ip=default_region_ip))
  File "/usr/lib/python3/dist-packages/maasserver/preseed.py", line 864, in get_node_preseed_context
    default_region_ip=default_region_ip),
  File "/usr/lib/python3/dist-packages/maasserver/compose_preseed.py", line 435, in compose_preseed
    node, token, base_url, default_region_ip=default_region_ip)
  File "/usr/lib/python3/dist-packages/maasserver/compose_preseed.py", line 312, in compose_commissioning_preseed
    default_region_ip=default_region_ip)
  File "/usr/lib/python3/dist-packages/maasserver/compose_preseed.py", line 365, in _compose_cloud_init_preseed
    node.get_boot_rack_controller(), default_region_ip=default_region_ip)
  File "/usr/lib/python3/dist-packages/maasserver/compose_preseed.py", line 45, in get_apt_proxy
    url = "http://:%d/" % maas_proxy_port
TypeError: %d format: a number is required, not NoneType

2018-08-16 05:20:49 regiond: [info] 172.17.100.123 GET /MAAS/metadata/latest/by-id/asfery/?op=get_preseed HTTP/1.1 --> 500 INTERNAL_SERVER_ERROR (referrer: -; agent: Cloud-Init/18.3
-9-g2e62cb8a-0ubuntu1~18.04.2)

I quickly checked in a fresh container, installed MAAS from scratch and think I see the same there. If I go into settings and change the proxy from "MAAS Built-in proxy" to "Don't use a proxy" the problem disappears and I can commission nodes again. List of installed packages:

# dpkg -l | egrep 'maas|cloud-init'
ii cloud-init 18.3-9-g2e62cb8a-0ubuntu1~18.04.2 all Init scripts for cloud instances
ii cloud-initramfs-copymods 0.40ubuntu1 all copy initramfs modules into root filesystem for later use
ii cloud-initramfs-dyn-netconf 0.40ubuntu1 all write a network interface file in /run for BOOTIF
ii maas 2.4.1-7032-g11e4fa330-0ubuntu1~18.04.1 all "Metal as a Service" is a physical cloud and IPAM
ii maas-cli 2.4.1-7032-g11e4fa330-0ubuntu1~18.04.1 all MAAS client and command-line interface
ii maas-common 2.4.1-7032-g11e4fa330-0ubuntu1~18.04.1 all MAAS server common files
ii maas-dhcp 2.4.1-7032-g11e4fa330-0ubuntu1~18.04.1 all MAAS DHCP server
ii maas-proxy 2.4.1-7032-g11e4fa330-0ubuntu1~18.04.1 all MAAS Caching Proxy
ii maas-rack-controller 2.4.1-7032-g11e4fa330-0ubuntu1~18.04.1 all Rack Controller for MAAS
ii maas-region-api 2.4.1-7032-g11e4fa330-0ubuntu1~18.04.1 all Region controller API service for MAAS
ii maas-region-controller 2.4.1-7032-g11e4fa330-0ubuntu1~18.04.1 all Region Controller for MAAS
ii python3-django-maas 2.4.1-7032-g11e4fa330-0ubuntu1~18.04.1 all MAAS server Django web framework (Python 3)
ii python3-maas-client 2.4.1-7032-g11e4fa330-0ubuntu1~18.04.1 all MAAS python API client (Python 3)
ii python3-maas-provisioningserver 2.4.1-7032-g11e4fa330-0ubuntu1~18.04.1 all MAAS server provisioning libraries (Python 3)

Related branches

Download full text (6.4 KiB)

Hi Stefan,

Thank you for you bug report. Can you please get some data for me:

maas <user> maas get-config name=maas_proxy_port

And attach /var/lib/maas/maas-proxy.conf

Once you do that, you can fix this by doing

maas <user> maas set-config name=maas_proxy_port value=8080

Thanks

On Thu, Aug 16, 2018 at 7:30 AM Stefan Fleischmann <
<email address hidden>> wrote:

> Public bug reported:
>
> I upgraded our MAAS server from 2.3.3 to 2.4.1 and cannot commission
> nodes anymore. On the commissioned node I can see that the retrieval of
> 'http://172.17.100.3:5240/MAAS/metadata/latest/by-
> id/asfery/?op=get_preseed' fails with status "500 Internal Server
> Error". In the log file /var/log/maas/regiond.log I see this:
>
> 2018-08-16 05:20:49 maasserver: [error] ################################
> Exception: %d format: a number is required, not NoneType
> ################################
> 2018-08-16 05:20:49 maasserver: [error] Traceback (most recent call last):
> File "/usr/lib/python3/dist-packages/django/core/handlers/base.py", line
> 185, in _get_response
> response = wrapped_callback(request, *callback_args, **callback_kwargs)
> File "/usr/lib/python3/dist-packages/maasserver/utils/views.py", line
> 277, in view_atomic_with_post_commit_savepoint
> return view_atomic(*args, **kwargs)
> File "/usr/lib/python3.6/contextlib.py", line 52, in inner
> return func(*args, **kwds)
> File "/usr/lib/python3/dist-packages/maasserver/api/support.py", line
> 70, in __call__
> response = upcall(request, *args, **kwargs)
> File "/usr/lib/python3/dist-packages/django/views/decorators/vary.py",
> line 21, in inner_func
> response = func(*args, **kwargs)
> File "/usr/lib/python3/dist-packages/piston3/resource.py", line 190, in
> __call__
> result = self.error_handler(e, request, meth, em_format)
> File "/usr/lib/python3/dist-packages/piston3/resource.py", line 188, in
> __call__
> result = meth(request, *args, **kwargs)
> File "/usr/lib/python3/dist-packages/maasserver/api/support.py", line
> 310, in dispatch
> return function(self, request, *args, **kwargs)
> File "/usr/lib/python3/dist-packages/metadataserver/api.py", line 1149,
> in get_preseed
> preseed = get_preseed(node, region_ip)
> File "/usr/lib/python3/dist-packages/maasserver/preseed.py", line 616,
> in get_preseed
> default_region_ip=default_region_ip)
> File "/usr/lib/python3/dist-packages/maasserver/preseed.py", line 936,
> in render_preseed
> default_region_ip=default_region_ip))
> File "/usr/lib/python3/dist-packages/maasserver/preseed.py", line 864,
> in get_node_preseed_context
> default_region_ip=default_region_ip),
> File "/usr/lib/python3/dist-packages/maasserver/compose_preseed.py",
> line 435, in compose_preseed
> node, token, base_url, default_region_ip=default_region_ip)
> File "/usr/lib/python3/dist-packages/maasserver/compose_preseed.py",
> line 312, in compose_commissioning_preseed
> default_region_ip=default_region_ip)
> File "/usr/lib/python3/dist-packages/maasserver/compose_preseed.py",
> line 365, in _compose_cloud_init_preseed
> node.get_boot_rack_controller(), default_region_ip=defaul...

Read more...

Changed in maas:
status: New → Incomplete

$ maas <user> maas get-config name=maas_proxy_port
Success.
Machine-readable output follows:
null

The same on the newly deployed instance.

PS: in the web interface going into the settings and disabling the proxy resets to "null", but changing back to "MAAS Built-in" does not set the port. Setting it via the CLI with the command you provided works and commissioning nodes works again. Thanks for the quick response!

maas proxy conf attached

Did the ‘set-config’ command fix your issue?

On Thu, Aug 16, 2018 at 9:35 AM Stefan Fleischmann <
<email address hidden>> wrote:

> maas proxy conf attached
>
> ** Attachment added: "maas-proxy.conf"
>
> https://bugs.launchpad.net/maas/+bug/1787381/+attachment/5176327/+files/maas-proxy.conf
>
> --
> You received this bug notification because you are subscribed to MAAS.
> https://bugs.launchpad.net/bugs/1787381
>
> Title:
> GET metadata with MAAS built-in proxy enabled fails with "500 Internal
> Server Error"
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/maas/+bug/1787381/+subscriptions
>
> Launchpad-Notification-Type: bug
> Launchpad-Bug: product=maas; status=Incomplete; importance=Undecided;
> assignee=None;
> Launchpad-Bug-Information-Type: Public
> Launchpad-Bug-Private: no
> Launchpad-Bug-Security-Vulnerability: no
> Launchpad-Bug-Commenters: andreserl sfleischmann
> Launchpad-Bug-Reporter: Stefan Fleischmann (sfleischmann)
> Launchpad-Bug-Modifier: Stefan Fleischmann (sfleischmann)
> Launchpad-Message-Rationale: Subscriber (MAAS)
> Launchpad-Message-For: andreserl
>
--
Andres Rodriguez (RoAkSoAx)
Ubuntu Server Developer
MSc. Telecom & Networking
Systems Engineer

Changed in maas:
milestone: none → 2.5.0alpha2

Yes the set-config command fixed it. But if one changes the proxy settings in the web-ui as I described in #3 this has to be done again via the set-config command. I.e. via the web-ui I can *disable* the proxy but cannot *enable* it.

Changed in maas:
status: Incomplete → Confirmed
importance: Undecided → Critical
Changed in maas:
assignee: nobody → Andres Rodriguez (andreserl)
status: Confirmed → In Progress
Moula BADJI (moulab1) wrote :
Changed in maas:
status: In Progress → Fix Committed
Andres Rodriguez (andreserl) wrote :

@Stephan,

This is now fixed and the package is available in ppa:maas/proposed. Would you mind testing that please?

Hi Andres,

I updated to 2.4.2-7034-g2f5deb8b8-0ubuntu1~18.04.1 from ppa:maas/proposed and can confirm that this bug is fixed. Commission and deploy work now with both settings (Don't use a proxy & MAAS built-in proxy) when maas_proxy_port is NOT set.

~# maas <user> maas get-config name=maas_proxy_port
Success.
Machine-readable output follows:
null

@moulab1, it seems like you have a different problem there. In my case the maas-proxy service was running fine.

Changed in maas:
status: Fix Committed → Fix Released
summary: - GET metadata with MAAS built-in proxy enabled fails with "500 Internal
- Server Error"
+ WebUI overrides proxy values that have not changed (was GET metadata
+ with MAAS built-in proxy enabled fails with "500 Internal Server Error")

Hi, after upgrading to MAAS 2.5.0-7442-gdf68e30a5-0ubuntu1~18.04 I had to set the proxy port manually again via
~# maas <user> maas set-config name=maas_proxy_port value=8000

because I was getting these errors endlessly repeated in /var/log/maas/rackd.log:
---
2019-01-17 22:34:16 provisioningserver.rackdservices.external: [critical] Failed to get external services configurations.
        Traceback (most recent call last):
        Failure: twisted.protocols.amp.UnhandledCommand: (b'UNHANDLED', 'Unknown Error [maas:pid=931:cmd=GetProxyConfiguration:ask=72c]')

2019-01-17 22:34:21 provisioningserver.rackdservices.external: [critical] Failed to get external services configurations.
        Traceback (most recent call last):
        Failure: twisted.protocols.amp.UnhandledCommand: (b'UNHANDLED', 'Unknown Error [maas:pid=929:cmd=GetProxyConfiguration:ask=73b]')

2019-01-17 22:34:26 provisioningserver.rackdservices.external: [critical] Failed to get external services configurations.
        Traceback (most recent call last):
        Failure: twisted.protocols.amp.UnhandledCommand: (b'UNHANDLED', 'Unknown Error [maas:pid=929:cmd=GetProxyConfiguration:ask=741]')

2019-01-17 22:34:31 provisioningserver.rackdservices.external: [critical] Failed to get external services configurations.
        Traceback (most recent call last):
        Failure: twisted.protocols.amp.UnhandledCommand: (b'UNHANDLED', 'Unknown Error [maas:pid=932:cmd=GetProxyConfiguration:ask=7aa]')
---

And the proxy was starting and stopping all the time.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers