Network names with dots cause internal server error when on node pages

[Liam Young]

MaaS 1.7.0~rc3+bzr3299 on 14.04

When using the WebUI to access information about some nodes (eg http://test01/MAAS/nodes/node-3f1251d2-b437-11e3-b1b6-d4bed9a84493/view/ ) I get "Internal server error. "

I am not getting this for all nodes. The corresponding stacktrace from maas-django.log:

ERROR 2014-11-12 08:52:03,644 django.request Internal Server Error: /MAAS/nodes/node-3f1251d2-b437-11e3-b1b6-d4bed9a84493/view/
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/django/core/handlers/base.py", line 137, in get_response
    response = response.render()
  File "/usr/lib/python2.7/dist-packages/django/template/response.py", line 105, in render
    self.content = self.rendered_content
  File "/usr/lib/python2.7/dist-packages/django/template/response.py", line 82, in rendered_content
    content = template.render(context)
  File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 140, in render
    return self._render(context)
  File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 134, in _render
    return self.nodelist.render(context)
  File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 840, in render
    bit = self.render_node(node, context)
  File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 854, in render_node
    return node.render(context)
  File "/usr/lib/python2.7/dist-packages/django/template/loader_tags.py", line 123, in render
    return compiled_parent._render(context)
  File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 134, in _render
    return self.nodelist.render(context)
  File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 840, in render
    bit = self.render_node(node, context)
  File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 854, in render_node
    return node.render(context)
  File "/usr/lib/python2.7/dist-packages/django/template/loader_tags.py", line 62, in render
    result = block.nodelist.render(context)
  File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 840, in render
    bit = self.render_node(node, context)
  File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 854, in render_node
    return node.render(context)
  File "/usr/lib/python2.7/dist-packages/django/template/defaulttags.py", line 203, in render
    nodelist.append(node.render(context))
  File "/usr/lib/python2.7/dist-packages/django/template/defaulttags.py", line 305, in render
    return nodelist.render(context)
  File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 840, in render
    bit = self.render_node(node, context)
  File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 854, in render_node
    return node.render(context)
  File "/usr/lib/python2.7/dist-packages/django/template/defaulttags.py", line 203, in render
    nodelist.append(node.render(context))
  File "/usr/lib/python2.7/dist-packages/django/template/defaulttags.py", line 447, in render
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/dist-packages/django/template/defaulttags.py", line 433, in render
    url = reverse(view_name, args=args, kwargs=kwargs, current_app=context.current_app)
  File "/usr/lib/python2.7/dist-packages/django/core/urlresolvers.py", line 536, in reverse
    return iri_to_uri(resolver._reverse_with_prefix(view, prefix, *args, **kwargs))
  File "/usr/lib/python2.7/dist-packages/django/core/urlresolvers.py", line 456, in _reverse_with_prefix
    (lookup_view_s, args, kwargs, len(patterns), patterns))
NoReverseMatch: Reverse for 'network-view' with arguments '(u'os.magners.qa.lexington-eth1',)' and keyword arguments '{}' not found. 1 pattern(s) tried: [u'networks/(?P<name>[\\w\\-]+)/view/$']

Circumstantially, I would say this error seems to appear once the unit has been used to house a windows guest.

[Liam Young (gnuoy)]

Probably not a great suprise but /MAAS/networks/ also returns an Internal server error.

ERROR 2014-11-12 09:11:34,402 django.request Internal Server Error: /MAAS/networks/
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/django/core/handlers/base.py", line 137, in get_response
    response = response.render()
  File "/usr/lib/python2.7/dist-packages/django/template/response.py", line 105, in render
    self.content = self.rendered_content
  File "/usr/lib/python2.7/dist-packages/django/template/response.py", line 82, in rendered_content
    content = template.render(context)
  File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 140, in render
    return self._render(context)
  File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 134, in _render
    return self.nodelist.render(context)
  File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 840, in render
    bit = self.render_node(node, context)
  File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 854, in render_node
    return node.render(context)
  File "/usr/lib/python2.7/dist-packages/django/template/loader_tags.py", line 123, in render
    return compiled_parent._render(context)
  File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 134, in _render
    return self.nodelist.render(context)
  File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 840, in render
    bit = self.render_node(node, context)
  File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 854, in render_node
    return node.render(context)
  File "/usr/lib/python2.7/dist-packages/django/template/loader_tags.py", line 62, in render
    result = block.nodelist.render(context)
  File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 840, in render
    bit = self.render_node(node, context)
  File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 854, in render_node
    return node.render(context)
  File "/usr/lib/python2.7/dist-packages/django/template/defaulttags.py", line 305, in render
    return nodelist.render(context)
  File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 840, in render
    bit = self.render_node(node, context)
  File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 854, in render_node
    return node.render(context)
  File "/usr/lib/python2.7/dist-packages/django/template/defaulttags.py", line 203, in render
    nodelist.append(node.render(context))
  File "/usr/lib/python2.7/dist-packages/django/template/defaulttags.py", line 447, in render
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/dist-packages/django/template/defaulttags.py", line 433, in render
    url = reverse(view_name, args=args, kwargs=kwargs, current_app=context.current_app)
  File "/usr/lib/python2.7/dist-packages/django/core/urlresolvers.py", line 536, in reverse
    return iri_to_uri(resolver._reverse_with_prefix(view, prefix, *args, **kwargs))
  File "/usr/lib/python2.7/dist-packages/django/core/urlresolvers.py", line 456, in _reverse_with_prefix
    (lookup_view_s, args, kwargs, len(patterns), patterns))
NoReverseMatc...

[Liam Young (gnuoy)]

I missed the most important point here. Once a node is in this state juju seems unable to deploy to it and juju status returns this:

  "1":
    agent-state-info: "getNetworkMACs failed: gomaasapi: got error back from server:
      404 NOT FOUND (<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Transitional//EN\"
      \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd\">\n<html xmlns=\"http://www.w3.org/1999/xhtml\"
      xml:lang=\"en\" lang=\"en\">\n <head>\n <meta http-equiv=\"Content-type\"
      content=\"text/html; charset=utf-8\" />\n <meta http-equiv=\"Content-Language\"
      content=\"en-us\" />\n <link rel=\"shortcut icon\" href=\"/MAAS/static/img/favicon.ico\"/>\n
      \ \n <link rel=\"stylesheet\" href=\"/MAAS/combo/maas/?css/base.css&css/typography.css&css/forms.css&css/layout.css&css/modifiers.css&css/components/flash_messages.css&css/components/pagination.cs
s&css/components/table_list.css&css/components/title_form.css&css/components/blocks.css&css/components/fluid.css&css/components/yui_panel.css&css/components/yui_overlay.css&css/components/yui_node_add.css&cs
s/components/data_list.css&css/components/search_box.css&css/components/slider.css&css/components/spinner.css&css/ubuntu-webfonts.css&css/multiselect_widget.css\"
      />\n <title>Error: Page not found | Sagacious MAAS\n</title>\n <script
      type=\"text/javascript\">\n<!--\nvar YUI_config = {\n debug: false,\n combine:
      true,\n filter: 'min',\n root: '',\n base: '/MAAS/combo/yui/?',\n comboBase:
      '/MAAS/combo/yui/?',\n};\nvar MAAS_config = {\n uris: {\n login: '/MAAS/accounts/login/',\n
      \ statics: '/MAAS/static/',\n maas_handler: '/MAAS/api/1.0/maas/',\n
      \ nodes_handler: '/MAAS/api/1.0/nodes/',\n account_handler: '/MAAS/api/1.0/account/',\n
      \ images_handler: '/MAAS/images/'\n },\n debug: false\n};\n//
      -->\n</script>\n<script type=\"text/javascript\"\n src=\"/MAAS/combo/yui/?yui-base/yui-base-min.js\">\n</script>\n<script
      type=\"text/javascript\"\n src=\"/MAAS/combo/maas/?js/image.js&js/image_views.js&js/license_key.js&js/morph.js&js/user_panel.js&js/node_add.js&js/node.js&js/prefs.js&js/utils.js&js/node_views.js&js/
node_check.js&js/shortpoll.js&js/enums.js&js/os_distro_select.js&js/power_parameters.js&js/nodes_chart.js&js/reveal.js\">\n</script>\n\n
      \ \n \n</head>\n<body class=\"\">\n \n <div class=\"center-page-wrapper\">\n
      \ <div class=\"center-page-content\">\n \n <div id=\"body\">\n <ul
      id=\"flash-messages\">\n \n \n </ul>\n \n <h1
      id=\"page-title\">Error: Page not found</h1>\n \n <div id=\"content\">\n
      \ <div id=\"sidebar\"></div>\n \n <h2>\n The requested
      URL /MAAS/api/1.0/networks/os.magners.qa.lexington-eth1/ was not found on this
      server.\n </h2>\n\n <div class=\"clear\"></div>\n </div>\n
      \ </div>\n <div id=\"footer\">\n <img src=\"/MAAS/static/img/ubuntu_logo_footer.png\"
      alt=\"Ubuntu\" />\n \n <p>&copy; 201...

[Graham Binns (gmb)]

On 12 November 2014 14:29, Liam Young <email address hidden> wrote:
> I missed the most important point here. Once a node is in this state
> juju seems unable to deploy to it and juju status returns this:

Definitely a separate bug in Juju, there…

[Liam Young (gnuoy)]

Raised Bug #1391941 for the juju status horrible handling of errors from MaaS

[Andres Rodriguez (andreserl)]

NoReverseMatch: Reverse for 'network-view' with arguments '(u'os.magners.qa.lexington-eth1',)' and keyword arguments '{}' not found. 1 pattern(s) tried: [u'networks/(?P<name>[\\w\\-]+)/view/$']
ERROR 2014-11-12 08:53:43,845 maasserver Unable to identify boot image for (ubuntu/amd64/generic/trusty/local): cluster 'os.magners.qa.lexington' does not have matching boot image.
ERROR 2014-11-12 08:54:27,793 maasserver Unable to identify boot image for (ubuntu/amd64/generic/trusty/local): cluster 'os.magners.qa.lexington' does not have matching boot image.
ERROR 2014-11-12 08:56:03,303 django.request Internal Server Error: /MAAS/nodes/node-3f1251d2-b437-11e3-b1b6-d4bed9a84493/view/
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/django/core/handlers/base.py", line 137, in get_response
    response = response.render()
  File "/usr/lib/python2.7/dist-packages/django/template/response.py", line 105, in render
    self.content = self.rendered_content
  File "/usr/lib/python2.7/dist-packages/django/template/response.py", line 82, in rendered_content
    content = template.render(context)
  File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 140, in render
    return self._render(context)
  File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 134, in _render
    return self.nodelist.render(context)
  File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 840, in render
    bit = self.render_node(node, context)
  File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 854, in render_node
    return node.render(context)
  File "/usr/lib/python2.7/dist-packages/django/template/loader_tags.py", line 123, in render
    return compiled_parent._render(context)
  File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 134, in _render
    return self.nodelist.render(context)
  File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 840, in render
    bit = self.render_node(node, context)
  File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 854, in render_node
    return node.render(context)
  File "/usr/lib/python2.7/dist-packages/django/template/loader_tags.py", line 62, in render
    result = block.nodelist.render(context)
  File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 840, in render
    bit = self.render_node(node, context)
  File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 854, in render_node
    return node.render(context)
  File "/usr/lib/python2.7/dist-packages/django/template/defaulttags.py", line 203, in render
    nodelist.append(node.render(context))
  File "/usr/lib/python2.7/dist-packages/django/template/defaulttags.py", line 305, in render
    return nodelist.render(context)
  File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 840, in render
    bit = self.render_node(node, context)
  File "/usr/lib/python2.7/dist-packages/django/template/base.py", line 854, in render_node
    return node.render(context)
  File "/usr/lib/python2.7/dist-packages/django/template/defaulttags.py", line 203, in render
    nodelist.append(node.render(context))
  File "/usr...

[Graham Binns (gmb)]

On 12 November 2014 18:30, Andres Rodriguez <email address hidden> wrote:
> [u'networks/(?P<name>[\\w\\-]+)/view/
> ]

Looks to me as though the regex in the url definition expects only A-Z, a-z, 0-9, _ and -. The dots in the name that's being looked up are what's blowing things up here.

[Graham Binns (gmb)]

I'll pick this up later; it's a quick fix.

[Christian Reis (kiko)]

Andres and I considered this today, and he even tried to reproduce by indicating a network name with a dot in it, which was caught by the validator. We both seemed to recall that we added validation to refuse dots in network names in the 1.7.0 milestone. Is that true?

[Christian Reis (kiko)]

If it is true and dots are illegal, do we just need to release note this?

Alternatively, do we need to do some data migration?

[Graham Binns (gmb)]

So, the problem is that the validator catches the dots, which is fine, but the URL pattern doesn't allow for them and blows up. We can fix the bug to cope with legacy names, or I can write a schema migration that would turn the dots to, say, hyphens. But if we don't do one of these, someone, somewhere, is going to hit this.

[Graham Binns (gmb)]

> We both seemed to recall that we added validation to refuse dots in network names in the 1.7.0 milestone. Is that true?

The validator was added back in February (r1882.3.2), at least according to bzr, so I'm not sure how this invalid network name actually ended up in the DB.

The more I think about it, the more I see that:

 1. We need to cope with people putting bad names into URLs. Blowing up because part of the URL doesn't match the regexp is just silly.
 2. We need to check that there are actually bad data in MAAS's database here, and that Juju isn't doing something strange.
 3. If there networks with dotted names in the MAAS DB, we should write a migration to make sure that doesn't bite someone.

Liam, can you run the following on the maas region and paste the output for us:

$ sudo maas-region-admin dumpdata maasserver.Networks

[Liam Young (gnuoy)]

$ sudo maas-region-admin dumpdata maasserver.Networks
CommandError: Unknown model: maasserver.Networks

but sudo maas-region-admin dumpdata maasserver.network did work and I've attatched the output

[Graham Binns (gmb)]

On 13 November 2014 07:17, Liam Young <email address hidden> wrote:
> but sudo maas-region-admin dumpdata maasserver.network did work and I've
> attatched the output

Never type commands from memory late at night! Thanks :)

[Graham Binns (gmb)]

Okay, so looking at the data, there are *definitely* invalid names in
there, which must've come from before the validation was added to
disallow dots in network names.

I'll write a migration to fix these up, which can be backported to
1.7.0(?), and I'll also look into the blowing-up problem, which is
still bizarre to me.

[Jeroen T. Vermeulen (jtv)]

ISTM the blowing-up happens because the URL pattern for network-view requires that network names consist of only "word" ([\w]) characters and dashes. When the name contains another character, that regex in urls.py no longer matches and lookup fails. Django goes looking for an alternate resolution, but finds none.

[Graham Binns (gmb)]

On 13 November 2014 09:29, Jeroen T. Vermeulen
<email address hidden> wrote:
> ISTM the blowing-up happens because the URL pattern for network-view
> requires that network names consist of only "word" ([\w]) characters and
> dashes. When the name contains another character, that regex in urls.py
> no longer matches and lookup fails. Django goes looking for an
> alternate resolution, but finds none.

Right. That behaviour seems oddly fragile to me — a typo in the URL
shouldn't cause a 500.

We could fix it by allowing any character in the URL and then
returning a 404 when we can't find the Network, which seems like the
sane thing to do.

[Graham Binns (gmb)]

I've now landed the migration to fix this. The second part of the problem — 500 errors when there's bad data in the database — isn't easy to fix. Django behaves entirely correctly when you type a URL wrong (or with invalid characters), and returns a 404.

The error happens when we're building the Networks list. reverse() blows up when trying to construct the link to the network detail view, because the regex is too restrictive.

We could fix the problem by relaxing the regexes, but if we're going to do that we need to fix more than just this case: the [\w-] regex is used in many places in src/maasserver/urls.py, so we may want to revisit this as a policy decision.