Selecting a datasource for bulk threshold config intermittently fails with "internal server error"

[Morten Brekkevold]

Threshold bulk config runs an AJAX request when a data source type is selected. This will intermittently fail with a "internal server error".

The traceback posted by Django via e-mail looks like this (NAV 3.14.1):

Traceback (most recent call last):

  File "/usr/lib/pymodules/python2.6/django/core/handlers/base.py", line 100, in get_response
    response = callback(request, *callback_args, **callback_kwargs)

  File "/usr/lib/pymodules/python2.6/nav/web/threshold/views.py", line 346, in netbox_search
    query = Netbox.objects.filter(rrdfile__rrddatasource__description=descr)

  File "/usr/lib/pymodules/python2.6/django/db/models/manager.py", line 141, in filter
    return self.get_query_set().filter(*args, **kwargs)

  File "/usr/lib/pymodules/python2.6/django/db/models/query.py", line 550, in filter
    return self._filter_or_exclude(False, *args, **kwargs)

  File "/usr/lib/pymodules/python2.6/django/db/models/query.py", line 568, in _filter_or_exclude
    clone.query.add_q(Q(*args, **kwargs))

  File "/usr/lib/pymodules/python2.6/django/db/models/sql/query.py", line 1128, in add_q
    can_reuse=used_aliases)

  File "/usr/lib/pymodules/python2.6/django/db/models/sql/query.py", line 1026, in add_filter
    negate=negate, process_extras=process_extras)

  File "/usr/lib/pymodules/python2.6/django/db/models/sql/query.py", line 1191, in setup_joins
    "Choices are: %s" % (name, ", ".join(names)))

FieldError: Cannot resolve keyword 'rrdfile' into field. Choices are: adjacencycandidate, alerthistory, alertqueue, arp, cam, category, connected_to_interface, device, discovered, eventqueue, id, info_set, interface, ip, job_log, memory, module, netboxcategory, netboxprefix, netboxvtpvlan, node_position_set, organization, powersupplyorfan, read_only, read_write, room, sensor, snmp_version, subcategories, sysname, type, unrecognizedneighbor, up, up_since, up_to_date

<ModPythonRequest
path:/threshold/netboxsearch/,
GET:<QueryDict: {}>,
POST:<QueryDict: {u'descr': [u'ifHCInOctets']}>,
COOKIES:{'nav_sessid': 'REDACTED'}
META:{'AUTH_TYPE': None,
 'CONTENT_LENGTH': '18',
 'CONTENT_TYPE': 'application/x-www-form-urlencoded; charset=UTF-8',
 'GATEWAY_INTERFACE': 'CGI/1.1',
 'HTTP_ACCEPT': 'application/json, text/javascript, */*; q=0.01',
 'HTTP_ACCEPT_CHARSET': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3',
 'HTTP_ACCEPT_ENCODING': 'gzip,deflate,sdch',
 'HTTP_ACCEPT_LANGUAGE': 'no,en-US;q=0.8,en;q=0.6',
 'HTTP_AUTHORIZATION': 'REDACTED',
 'HTTP_CONNECTION': 'keep-alive',
 'HTTP_CONTENT_LENGTH': '18',
 'HTTP_CONTENT_TYPE': 'application/x-www-form-urlencoded; charset=UTF-8',
 'HTTP_COOKIE': 'nav_sessid=REDACTED',
 'HTTP_HOST': 'nav.example.org',
 'HTTP_ORIGIN': 'https://nav.example.org',
 'HTTP_REFERER': 'https://nav.example.org/threshold/',
 'HTTP_USER_AGENT': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.43 Safari/537.31',
 'HTTP_X_NAV_AJAX': 'true',
 'HTTP_X_REQUESTED_WITH': 'XMLHttpRequest',
 'PATH_INFO': u'/threshold/netboxsearch/',
 'PATH_TRANSLATED': None,
 'QUERY_STRING': None,
 'REMOTE_ADDR': 'REDACTED',
 'REMOTE_HOST': None,
 'REMOTE_IDENT': None,
 'REMOTE_USER': None,
 'REQUEST_METHOD': 'POST',
 'SCRIPT_NAME': '',
 'SERVER_NAME': 'nav.example.org',
 'SERVER_PORT': 443,
 'SERVER_PROTOCOL': 'HTTP/1.1',
 'SERVER_SOFTWARE': 'mod_python'}>

[Morten Brekkevold (mbrekkevold)]

Seems the order in which the Netbox and RrrFile models are imported play a key role in this.

[Morten Brekkevold (mbrekkevold)]

It was quite impossible to reproduce this bug reliably, due to the intermittent nature of the bug, but it has become clear that the import order is not significant at all.

Rather, it is a more complex issue of import and usage order in the server process as a whole. Imagine this scenario:

>>> from nav.models.manage import Netbox
>>> n = Netbox.objects.filter(sysname__startswith='uninett-gw')
>>> n = n[0]
>>> n.rrdfile_set
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'Netbox' object has no attribute 'rrdfile_set'
>>> import nav.models.rrd
>>> n.rrdfile_set
<django.db.models.fields.related.RelatedManager object at 0xb6fc288c>
>>> n = Netbox.objects.filter(rrdfile__path__startswith='/usr/local/nav')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/pymodules/python2.6/django/db/models/manager.py", line 141, in filter
    return self.get_query_set().filter(*args, **kwargs)
  File "/usr/lib/pymodules/python2.6/django/db/models/query.py", line 550, in filter
    return self._filter_or_exclude(False, *args, **kwargs)
  File "/usr/lib/pymodules/python2.6/django/db/models/query.py", line 568, in _filter_or_exclude
    clone.query.add_q(Q(*args, **kwargs))
  File "/usr/lib/pymodules/python2.6/django/db/models/sql/query.py", line 1128, in add_q
    can_reuse=used_aliases)
  File "/usr/lib/pymodules/python2.6/django/db/models/sql/query.py", line 1026, in add_filter
    negate=negate, process_extras=process_extras)
  File "/usr/lib/pymodules/python2.6/django/db/models/sql/query.py", line 1191, in setup_joins
    "Choices are: %s" % (name, ", ".join(names)))
django.core.exceptions.FieldError: Cannot resolve keyword 'rrdfile' into field. Choices are: adjacencycandidate, arp, cam, category, connected_to_interface, device, discovered, id, info_set, interface, ip, job_log, memory, module, netboxcategory, netboxgroups, netboxprefix, netboxvtpvlan, organization, powersupplyorfan, read_only, read_write, room, sensor, snmp_version, sysname, type, unrecognizedneighbor, up, up_since, up_to_date

Contrast this with:

>>> from nav.models.manage import Netbox
>>> Netbox.rrdfile_set
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: type object 'Netbox' has no attribute 'rrdfile_set'
>>> import nav.models.rrd
>>> Netbox.rrdfile_set
<django.db.models.fields.related.ForeignRelatedObjectsDescriptor object at 0x8d25a4c>
>>> n = Netbox.objects.filter(rrdfile__path__startswith='/usr/local/nav')
>>>

Having used Netbox.objects.filter() before the rrdfile relation was imported makes the rrdfile relation unavailable for Netbox.objects.filter() for the remainder of the process lifetime.

I'd say this is a bug in Django, but I haven't checked to see whether it has been fixed in later versions than what's on Debian Squeeze (1.2)

[Morten Brekkevold (mbrekkevold)]

The best fix seems to be to use's Django's ability to reference foreign key models as strings, so we can import nav.models.rrd from nav.models.manage without causing import cycles.

Fix here: https://nav.uninett.no/hg/stable/rev/9437072a5f10