ipdevpoll multiprocess mode broken in NAV 4.4.0

[Morten Brekkevold]

Attempting to run ipdevpoll in multiprocess mode under NAV 4.4.0 results in tracebacks from each of the spawned worker processes. The processes end up stalled, and no collection takes place.

The traceback looks like this:

2016-01-19 12:37:12,776 [31470] [INFO plugins] Imported 26 plugin classes, 26 classes in plugin registry
Unhandled error in Deferred:
Unhandled Error
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 525, in __bootstrap
    self.__bootstrap_inner()
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
--- <exception caught here> ---
  File "/usr/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 207, in _worker
    result = context.call(ctx, function, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext
    return func(*args,**kw)
  File "/usr/lib/python2.7/dist-packages/nav/ipdevpoll/db.py", line 91, in _reset
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/nav/ipdevpoll/dataloader.py", line 80, in load_all_s
    'snmpAgentState').values_list('netbox__id', flat=True))
  File "/usr/lib/python2.7/dist-packages/nav/models/event.py", line 498, in unresolved
    return self.filter(filtr)
  File "/usr/lib/python2.7/dist-packages/django/db/models/manager.py", line 92, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 691, in filter
    return self._filter_or_exclude(False, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 709, in _filter_or_exclude
    clone.query.add_q(Q(*args, **kwargs))
  File "/usr/lib/python2.7/dist-packages/django/db/models/sql/query.py", line 1287, in add_q
    clause, require_inner = self._add_q(where_part, self.used_aliases)
  File "/usr/lib/python2.7/dist-packages/django/db/models/sql/query.py", line 1309, in _add_q
    current_negated)
  File "/usr/lib/python2.7/dist-packages/django/db/models/sql/query.py", line 1314, in _add_q
    current_negated=current_negated, connector=connector)
  File "/usr/lib/python2.7/dist-packages/django/db/models/sql/query.py", line 1138, in build_filter
    lookups, parts, reffed_aggregate = self.solve_lookup_type(arg)
  File "/usr/lib/python2.7/dist-packages/django/db/models/sql/query.py", line 1076, in solve_lookup_type
    _, field, _, lookup_parts = self.names_to_path(lookup_splitted, self.get_meta())
  File "/usr/lib/python2.7/dist-packages/django/db/models/sql/query.py", line 1339, in names_to_path
    field, model, direct, m2m = opts.get_field_by_name(name)
  File "/usr/lib/python2.7/dist-packages/django/db/models/options.py", line 416, in get_field_by_name
    cache = self.init_name_map()
  File "/usr/lib/python2.7/dist-packages/django/db/models/options.py", line 445, in init_name_map
    for f, model in self.get_all_related_m2m_objects_with_model():
  File "/usr/lib/python2.7/dist-packages/django/db/models/options.py", line 563, in get_all_related_m2m_objects_with_model
    cache = self._fill_related_many_to_many_cache()
  File "/usr/lib/python2.7/dist-packages/django/db/models/options.py", line 577, in _fill_related_many_to_many_cache
    for klass in self.apps.get_models():
  File "/usr/lib/python2.7/dist-packages/django/utils/lru_cache.py", line 101, in wrapper
    result = user_function(*args, **kwds)
  File "/usr/lib/python2.7/dist-packages/django/apps/registry.py", line 168, in get_models
    self.check_models_ready()
  File "/usr/lib/python2.7/dist-packages/django/apps/registry.py", line 131, in check_models_ready
    raise AppRegistryNotReady("Models aren't loaded yet.")
django.core.exceptions.AppRegistryNotReady: Models aren't loaded yet.

[Morten Brekkevold (mbrekkevold)]

The nav.models module performs a very naive verification of the DJANGO_SETTINGS_MODULE environment variable, only running "django.setup()" if it was unset at module load time.

ipdevpoll, and other binaries in NAV, defer to this module to ensure all the Django models are properly loaded under Django 1.7, and do not perform this check themselves.

The problem here is that the environment variable is already set as the ipdevpoll child processes start (and it could be for any NAV program in a production setup), cause the django.setup() to never be run. This results in the django.core.exceptions.AppRegistryNotReady exception.

[Morten Brekkevold (mbrekkevold)]

fix here: https://nav.uninett.no/hg/stable/rev/7c19c4031375

There are followup fixes for potential deadlock problems. Thanks, Django.