Implement bulk insert using Django

Bug #829211 reported by Mattia Barbon
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenQuake (deprecated)
Fix Released
Medium
Mattia Barbon

Bug Description

The current bulk insert implementation uses SQLAlchemy; it should be rewritten to use Django ORM.

Depends from https://bugs.launchpad.net/bugs/820346
Related to https://bugs.launchpad.net/bugs/809780

There isn't a bulk inserter built-in in Django; some third party solutions:

http://djangosnippets.org/snippets/446/
+ very complete and handles relations (not needed for now but nice)
- not maintained (last updated in 2008)
- might be a pain to debug/update

http://djangosnippets.org/snippets/2362/
+ simple and basically what we have now
- does not handle relations (but we do not need them ATM)

http://people.iola.dk/olau/python/bulkops.py
Similar to snippet 2362 but uses .executemany rather than bulk insert
- executemany is slower than bulk insert

http://stackoverflow.com/questions/2655670/how-do-i-perform-a-batch-insert-in-django
Uses a custom manager so the bulk insert method is available directly in the model class
- I like the separate bulk insert class better, but it's a personal preference

Changed in openquake:
status: New → Confirmed
assignee: nobody → Mattia Barbon (mattia.barbon)
milestone: none → 0.4.3
Revision history for this message
Mattia Barbon (mattia.barbon) wrote :
Changed in openquake:
status: Confirmed → In Progress
Changed in openquake:
status: In Progress → Fix Committed
tags: added: database performance
Changed in openquake:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Related blueprints

Remote bug watches

Bug watches keep track of this bug in other bug trackers.