migrate-upload-data uses lots of memory.
Bug #1189808 reported by
Daniel Holbach
This bug affects 1 person
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
D-A-T Overview |
In Progress
|
Medium
|
Andrew Starr-Bochicchio |
Bug Description
I couldn't quite figure out why on a quick glance, but on a machine with little memory usage, 1.5G of RAM and 2G of Swap this process gets killed.
Related branches
lp:~andrewsomething/dat-overview/lp1189808
- Daniel Holbach: Approve
-
Diff: 60 lines (+22/-5)1 file modifiedoverview/uploads/management/commands/migrate-upload-data.py (+22/-5)
Changed in dat-overview: | |
importance: | Undecided → Medium |
status: | New → Triaged |
Changed in dat-overview: | |
status: | Triaged → In Progress |
assignee: | nobody → Andrew Starr-Bochicchio (andrewsomething) |
To post a comment you must log in.
Arg... Was this doing an initial import or just updating?
My quick off hand guess is that it's because I use Uploads. objects. bulk_create( bulk_insert) . This speeds things up dramatically as it allows adding a lot of rows to the database while only calling save once instead of saving on each row, but it also means everything is loaded into one variable. Maybe there's a way to do it in batches?