Upload users from CSV performance improvements.

Bug #1353802 reported by Nigel Cunningham on 2014-08-07
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mahara
Medium
Nigel Cunningham

Bug Description

In doing the memory limit work, I've prepared a number of patches to improve the performance of the various import and export functions. This bug is for submitting those patches against master.

Reviewed: https://reviews.mahara.org/3610
Committed: http://gitorious.org/mahara/mahara/commit/4bd9d4930153eef45cbebcc84ea07deee62a7345
Submitter: Robert Lyon (<email address hidden>)
Branch: master

commit 4bd9d4930153eef45cbebcc84ea07deee62a7345
Author: Nigel Cunningham <email address hidden>
Date: Tue Jul 29 14:04:56 2014 +1000

(Bug1353802) Remember and reuse last db_format_timestamp output if poss.

This patch adds support for remembering the last parameter used
by ArtefactType::commit to db_format_timestamp and the output of
the function, and reusing the output if the input is the same on
the next call.

In a test import of 2000 users, this patch reduced the number of
invocations of db_format_timestamp from 80,000 (3.6s) to 14,800
(.3s).

Change-Id: I6f5575f9d3649ff53514f9927744a01966d103dd
Signed-off-by: Nigel Cunningham <email address hidden>

There are several patches for this. So please do not put the bug to finished.

Nigel: It would be better if you had 1 bug for each of your patches to keep track of them more easily esp. when they are separated patches and not connected.

Changed in mahara:
status: New → In Progress
importance: Undecided → Medium
assignee: nobody → Nigel Cunningham (nigelc-g)
milestone: none → 1.10.0
Nigel Cunningham (nigelc-g) wrote :

Hi Kristina.

Thanks for your email - yes, I wanted to split them up, and have found time to do that this morning. I've made new bugs for each of the patches, so this one can now be closed. Not sure what status to pick though. Suggestions?

Hi Nigel,

I'll just set it to "Invalid". Then it should really show up anymore.

Changed in mahara:
status: In Progress → Invalid
milestone: 1.10.0 → none
Mahara Bot (dev-mahara) wrote :

Reviewed: https://reviews.mahara.org/3612
Committed: http://gitorious.org/mahara/mahara/commit/a7a32f257839692d4e7e757efe196f02d6853b26
Submitter: Robert Lyon (<email address hidden>)
Branch: master

commit a7a32f257839692d4e7e757efe196f02d6853b26
Author: Nigel Cunningham <email address hidden>
Date: Tue Jul 29 15:12:27 2014 +1000

(Bug1353802) Don't call delete_records for new artefacts.

In ArtefactType::commit, don't bother calling delete_records
if we know the artefact that is being committed is new. In
testing the addition 2000 users from a CSV file, this patch
shaved 4 seconds off the completion time.

Change-Id: I5b22cb5fc936e67e9a37a0022a4488fc30604d03
Signed-off-by: Nigel Cunningham <email address hidden>

Changed in mahara:
status: Invalid → Fix Committed
milestone: none → 1.10.0
Aaron Wells (u-aaronw) on 2014-10-21
Changed in mahara:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers