Upload users from CSV could do with a progress bar

[Nigel Cunningham]

When you click the submission button in the form for uploading users from a CSV file, current master displays changes the button to "Processing...". Since the operation can take quite a while, it would be better if a new form along the lines of the progress bar in the bulk user export was displayed.

[Robert Lyon (robertl-9)]

To have a progress bar here we would need to work out early on how many users we are to add and then have the uploader report back how many we have done as we loop through the making of new users.

[Nigel Cunningham (nigelc-g)]

That's entirely doable. We already check the number of lines in the CSV very early in the process (search for $maxcsvlines).

I'll prepare a patch when I get done with the memory limit bug that prompted this :)

[Eduardo (o-eduardo)]

Tested on Mahara version 1.10.0dev (2014091600). 3 times to make sure it works. How to test:

1. Log in as admin (tested both site admin and institution admin)
2. Go to Users → Add users by CSV
3. Click on Browse... → Select your CSV file with 1000 users.
4. Click on Add users by CSV → A progress bar should appear → Passed
5. Go to Groups → Add groups by CSV
6. Click on Browse... → Select your groups CSV file
7. Click on Add groups by CSV → A progress bar should appear → Passed
8. Go to Update group members by CSV
9. Click on Browse... → Select your update groups CSV file
10. Click on Update group members by CSV → A progress bar should appear → Passed

[Kristina Hoeppner (kris-hoeppner)]

patch at https://reviews.mahara.org/#/c/3624/

[Mahara Bot (dev-mahara)]

Reviewed: https://reviews.mahara.org/3624
Committed: http://gitorious.org/mahara/mahara/commit/55a8deb8cbd2e1455c7af3a35c50b7cad76d63df
Submitter: Robert Lyon (<email address hidden>)
Branch: master

commit 55a8deb8cbd2e1455c7af3a35c50b7cad76d63df
Author: Nigel Cunningham <email address hidden>
Date: Fri Sep 5 08:49:32 2014 +1000

(Bug1352028) Add a JSON progress bar for bulk operations.

This patch adds a JSON progress meter (I'll call it that to avoid confusion
with progress bars) to the bulk uploading of users, groups and group
memberships and the bulk export and import of users (LEAP), so the user can see
the progress of the operation and not just the submit button changed to
'Processing..' and whatever indication their browser gives while waiting for
content.

The bulk export and import are minor rewrites, replacing the old iframe based
progress bar and the associated multiple pages and additional template file in
the case of the bulk export, and the recursive redirect-to-self of the bulk
import.

To accomplish the display of the progress bar during the operation, we make the
PHP session be closed (read only) except when changes need to be made. This is
for the most part a straightforward change in session.php as it's the only
direct accessor. In other places, we replace direct accessing of the session
variable ($_SESSION) with use of the session class ($SESSION) so that it can
reopen the session, make the change and close the session again.

There is one more aspect to all of this: with previous behaviour, multiple
requests for the same session would queue, taking the session lock in turn.
After this patch is applied, they can proceed in parallel, allowing greater
throughput. There is no additional locking requirement because the issues are
the same as those already dealt with in allowing multiple PHP threads to
process requests from different sessions at the same time.

I have sought to make the progress meter nice and generic, so it can be used in
the other bulk imports and exports too.

Paradoxically, these changes don't just make the import seem to be faster, it
actually is.. at least in the case of users and groups.

Times for importing 1000 users, groups and memberships, averaged over 3 runs
each (Wall time, not CPU time - but the relationship is the same).

Without Progress With Progress
Users 166s 155s
Groups 85s 78s
Memberships 20s 19s

Change-Id: Iec15c57db32c77994edb80c71d65591de51a95e4
Signed-off-by: Nigel Cunningham <email address hidden>