Web Client: Pending User Buckets for more than 100 users

[Anna Goben]

When uploading a batch file of patrons to the Pending interface of User Buckets, if the list that you wish to add is more than 100 patrons long, you cannot send the entire list to the User Bucket. Only the number shown on the first page are sent and the rest are autocleared. Since the lists don't load consecutively (consistently), and you have to exit and reenter the User Bucket interface to reload the same list, batch loading is limited to a maximum of 100 patron accounts at a time.

If the whole list (however many pages there are) cannot be selected and moved to the bucket at once, then only the entries that have already been added to the bucket should autopurge from the Pending interface.

[Terran McCanna (tmccanna)]

I suspect this has the same underlying cause as this other pending user bucket bug: https://bugs.launchpad.net/evergreen/+bug/1749970

[Anna Goben (agoben)]

I thought that bug was for Pending Patrons not the User Bucket interface. Does Pending Patrons use User Buckets as a frame?

FWIW, I can load and view a few hundred patrons into the pending tab of the User Bucket interface without any issue, but actually moving them en masse fails. :(

[Terran McCanna (tmccanna)]

Sorry, I saw "Pending Users" and thought "Pending Patrons" - never mind!

[Angela Kilsdonk (akilsdonk)]

I can confirm the behavior Anna sees in version 3.0.5.

[Terran McCanna (tmccanna)]

Also related - I tried importing a CSV file of 734 patrons into a pending user bucket and it only imported 122 of them. (And confirmed when I tried transferring those 122 to a bucket, it only transferred 100 of them and the other 22 disappeared.)

Confusingly, the ones that it imported weren't the first chunk or the last chunk of the CSV file. They turned out to be the first if they were sorted by barcode (which the CSV was not).

[Benjamin Murphy (benjamin.murphy)]

I've confirmed that this is still an issue in 3.1.6

[Heather Lindskold (heatherlindskold)]

SPARK/PaILS is running 3.1.10 and we are still experiencing this issue.

[Terran McCanna (tmccanna)]

Jennifer, if you apply the patch for https://bugs.launchpad.net/evergreen/+bug/1749970 does it resolve this issue?

[Jennifer Bruch (jbruch)]

Oops, sorry about that! I must have posted on the wrong bug. I meant to
find the one about the different wording used for Patron Buckets, which are
called User Buckets on the Circ Drop Down Menu. That might be an issue with
our documentation, though, so back to the drawing board.

*Jennifer Bruch*
Bethlehem Area Public Library
11 W. Church Street
Bethlehem, PA 18018
610-867-3761 x232

On Mon, Nov 18, 2019 at 1:01 PM Terran McCanna <
<email address hidden>> wrote:

> Jennifer, if you apply the patch for
> https://bugs.launchpad.net/evergreen/+bug/1749970 does it resolve this
> issue?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1754387
>
> Title:
> Web Client: Pending User Buckets for more than 100 users
>
> Status in Evergreen:
> Confirmed
>
> Bug description:
> When uploading a batch file of patrons to the Pending interface of
> User Buckets, if the list that you wish to add is more than 100
> patrons long, you cannot send the entire list to the User Bucket.
> Only the number shown on the first page are sent and the rest are
> autocleared. Since the lists don't load consecutively (consistently),
> and you have to exit and reenter the User Bucket interface to reload
> the same list, batch loading is limited to a maximum of 100 patron
> accounts at a time.
>
> If the whole list (however many pages there are) cannot be selected
> and moved to the bucket at once, then only the entries that have
> already been added to the bucket should autopurge from the Pending
> interface.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/evergreen/+bug/1754387/+subscriptions
>

[Jennifer Bruch (jbruch)]

Yes, Terran. Thanks, I posted it to the wrong bug. Please ignore my post;)

[Terran McCanna (tmccanna)]

This bug is resolved after applying the fix for https://bugs.launchpad.net/evergreen/+bug/1749970

[Terran McCanna (tmccanna)]

My mistake - it lets you upload more than a hundred to pending and page through them, but it will still only transfer the ones you have marked to the bucket, and the rest disappear off the pending screen.

To duplicate problem:
1. Create a text file of more than 100 patron barcodes
2. Go to Circulation > user Buckets
3. Click on Bucket View tab and select or create a patron bucket
4. Click on Pending Users tab and browse for text file
5. Select all and click Actions > Add to Bucket
6. Switch to Bucket View and verify that the entire bucket did not transfer

[Josh Stompro (u-launchpad-stompro-org)]

We just ran into this, a co-worker was trying to batch edit the 5000 users that had accounts that expired in 2016, and was going to load them into a user bucket and batch edit them. But since only 100 can be moved to a bucket at a time, it doesn't seem possible to use the user buckets like that.

EG 2.3.4

It looks like the whole pending users list is cleared any time any users are moved to a bucket.

$scope.resetPendingList() gets called after all pending users are copied to a bucket.

https://git.evergreen-ils.org/?p=Evergreen.git;a=blob;f=Open-ILS/web/js/ui/default/staff/circ/patron/bucket/app.js;hb=HEAD#l107

So that is why you can only load the first 100.

So one fix would be to only remove the pending users that were loaded into a bucket from the pending user list, so that the user can go through and move 100 at a time into a bucket. I wonder how feasable it would be to have a function that just removes all the pendingList entries that are currently selected?

The other issue is that the max number of items per page is 100. An easy way around that is to add the 'allowAll' feature to the pending user grid. That would allow up to 10,000 users to be moved to a bucket. We are going to try that to see if we can get the users we need into a bucket.

https://git.evergreen-ils.org/?p=Evergreen.git;a=blob;f=Open-ILS/src/templates/staff/circ/patron/bucket/t_pending.tt2;h=2df627e8fe328a24322581e3bc462e62a678e6ca;hb=HEAD#l28

Josh

[Josh Stompro (u-launchpad-stompro-org)]

Another issue is that loading 5K users by barcode places a high load on the server. The item status file loader is setup to load serially because of this same issue I believe, I think that would be a good change for loading users also.

Josh

[Josh Stompro (u-launchpad-stompro-org)]

I retract what I said in comment 15, the file load feature is quite efficient, it sends all the barcodes in one query. That is why the order of the barcodes in the loaded file gets lost, or at least part of the reason. Postgres returns the results in no particular order. Then the grid sorts them by user id.

Loading them with one query does mean that getting feedback about not found, or invalid user barcodes is harder to get.

I'm not sure if it is the grid fleshing out entries, or moving pending to a bucket that causes a high number of drones to be used on the server.

And since it directly queries actor.card, inactive and non-primary barcodes will load patron records, which is fine with me.

Another issue I noticed, The pending user screen will allow you to move users to the bucket that are already in the bucket, which results in duplicates in the bucket. But the duplicates get deduped when displaying the bucket. If you remove the visible user that has duplicates, then the non-visible entry becomes visible. So if you add one user 4 times to a bucket, you have to remove that user 4 times from the bucket to fully get rid of them.

I have some changes that address the original 2 issues, only being able to view 100 pending records at a time, and only being able to move 100 at a time into a bucket.

Branch coming soon.
Josh

[Josh Stompro (u-launchpad-stompro-org)]

Here is a branch that makes the following changes.

1. Sets the default number of items to 100 and allows choosing the 500,1000,10000 options for the pager. If the user picks a different page size and saves the columns that size will stick for the user.

2. When moving users from pending to a bucket, only removes the moved users from the pending list. This allows the user to move a larger number of users to a bucket from pending, a batch at a time.

Working Branch: user/stompro/lp1754387_user_bucket_pending
https://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/stompro/lp1754387_user_bucket_pending

Testing Plan:

Before Changes:

1. Load a file with 110 patron barcodes in the User Buckets, Pending Users interface.
2. See that the pager only allows a max of 100 for the page size.
3. Move one user to a bucket and note that the pending list gets cleared.

After Changes:

1. Load a file with 110 patron barcodes.
2. See that the pager allows for picking larger page sizes.
3. Move one user to a bucket, and note that the pending list removes just that one user.

Another way to see the changes is to pick a pager size of 25, and select all the entries. Then move them to a bucket and see that you can continue to move batches of 25 until you have moved all the pending users.

Josh

[Gina Monti (gmonti90)]

I, Gina Monti, have tested this code during the Feedback Fest and consenting to sign off. Email: <email address hidden>.

[Galen Charlton (gmc)]

Pushed to master, rel_3_5, and rel_3_4. Thanks, Josh and Gina!