web client: copy templates created on XUL not displayed

[[e-mail]]

Copy templates created via Copy Editor on XUL clients are not listed on the Web client when creating/editing an item.

[Jason Boyer (jboyer)]

There are multiple issues causing this. First, the format of the templates has changed significantly. This isn't a big deal since a one-time conversion is possible and not that difficult. By far the worse issue though, is that currently web client copy templates are stored in the browser / Hatch only. So every time you change workstations (or browser) all of your templates are gone. Copy templates also need to be moved back to database backed actor.usr_settings settings.

[Jason Etheridge (phasefx)]

if we do decide to use a server-side store, I'd advocate for not using actor.usr_setting like we did in the XUL client. I regret implementing it like that; it's not fun to edit when needed and at least once it caused problems due to size (in OpenSRF? jabber? I forget) for book jobbers using the client.

[Jason Etheridge (phasefx)]

also, if I recall correctly, we used labels as keys in the datastructure, which could/would break if switching locales

[Jason Boyer (jboyer)]

It did use the labels as keys, yes. Fortunately a conversion from the old style to the new (uses field names, all stat cats in a sub-hash) is fairly straightforward and has already been written and lightly tested. :) Given recent OpenSRF chunking changes and that the web client is the way forward (and I plan to do a one-time conversion from xul style to web as part of this if I can get it done) any issues with the xul client shouldn't matter anymore.

[Jason Boyer (jboyer)]

A plea for assistance at this late stage in the game. Most of the difficult parts of this are handled in this branch: http://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/collab/jboyer/lp1691269_webstaff_acp_templates EXCEPT that the templates aren't appearing in the interface. The JS debugger shows them being loaded, converted, and assigned to $scope, but later in the initialization the template related fields of $scope are empty, as if there's a scoping issue or something similar. I've banged my head against this for hours and hours and can't find any way around it. I would be happy to finish getting this in shape but I'm apparently out of my depth at the moment.

[Jason Boyer (jboyer)]

Ah, the reason I'm so desperate for assistance: I'm traveling for work for the next week solid and will have almost no time whatsoever to devote to this or anything else.

[Elaine Hardy (ehardy)]

It would be a very big help to PINES catalogers if we could convert our item templates. Many of our libraries have more than 50 templates that would need to be recreated otherwise.

[Jason Boyer (jboyer)]

Not to get anyone's hopes up *too* high on a Friday but I have seen my transfer method work correctly once so far during some downtime at a library migration (and then the downtime ended :-/ ). I feel like it's very close and hope to give it a lot of testing very soon. Also, I realized recently that the branch I pushed when I hollered "HALP" didn't actually have any changes associated with it, so... oops there. I'm hopeful there will something to test in the next few days.

[Jason Boyer (jboyer)]

Sunday, Sunday, Sunday! We'll sell you the whole seat but you'll only need the TEST:
http://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/collab/jboyer/lp1691269_webstaff_acp_templates

To test:
Build some XUL Client templates on at least 2 accounts before loading the copy editor.
Load either the Copy Editor or Template Editor.
Verify that the XUL Templates that you created before are 1, present, and 2, in alpha order.
Switch to another user wih XUL templates, verify that only their templates appear, not the initial set or a combination of the two.
Add, edit, delete templates in both clients, see that they neither interfere or transfer further between them (unless you delete the actor.usr_settings webstaff entry for the user.)

[Elaine Hardy (ehardy)]

Jason,

I am planning on testing this as soon as possible. Thanks!!

[Elaine Hardy (ehardy)]

Chris put this on our test server for the web client. The test was not successful. I have templates so I imported them using the edit volume/copy templates interface. I also created on in the XUL client just in case. The list was there but, as before, nothing happened when I attempted to apply. I pulled up a bib record and, in add copies, was going to test there but the list wasn't even available in the dropdown menu for templates.

I then logged out and back in and the list and the list I had just imported no longer was available in the dropdown menu in the edit vol/copy templates interface.

I reimported the list and this time clicked on save. Then logged back in. The list was available but nothing happened when I tried to apply some, including the newly created one. I then went back to add copies in a bib record. The list was available but again no template was applied.

Elaine

[Jason Boyer (jboyer)]

Hi Elaine, I should have been more specific in my testing notes; The import/export function does not work for this. It works completely silently in the background and if you have any kind of webstaff templates in place it ignores the old templates. To test you may have to export and import your existing templates through the XUL client, and then open the volume / copy editor in the web client for it to transfer them. If you would like to use the same test environment someone with database access will need to delete the webstaff template entries from your account first.

[Jason Boyer (jboyer)]

I just realized that wasn't quite what I wanted to say either. I meant to say that you have to export your existing templates through the XUL client on your normal server, import them through the XUL client on your test server, then load the volume copy editor in the web client on the test server for it to transfer them automatically in the background. This is a slightly more involved patch to test in this way since it's designed to work seamlessly in the background on existing data, it doesn't make any changes to the XUL or web client import or export functions. Someone with database access will have to delete your imported webstaff templates from teh actor.usr_settings table before you can try again.

[Elaine Hardy (ehardy)]

Want to make sure I understand --

   1. Delete the templates from the webclient
   2. Export any existing templates from the XUL client
   3. Import those
   ​ templates​
   into the 3.0 XUL client.
   4. Then open them in the web client?

J. Elaine Hardy
PINES & Collaborative Projects Manager
Georgia Public Library Service/PINES
1800 Century Place, Ste. 150
Atlanta, GA 30045

404.235.7128 Office
404.548.4241 Cell
404.235.7201 FAX

On Thu, Oct 26, 2017 at 7:13 AM, Jason Boyer <email address hidden>
wrote:

> Hi Elaine, I should have been more specific in my testing notes; The
> import/export function does not work for this. It works completely
> silently in the background and if you have any kind of webstaff
> templates in place it ignores the old templates. To test you may have to
> export and import your existing templates through the XUL client, and
> then open the volume / copy editor in the web client for it to transfer
> them. If you would like to use the same test environment someone with
> database access will need to delete the webstaff template entries from
> your account first.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1691269
>
> Title:
> web client: copy templates created on XUL not displayed
>
> Status in Evergreen:
> New
>
> Bug description:
> Copy templates created via Copy Editor on XUL clients are not listed
> on the Web client when creating/editing an item.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/evergreen/+bug/1691269/+subscriptions
>

[Jason Boyer (jboyer)]

That's right, except that you can't do step 1 in the web client. Chris or someone with database access will have to delete the webstaff.cat.copy.templates entry from actor.usr_settings before the web client will transfer them.

[Elaine Hardy (ehardy)]

I will try to get this tested tomorrow
​

Thanks!​

J. Elaine Hardy
PINES & Collaborative Projects Manager
Georgia Public Library Service/PINES
1800 Century Place, Ste. 150
Atlanta, GA 30045

404.235.7128 Office
404.548.4241 Cell
404.235.7201 FAX

On Thu, Oct 26, 2017 at 11:47 AM, Jason Boyer <email address hidden>
wrote:

> That's right, except that you can't do step 1 in the web client. Chris
> or someone with database access will have to delete the
> webstaff.cat.copy.templates entry from actor.usr_settings before the web
> client will transfer them.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1691269
>
> Title:
> web client: copy templates created on XUL not displayed
>
> Status in Evergreen:
> New
>
> Bug description:
> Copy templates created via Copy Editor on XUL clients are not listed
> on the Web client when creating/editing an item.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/evergreen/+bug/1691269/+subscriptions
>

[Beth Willis (willis-a)]

Hi Jason,

I tested this patch on NOBLE's test server, but did not find the expected results. Here is what I did:

1. Imported copy templates into the XUL client (3.0.1)
2. Verified that there were no existing copy templates in the Web client
3. Searched for a bib record
4. Selected "Add Volumes"
5. Selected the "Template" drop down menu, but not templates displayed

If there is additional testing that I can do, please let me know.

Thank you.
Beth

[Jason Boyer (jboyer)]

This branch has been updated and force-pushed so it will need pulling down again. Also, it seems that occasionally you may want to reload the page the first time to make them appear.

Also, there is a db upgrade script that's part of this which is easy to miss.

[Jason Boyer (jboyer)]

An updated patch to this was force-pushed during the hackaway. Testing method hasn't changed.

[Jason Boyer (jboyer)]

Note 11/15: I've force-pushed an update for the importer for webstaff templates; it was essentially nonfunctional. I also changed it to append the imported templates rather than discarding your existing templates to match the XUL client importer. IMPORTANT: It still can't import XUL templates this way; you still need to import through whichever client you exported from.

[Jason Etheridge (phasefx)]

Jason, how horrible would it be to split this into two bugs at this point? The conversion for one aspect of it, and the move to server-side storage for the other? I don't know what discussions happened at the hack-a-way around remote storage, but it feels to me like this one bug is doing too much.

[Jason Boyer (jboyer)]

It's possible since the conversion is self-contained in a single function. But since I was at the discussion and heard how early days that project is I'm not so inclined at all. This needs to be in 3.X as soon as possible (ideally 3.0.2 or .3) since catalogers don't give the web client a second thought once they hear that all of their templates are gone (and have to be remade on every machine they use).

Since there's only one other function that needs changed to determine what "remote" means with respect to remote storage in this instance, I think it would be more beneficial for staff to get this in ASAP and then work out the migration path when there's actually somewhere else to put them.

[Elaine Hardy (ehardy)]

I was unaware that the templates need to be remade for each workstation.
That is a major red flag, particularly in conjunction with the general UI
problems and some of the other bugs. Catalogers for libraries with multiple
branches will have in excess of 50 templates. To have to recreate them if
they change workstations is not a good thing at all.

We were able to get a XUL version of 3.0 up and I did a test yesterday and
the templates did not transfer. I will try again as soon as I can. I did
use existing templates and not created ones specifically for the XUL
client.

Elaine

J. Elaine Hardy
PINES & Collaborative Projects Manager
Georgia Public Library Service/PINES
1800 Century Place, Ste. 150
Atlanta, GA 30045

404.235.7128 Office
404.548.4241 Cell
404.235.7201 FAX

On Thu, Nov 16, 2017 at 7:23 PM, Jason Boyer <email address hidden>
wrote:

> It's possible since the conversion is self-contained in a single
> function. But since I was at the discussion and heard how early days
> that project is I'm not so inclined at all. This needs to be in 3.X as
> soon as possible (ideally 3.0.2 or .3) since catalogers don't give the
> web client a second thought once they hear that all of their templates
> are gone (and have to be remade on every machine they use).
>
> Since there's only one other function that needs changed to determine
> what "remote" means with respect to remote storage in this instance, I
> think it would be more beneficial for staff to get this in ASAP and then
> work out the migration path when there's actually somewhere else to put
> them.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1691269
>
> Title:
> web client: copy templates created on XUL not displayed
>
> Status in Evergreen:
> New
>
> Bug description:
> Copy templates created via Copy Editor on XUL clients are not listed
> on the Web client when creating/editing an item.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/evergreen/+bug/1691269/+subscriptions
>

[Elaine Hardy (ehardy)]

Also, going forward, templates need to be fully exportable and importable
from within the web client, without having to maintain a XUL staff client
for every update of the web client.

J. Elaine Hardy
PINES & Collaborative Projects Manager
Georgia Public Library Service/PINES
1800 Century Place, Ste. 150
Atlanta, GA 30045

404.235.7128 Office
404.548.4241 Cell
404.235.7201 FAX

On Thu, Nov 16, 2017 at 7:23 PM, Jason Boyer <email address hidden>
wrote:

> It's possible since the conversion is self-contained in a single
> function. But since I was at the discussion and heard how early days
> that project is I'm not so inclined at all. This needs to be in 3.X as
> soon as possible (ideally 3.0.2 or .3) since catalogers don't give the
> web client a second thought once they hear that all of their templates
> are gone (and have to be remade on every machine they use).
>
> Since there's only one other function that needs changed to determine
> what "remote" means with respect to remote storage in this instance, I
> think it would be more beneficial for staff to get this in ASAP and then
> work out the migration path when there's actually somewhere else to put
> them.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1691269
>
> Title:
> web client: copy templates created on XUL not displayed
>
> Status in Evergreen:
> New
>
> Bug description:
> Copy templates created via Copy Editor on XUL clients are not listed
> on the Web client when creating/editing an item.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/evergreen/+bug/1691269/+subscriptions
>

[Jason Boyer (jboyer)]

Elaine, the problem with having to re-build your templates for every machine is the current state of things in 3.0 that I'm trying to change with this branch. As for importing and exporting, you don't need to do that over the normal course of things; the way it works is that it pulls your existing xul templates, converts them, and saves them as web client templates the first time you load the copy editor. The annoying import / export steps are only necessary because I'm assuming there weren't existing xul templates on your test server.

A more accurate test in the spirit of the patch would be to log in to the xul client on the test server before doing anything else, create a couple copy templates, then log in to the web browser for the first time after having done that, load the volume/copy editor (wait a few seconds) and click the dropdown to see if they appear. I will note that I have seen instances where the page has to be reloaded one or two times at this point before they will appear. It's a one-time only process; once you have web client templates they're not touched again unless you do something to them manually.

[Elaine Hardy (ehardy)]

Just to make sure -- we do need the ability to export and import within the
web client for those times we have to share templates with new people, for
example. Also, it isn't unusual in PINES libraries for one person to create
the templates and then send them to all catalogers and to vendors.

I will try to get back to testing this afternoon. I have to train someone
on OCLC ILL this morning....

J. Elaine Hardy
PINES & Collaborative Projects Manager
Georgia Public Library Service/PINES
1800 Century Place, Ste. 150
Atlanta, GA 30045

404.235.7128 Office
404.548.4241 Cell
404.235.7201 FAX

On Fri, Nov 17, 2017 at 9:29 AM, Jason Boyer <email address hidden>
wrote:

> Elaine, the problem with having to re-build your templates for every
> machine is the current state of things in 3.0 that I'm trying to change
> with this branch. As for importing and exporting, you don't need to do
> that over the normal course of things; the way it works is that it pulls
> your existing xul templates, converts them, and saves them as web client
> templates the first time you load the copy editor. The annoying import /
> export steps are only necessary because I'm assuming there weren't
> existing xul templates on your test server.
>
> A more accurate test in the spirit of the patch would be to log in to
> the xul client on the test server before doing anything else, create a
> couple copy templates, then log in to the web browser for the first time
> after having done that, load the volume/copy editor (wait a few seconds)
> and click the dropdown to see if they appear. I will note that I have
> seen instances where the page has to be reloaded one or two times at
> this point before they will appear. It's a one-time only process; once
> you have web client templates they're not touched again unless you do
> something to them manually.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1691269
>
> Title:
> web client: copy templates created on XUL not displayed
>
> Status in Evergreen:
> New
>
> Bug description:
> Copy templates created via Copy Editor on XUL clients are not listed
> on the Web client when creating/editing an item.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/evergreen/+bug/1691269/+subscriptions
>

[Jason Boyer (jboyer)]

Of course. Importing and exporting work fine; you just can't export from one type of client and import into the other.

[Jason Boyer (jboyer)]

(But let me know if I'm misunderstanding and you're requesting that feature be added)

[Elaine Hardy (ehardy)]

Just wanted to make sure since it would be better to know now than after we
go live. But I will test to make sure!

J. Elaine Hardy
PINES & Collaborative Projects Manager
Georgia Public Library Service/PINES
1800 Century Place, Ste. 150
Atlanta, GA 30045

404.235.7128 Office
404.548.4241 Cell
404.235.7201 FAX

On Fri, Nov 17, 2017 at 10:05 AM, Jason Boyer <email address hidden>
wrote:

> (But let me know if I'm misunderstanding and you're requesting that
> feature be added)
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1691269
>
> Title:
> web client: copy templates created on XUL not displayed
>
> Status in Evergreen:
> New
>
> Bug description:
> Copy templates created via Copy Editor on XUL clients are not listed
> on the Web client when creating/editing an item.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/evergreen/+bug/1691269/+subscriptions
>

[Jason Etheridge (phasefx)]

re: bundling concerns into the ticket

Fair enough; I'll just throttle back my autism over the bug title :)

[Elaine Hardy (ehardy)]

I have tried testing this several times and cannot get the templates to import. The are listed but nothing happens when I try to apply them. See attached screen shots.

[Cesar V (cesardv)]

Successfully tested this using some sample XUL copy templates on 3.0 dev VM. Thanks Jason.
Everything worked in the background converting the XUL templates and persisting them as expected.
Signed off branch below...

Lastly I will say I kinda have the same concern as that Jason Etheridge voiced... but after some pondering I think we can get this fix out, so that XUL templates have somewhere to live server-side, and perhaps afterwards we can come up with a better way to deal with better ways of storing templates and settings, and caching them to local storage when it makes sense.

http://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/cesardv/jboyer_lp1691269_webstaff_acp_templates_signoff

[Jason Stephenson (jstephenson)]

I agree with Cesar's suggestion of finding a better way to store copy templates. Having them in usr_settings always struck me as a bit odd.

Bug 1131238 suggests moving the MARC templates into the database instead of them being stored in the file system and referenced from the config files. Perhaps the copy templates could be moved to a new table as part of any development on that bug with a new permissions/ownership/sharing/visibility scheme for these. As I understand it, the MARC templates for bib records are shared consortium-wide and copy templates are limited to individuals only. It makes sense to use the same method of exposing both MARC and copy templates so that different levels of the hierarchy can have their own local versions and some can be consortium-wide.

[Beth Willis (willis-a)]

Thanks to all who worked on this bug. I did not have the issue that Elaine reported. In the Web client was able to see all templates created in the XUL client and to apply them to the current copy. I did not test importing or exporting the templates.

[Kathy Lussier (klussier)]

I also agree with the suggestion of a better way to store copy templates, but I see that as being a new feature, whereas this code from Jason is a bug fix that is necessary to get people to move to the web client.

I'll reiterate what I said at the hack-a-way, though, in that I would love the future copy templates to be applied at the org unit, workstation, or user level. We did something similar with the old xul client toolbars, and our users appreciated the flexibility that these different levels gave them.

An org unit would be able to provide a common set of templates that would be useful for everyone working at that org unit, with no need to do the laborious importing/exporting that we now have to do. However, if a copy template is only applicable to one workstation or one users, we would have the flexibility to create templates at that level too.

Jason S. - do you have any concerns about merging Jason B's code as is as a bug fix, or do you agree with Jason E. and Cesar that we can merge this as a temporary fix? I'm testing it now and don't want to merge it if there are concerns. Also, I wanted an excuse to refer to all three Jasons in one sentence.

[Jason Stephenson (jstephenson)]

I have zero objection to merging Jason Boyer's code as a bug fix. My comment was aimed at future directions and to remind everyone that there is another bug to do go in a similar direction for the MARC templates so they should share some features if possible.

[Jason Boyer (jboyer)]

I'll go ahead and chime in again to say that I'm all in favor of moving these templates to the proposed storage service; my primary concern was to make these available in the web client now because I have no idea when the new settings storage system will be available and we've been saying for some time that 3.0 is production ready. I'll be happy to be the one to pull these back out of aus and throw them into the new service when it's available. :)

[Kathy Lussier (klussier)]

Thanks for your work on this Jason B! I created templates in the Xul client and was able to retrieve them in the web client. I was also able to apply most of the template to my copies from the edit tab and from the copy templates tab in the web client's volume/copy editor.

I came across one small issue. Because we used the unified editor in the XUL client, our libraries often stored Call Number Classification scheme, Call Number Prefix, and Call Number Suffix data in our copy templates. However, I am unable to apply any of those fields from the copy templates that originated in XUL.

For the XUL-originated copy templates, I see the following in local storage:
{"Reference":{"batch_class_menulist":3,"location":101,"batch_prefix_menulist":1}}

However, if I store this volume-level data in a copy template via the web client, I see the following in local storage:
"WEBREF":{"statcats":{},"callnumber":{"prefix":1,"classification":2},"ref":"t","circ_lib":4,"location":102}}

I only tested with prefix and classification, not with suffix, but I assume the same happens there too.

Everything else works as expected.

[Jason Boyer (jboyer)]

Oh, thanks Kathy. The only call number related field I've noticed in the templates I've tested are the owning_ou which looks like it is grabbed unintentionally when saving templates in a live copy editor vs the template editor itself. I'll find a way to build these locally and get a fix tested asap.

[Galen Charlton (gmc)]

The branch user/gmcharlt/lp1691269_now_with_callnumbers signs off on the previous patches and adds:

  * support for call number prefix, suffix, and classification scheme in converted templates
  * a unit test
  * some minor cleanup

Given the size of the follow-ups, leaving to another committer to review the results before merging.

[Mike Rylander (mrylander)]

I like it. Picked to master and 3.0, improving modernity for posterity!

Thanks, Jason and Galen!