wishlist: Simple Reporter

[Andrea Neiman]

This work is sponsored by CW MARS and Missouri Evergreen and development is by Equinox Open Library Initiative.

Equinox is creating a second reports interface, known as the Simple Reporter, which will offer frontline staff a guided interface to create certain common types of Evergreen reports. It is in no way intended to replace the main Reports interface, but rather give users a more approachable way to build reports.

Work will proceed according to this specification:
https://yeti.equinoxoli.org/dev/public/techspecs/simple_reporter_202108.pdf

A public branch will be posted once partner testing is complete.

[Jason Boyer (jboyer)]

Have I got a branch for everyone, and nearly 4 days early! The branch implementing Simple Reports is available here: https://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/collab/jboyer/simple-reporter / working/collab/jboyer/simple-reporter

Note that there is a separate commit just to modify the IDL to make it validate according to the XSD. While there are minor changes to a number of locations they are primarily just field name or datatype typos and there shouldn't be any substantive changes in that commit, but it has been separated from other IDL changes just in case.

Some testing notes:
Pre-patch: It is hard for many staff to write a successful report. This is frustrating.
Apply patch
Post-patch: It is easier for more staff to write reports. This is great.

Slightly more in-depth post-patchery procedure:

Choose Simple Reporter in the Administration Menu next to the regular Reporter

Choose which of the 5 types of reports available depending on the subject that you want to report on (Circulation, Collections, Weeding, Patrons, and Billing and Payment Transactions Summary) and give the report a name.

Select the desired fields to display for your report in the Display Fields tab

*New* Select the order and direction that you would like such fields to be sorted in Output Order

Choose fields to filter by and their values in the Filters tab

Select preferred output types and set the schedule for the report in Output Options

Click Save and Schedule Report to save the report and schedule it to run.

Results are available in the My Outputs tab.

Things to note when testing:
Reports and schedules can be edited at any time.
Fields can recommend transforms and operators when selected.
Reporting sources can be setup to restrict output to various permissions. Of these 5 report types only Patrons currently implements this, limiting output to those patrons visible according to the staff user's VIEW_USER permission. Take note of your user permissions if testing this specifically.

[Andrea Neiman (aneiman)]

Docs draft here to facilitate testing:

https://docs.google.com/document/d/1cmm3ber2f8H6H8rXrT61YCqUP1isEaer54lRM-1HiGw/edit?usp=sharing

[Lindsay Stratton (lstratton)]

Working through simple reports, with and without using the draft documentation. There appears to be some quirky behavior regarding report names, output names, and the browser tab labels when viewing HTML output.

Leaving the report name as "Whatever Report (clone)" when initially cloning a report, and later changing the report name in the report editor, i.e. "New Report Name", the output can retain the "Whatever Report (clone)" name in the HTML view browser tab.

Similarly, editing a report name may not change the name in the HTML output display tab. Both the original and the renamed output display the new name in the list, but retain the original report name in the browser tab.

This seems to relate to where and when the cloned report or edited report is renamed and saved, and output run - saving the report name using the Save button next to the report name field (names and labels match up), vs. changing the name and then setting output options and clicking Save and Schedule Report (names and labels do not match).

In either case, the reports all run correctly. May need a little more specificity in the docs to prevent confusion?

[Llewellyn Marshall (lbmarshallv)]

loving it overall. I'm a big fan of being able to change the sort order. My biggest suggestion would be to display the data type next to the display field's name. Coming from the classic reporter, I was expecting the "shelving location" on a circulation report to be referring to the ID. Having it be the shelving location's name is definitely more intuitive, but could use some documentation.

[Ruth Frasur Davis (redavis)]

I have tested this code and consent to signing off on it with my name, rfrasur and my email address, <email address hidden>.

Also, this is pretty cool. It definitely takes a minute to context switch from class reporter. One little note - the output links look like buttons but aren't. This was a little disconcerting, but not a big deal.

[Elizabeth Davis (elidavis)]

Overall, this looks great. It does take a minute to shift how you approach it if you are used to the classic reporter.

I did have two questions:
1. The filter options for Item Deleted?, Item Holdable?, Item Circulate? are yes, no, don't care. The "don't care" throws me off. Does that mean it will give you both?

2.How does it handle relative dates? We have a few libraries that like to have a scheduled report of the library's circulation the day prior. I see options for Month ago and Quarter ago in the transform, but not Day Ago.

[Jason Boyer (jboyer)]

Replies!

Lindsay:
With respect to the naming, because SR is built on the back of the existing reporter backend (meaning it still has templates, reports, and outputs, just different looking) it's possible for the report name and output names to drift if you re-run reports or change their names after the fact. A note explaining some of this can be added to the docs though.

Llewellyn:
SR tries to be as friendly as possible toward people that don't know the database schema and will in some instances intentionally not tell you exactly what's happening. For instance, if you enable the "Shelving Location" field in Display it will be the acpl.name field, but if you use it as a Filter then it will be the acpl.id field (so you can use the shelving location picker). This may also be something called out more in the docs.

Elizabeth:
1. Don't Care does indeed mean true or false. It's there to make it easier to clone reports and make minimal changes or simply re-run the same report with the ability to stop filtering on that particular boolean if desired.
2: A lot of the dates have been exploded into many fields. Some are pre-transformed into the year only, year + month, dates, and X months and years ago, etc. The fields that have Date/Time in the name though are the "raw" datetime fields in the database with all of the reporter transforms available to them, like Age. For Days Ago you can select a Date/Time field, change the transform to Age, and then you have the option of choosing hours, days, weeks, months, and years.

Additionally, the "X months ago" fields have been rounded to integers so the Display won't have things like "3 months 1 day 6 hours 15 minutes 10 seconds" that you might end up with when using the Age transform there.

[Jason Boyer (jboyer)]

Oh, and as a followup to the exploded dates; Extending and changing SR was intentionally made fairly easy provided you're /very/ comfortable with SQL and fairly comfortable editing the IDL. So a pre-transformed Days Ago field could be added locally if there is need.

I also expect that once this sees use in more places that some new patterns will emerge and there will almost certainly be followup LPs to add / change fields. The best thing about SR reports is that if the IDL definition changes they don't have to be re-written from scratch like existing IDL-based reports!

One good followup bug for those institutions with a lot of local SQL / IDL knowledge would be auto-generating the list of report types from the IDL so you could simply add a new SR IDL class, restart the opensrf.settings service and have it appear in the interface. (Currently the 5 existing report types are hard-coded in the Angular client.)

[Andrea Neiman (aneiman)]

Confirming that as author of said docs, I will update them to include additional info, once the branch is committed.

[Elaine Hardy (ehardy)]

Consider changing Don't Care? to Both, since don't care in contexts outside of Evergreen usually means random results and can be confusing.

[Elizabeth Davis (elidavis)]

+1 to Elaine's comment.

Thank you for the clarification on the dates.

[Terran McCanna (tmccanna)]

+1 to using "Both" instead of "Don't Care"

[Jason Boyer (jboyer)]

Good point re: "Don't Care" but rather than changing it to Both I've updated the branch to say Either. At first glance "Both" reads more like "true and false on the same field" which isn't possible, where "Either" is pretty clear that either value is fine. You may need a shift + reload to see that change.

[Elaine Hardy (ehardy)]

Aren't you getting both, though? Either implies that the reported will randomly decide which value to use.

[Elizabeth Davis (elidavis)]

+1 Either does imply it could be random.

[Joan Kranich (jkranich)]

Would "All" work?

[Terran McCanna (tmccanna)]

+1 to "All"

[Jason Boyer (jboyer)]

My feelings about it aren't that strong; Both it is!

[Jason Boyer (jboyer)]

ALSO, if there are strong feelings re: that option the committer can weigh the options when committing it.

[Mike Rylander (mrylander)]

Oh, happy day, everyone! I've merged this to master for 3.9.

Commit-time changes include:
 * Several conflicts, all relatively simple, including permission ID (fixed in baseline and upgrade script)
 * Squashed last two cleanup commits into one, reflecting the final consensus but eliding the churn
 * Signoffs from Ruth and me

Thanks, Jason, and to all testers and feedback providers!