Activity metric score should include lack of badge as a factor in calculation

Bug #1796176 reported by Kathy Lussier on 2018-10-04
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Evergreen
Status tracked in Master
3.1
Undecided
Unassigned
3.2
Undecided
Unassigned
Master
Undecided
Unassigned

Bug Description

Here's the use case.

A system has created three activity metric badges to identify popular materials in their collection: one badge for titles owned by the most org units, one for most current holds, and one for most circs over time.

The video game, Harry Potter Lego, gets a score of five for the most org units badge, but does not get a badge for holds or circs.

The book, Harry Potter & the Deathly Hallows, gets all three badges. It scores 5 on the org units badge, a 1 on most current holds, and a 4 on holds over time.

Clearly, the Harry Potter book is the title that should be getting the highest score. However, the way we calculate the total activity score is to only average the badges that were earned and discounting the badges that were not earned.

Therefore, Harry Potter Lego gets a score of 5 and ends up getting bumped ahead of Harry Potter & the Deathly Hallows, which gets a score of 3.3. Note: I do know the calculation is not just averaging these scores, but I'm saying 'average' for the sake of simplicity.

After using the activity metric for over a year now, we've found that this method of calculating the total score is frequently pushing up materials with less activity above those that are truly popular. It would be better if the system assigned a zero to any badge that did not get earned by the record, so that the final score looked more like:

Lego Harry Potter: 1.6
Harry Potter and the Deathly Hallows: 3.3

Michele Morgan (mmorgan) on 2018-10-24
Changed in evergreen:
assignee: nobody → Michele Morgan (mmorgan)
Michele Morgan (mmorgan) wrote :

Here is a patch that changes the badge score calculation to include all applicable badges for the search when averaging the scores.

http://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/mmorgan/lp1796176_use_all_badges_in_scope_to_calculate_total_score

Current behavior calculates total score by taking the sum of (badge weight * badge score) for all earned badges and dividing that by the sum of badge weight for *all earned badges*

The patch changes the calculation to take the sum of (badge weight * badge score) for all earned badges and dividing that by the sum of badge weight for *all badges in the scope*

Without this patch, for a scenario similar to Kathy's use case above, total scores would be calculated as follows:

Assuming the following badges and weights:

Badge: Circs over time, weight: 1
Badge: Ownership, weight: 2
Badge: Holds over time, weight 3

For Harry Potter Lego, with badge and score:

Ownership:5

The calculation for total score would be:

5*2/2 = 5

For Harry Potter and the Deathly Hallows with badges and scores:

Circs over time:3
Ownership:5
Holds over time:1

The calculation for total score would be:

(3*1+5*2+1*3)/(1+2+3) = 2.7

After applying the patch, the total score for Harry Potter and the Deathly Hallows would still be 2.7, but Harry Potter Lego's score would calculate as follows:

5*2/(1+2+3) = 1.67

So the clearly more popular item now gets the higher total score.

Changed in evergreen:
assignee: Michele Morgan (mmorgan) → nobody
tags: added: pullrequest
Mike Rylander (mrylander) wrote :

Hi Michele,

There's a problem calculating the weighted average that way, because not all records are in the original population for every badge that is in scope. There are record-, location- and copy-related filters that preclude records from even having a chance to earn a give badge, and dinging them for not earning impossible badges will only become more of an issue as new, cool badges are invented. Then there are the static badges, and eventually patron and staff rating badges, that won't require a calculated population, but would ding every unrated record.

From a mathmatical point of view, we'll need to figure out some way of recording original population inclusion for each badge+record combination. And I think we'll further need to create a concept of badge groups, where the members of the group are averaged together separate from other groups to model different facets of popularity (usage vs curation vs "reader attachment", etc). There is the naive was of just recording a 0 for population members that fall outside the thresholds, but there are probably more compact representations that would be faster at search time, especially for the set of unearned badges, which will be the large majority of badge+record combinations.

Remington Steed (rjs7) wrote :

Based on Mike's feedback in #2, I'm removing the pullrequest tag and milestones.

tags: removed: pullrequest
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers