Missing ratings in header

[David Planella]

Comparing https://apps.ubuntu.com/cat/applications/qreator/ with https://apps.ubuntu.com/cat/applications/leds, I've noticed that qreator is missing the ratings on the header.

[Michael Nelson (michael.nelson)]

From the code, it looks like the most likely reason is that Application.ratings_average == 0 for qreator - achuni can check in the admin.

This could be caused by the import-for-purchase-data command [1] not being run. I've triggered a task over an hour ago now [2] and it's still pending. And checking the import-for-purchase-data.log also doesn't give any evidence that it's imported anything since 20120712.

[1] We could now update this to be import-sca-apps - import-for-purchase-data is just a symlink to import-sca-apps.
[2] https://apps.ubuntu.com/cat/task_status/2d47dd92-af8f-4237-b9b5-e119f73ae04b/

[Michael Nelson (michael.nelson)]

Sorry, all of the above should be s/import_for_purchase_data/import_ratings_stats/, but the same info - I don't see any new log entries since 20120803. I've triggered a job to run it manually which is pending:

https://apps.ubuntu.com/cat/task_status/9ee015b5-b7a4-4176-a90d-b7452eddde90/

[Anthony Lenton (elachuni)]

Indeed, qreator seems to have no rating histogram in the DB, and no Wilson score. We need to investigate why these tasks aren't being run on production, it sounds likely that is causing these symptoms.

[Michael Nelson (michael.nelson)]

Switched to rnr_server, as the feed generated by reviews.ubuntu.com [1] actually specifies zero ratings for qreator, even though there is obviously at least one.

[1] https://reviews.ubuntu.com/reviews/api/1.0/review-stats/

[Michael Nelson (michael.nelson)]

We're waiting for bug 881051 to be deployed for rnr so that we can then use the update_stats command, which may resolve this issue without changes (as currently the stats are not updated, and have not been updated for a while, because of bug 881051).

[David Planella (dpm)]

Not sure if it's a private bug or a typo, but when clicking on bug 881051 Launchpad shows up a popup saying that that bug cannot be found.

[Michael Nelson (michael.nelson)]

So it turns out QReator does have a populated entry on the reviews api [1], it's just not the only one.

I've pinged WebOps to find out more about the dups with two SQL queries [2], here's my current pre-imp plan to fix the issue on rnr:

   * Add SIInRepository, SIInOrigin to admin (achuni wasn't able to query these as we didn't display them in the admin)
   * Migrate any dup SoftwareItems (and refs).
   * Add unique constraint on SI.package_name.
   * Sort out why media not working on dev server.
   * If time - replace Review.softwareitem and Review.repository with Review.softwareitem_in_repo.

[1] https://reviews.ubuntu.com/reviews/api/1.0/review-stats/
[2] https://pastebin.canonical.com/73561/

[Michael Nelson (michael.nelson)]

Results of the above SQL queries show only Qreator having this issue. achuni removed the dup manually from the DB, so the attached branch just adds the admin updates as well as the unique constraint.

[Michael Nelson (michael.nelson)]

Great, marking this as fixed. The linked branch will stop it happening in the future also.

[Michael Nelson (michael.nelson)]

The ratings for qreator are again not displaying on apps.u.c [1].

On 2012-09-04 the ratings were displayed, but dpm also noticed that the app's short-description and name seemed to be reversed (which is the normal display for an app from the archive, ie. without extra meta-data from sca) [2].

Extras apps are still displaying with short-description+packagename, and the job that I requested to import_sca_apps is still pending [3], so it may be necessary to run this command manually via a web_op to see the sca-imported details.

Two further details:
 * apps.u.c only imports stats that have changed in the last X days (1, 3, 7), depending when the last time the import of stats was run. Currently qreator's stats are not imported, as they have not changed in the last 7 days. This makes sense of the current app not being updated with stats.
 * I don't yet see how we went from a qreator with stats on 2012-09-04 to one without stats now (this *would* make sense if the current displayed app was the sca imported app, but that isn't the case).

[1] https://apps.ubuntu.com/cat/applications/qreator/
[2] https://bugs.launchpad.net/ubuntu-webcatalog/+bug/1045691 see comment 3
[3] https://apps.ubuntu.com/cat/task_status/be0f7892-38d5-41a8-b2e0-6d5325cf0fbb/

[Michael Nelson (michael.nelson)]

If I run `fab manage:import_sca_apps && fab manage:'import_ratings_stats precise'` locally, everything works fine and the ratings are displayed (see attached).

If I request a ratings import via /tasks/ it completes successfully, and the import logs on /srv/admin.ca.canonical.com/ca-logs/saci/uwc also show successful imports of ratings for each day.

This leaves one possibility that I can think of - as we're only ever importing the ratings data from the past day, we never import this older data, nor do we provide a way to force the import of all data rather than the past days.

The attached branch will both:
 1) Add an option to the /admin/ UI to be able to view and modify when the last import for a distro series happened.
 2) Add an option to the command to allow the full download of stats, and setup the web-task to do this by default (or add an extra task) so that we can initiate this without webops.

[Michael Nelson (michael.nelson)]

I've also confirmed that I can reproduce the exact issue with:

1) `fab manage:import_sca_apps && fab manage:'import_ratings_stats precise'`
2) Verify stats are present for qreator
3) Manually set stats and histogram to zero to create test scenario via admin/console and verify stats summary not displayed
4) Re-run `fab manage:'import_ratings_stats precise` and verify that stats are still missing.
then with the linked branch:
5) Run `fab manage:'import_ratings_stats precise --download-all'` and verify the stats are now present.