Import of for-purchase apps broken

[Michael Nelson]

jpugh was alerted that recently published for-purchase apps aren't showing up in apps.ubuntu.com.

Checking the log shows the import-for-purchase-data processing erroring with "IntegrityError: duplicate key value violates unique constraint "webcatalog_department_slug_uniq" [1]

[1] https://pastebin.canonical.com/89590/

[Michael Nelson (michael.nelson)]

The reason is that apps.u.c currently has a department like:

Name: Role-Playing, slug: role-playing

while the for-pay data being imported has:
Name:Role Playing, slug:role-playing

UWC looks for a department based on the name (rather than the slug) and if not found tries to create a new department with the name and slug.

Updating UWC to instead look for an existing dept. based on the slug and update the name if necessary.

[Michael Nelson (michael.nelson)]

To get apps.u.c updating again before this fix is deployed, someone with /admin/ access should be able to:

1) Go to /admin/webcatalog/department/ and click on "Role-Playing"
2) Update the department name to "Role Playing" (ie. s/-/ /) and save.

This will enable the importer to work currently (as long as the name doesn't change again) until the fix lands (which will enable apps.u.c to simply update to whatever name the sca available apps exports).

[Michael Nelson (michael.nelson)]

Sorry, the other way around - apps.u.com currently has the dept. as "Role Playing":

https://apps.ubuntu.com/cat/department/role-playing/

while sca exports "Role-Playing":

http://software-center.ubuntu.com/api/2.0/applications/en/ubuntu/any/any/

So in the apps.u.com admin, need to change "Role Playing" -> "Role-Playing".

[Michael Nelson (michael.nelson)]

I checked the logs this morning, and the error is still there... so a closer look at the data. I'd checked what's currently in the production apps.u.com departments [1] with what I could see published on sca [2] and found the different use of "Role-Playing" vs "Role Playing". What I didn't notice is that sca currently publishes different apps using *both* versions of role playing [3]. Most apps are published with "Role-Playing", but two have "Role Playing" - magicaldiary and spirited-heart (you can see this by looking at [2] and ctrl-f for 'Role Playing').

If apps.ubuntu.com is to update before the attached branch is deployed, we'll need to republish those two apps with the standard "Role-Playing" department (you may want to delete the other "Role Playing" to avoid confusion, although once the branch is deployed the name variants wont both the import on apps.ubuntu.com).

[1] https://pastebin.canonical.com/89727/
[2] http://software-center.ubuntu.com/api/2.0/applications/en/ubuntu/any/any/
[3] Grab [2], iterate to get unique departments and order https://pastebin.canonical.com/89953/

[Martin Albisetti (beuno)]

I don't think I know what to do. I don't see "Role Playing" anywhere. Not in SCA or apps.u.c admins, nor in the application's.
I don't know where else to look!

[Michael Nelson (michael.nelson)]

Martin's irc comments (that both apps seemed to already have Role-Playing selected) enabled me to resolve the sca issue - both apps just needed to be republished (which I could do by simply approving the "Role Playing"->"Role-Playing" change which was already highlighted on the changerequest page [1]. Clicking on "Approve changes" for both apps automatically re-published the apps so you no longer see the "Role Playing" department on sca's app list.

There is a dashboard achuni setup to request manual runs of the import scripts [2] without bothering webops, but it doesn't seem to be working - at least, each time I've requested a manual run of the import, it remains pending for A Long Time [3]. So hloeung ran it manually. I now see the missing apps that I was aware of [4]

[1] There are no longer any requested changes to approve and so there's no-longer a link to it from the app details page, but it was at https://myapps.developer.ubuntu.com/dev/apps/67/changerequest/
[2] https://apps.ubuntu.com/cat/tasks/
[3] https://apps.ubuntu.com/cat/task_status/ae17f316-f640-4197-ba04-73f588abee1d/
[4] eg https://apps.ubuntu.com/cat/applications/gravi/ https://apps.ubuntu.com/cat/applications/zeegaree-lite/