accept-language header, api response incorrect

[Stephen Stewart]

Any reference to fr lang in the Accept-Language header results in results being returned in French (if available), despite en being preferred in the header:

sstewart at Kelpie in ~
$ curl -H "Accept-Language: en-GB,en" https://search.apps.ubuntu.com/api/v1/package/mobileotptransitplan.kalon33 | jq '.title'
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
100 3006 100 3006 0 0 12203 0 --:--:-- --:--:-- --:--:-- 12170
"Mobile OTP Transit Planner for Paris"

sstewart at Kelpie in ~
$ curl -H "Accept-Language: en-GB,en;q=0.8,en-US;q=0.6,fr;q=0.4" https://search.apps.ubuntu.com/api/v1/package/mobileotptransitplan.kalon33 | jq '.title'
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
100 2888 100 2888 0 0 13505 0 --:--:-- --:--:-- --:--:-- 13558
"Planificateur de trajets en transports en commun pour Paris"

[James Tait (jamestait)]

After some further investigation, I finally figured out the cause of this:

 * Package metadata is assumed to be English by default.
 * Translations are embedded within the package metadata.
 * When an Accept-Language header is passed by the client:
  * The languages are ordered by preference.
  * The default "English" metadata is used as a starting point.
  * The translations are checked for each of the preferred languages.
  * When a match is found, the translated data is overlaid.

This means that English rarely, maybe never, appears in the translations. Consequently, a client specifying English as its preference and another language as a fall-back will always get the fall-back language if it is present in a package's metadata.

[James Tait (jamestait)]

I've done some experimentation with this in the course of other work, and I think we can just use `copy_to` to copy the untranslated metadata to the English translation fields. I need to verify how well it behaves if there's already an English translation present, if that's even possible in the myapps UI.

[James Tait (jamestait)]

While `copy_to` works fine for the search side of things, it doesn't modify the original source document, which is what we return, so it's not a solution we can use to solve the localised response problem we have here. Instead, we'd need to copy the metadata as entered in myapps into the translatable fields under translations.en, which I think would probably be better done in the source document coming from myapps. I'm going to add a bug task there rather than reassign it for now - we can mark the relevant task invalid depending on which side of the fence we decide to fix this.