Provide a command to batch-upload merchant fees

[Anthony Lenton]

Along the same lines of update_earnings, sca should provide a command to batch-upload merchant fees. This is needed to fill in the finance report. Eventually this information will be available via Ubuntu Pay's API, but currently it's only available in the monthly vat report.

The Subscription model should get a new "merchant_fee" attribute, to contain this information.

The exact format of the csv should be checked with ISD.

[Michael Nelson (michael.nelson)]

So currently the report that sca imports each day (currently for vat data to calculate earnings) is generated on a daily cron running:

https://pastebin.canonical.com/64827/

Note: It's using the --field=captured option, which means that:
    a) we get a report for all the items which were captured (rather than settled [1]) during the previous day only,
    b) the data won't include the merchant fees or refunds (as these are part of the backend data received after settlement [2]

So, if we also want to import refunds and merchant costs, we'd need to either:
 * do a separate run of vat_report for sca daily on pay using the default settled field so we get the backend data (with refunds and merchant costs), or
 * update the current job to just report on the default settled field, which may mean that the data is not as current as finance need it? (where did the reason come from for needing --field=captured?)

Personally, I think second option is more sound is the better option, as we can just be importing one file with all the data we need, but the delay in online reports may not be an option. Other ideas?

[1] Apparently a payment can be settled between 3-5 days after it is captured for RBS, but it then has other information available, such as merchant fees.
[2] https://bazaar.launchpad.net/~canonical-isd-hackers/canonical-payment-service/trunk/view/head:/src/paymentservice/jobs/monthly/vat_report.py#L117

[Michael Nelson (michael.nelson)]

Anthony pointed out that we've landed changes now that allow us to get the VAT info via the API, so we may indeed be able to switch the vat_report.sh to use the default backend data.

[Michael Nelson (michael.nelson)]

Here's the tail of /srv/admin.isd.canonical.com/isd-logs/rangpur/sca/crontab-logs/process_subscriptions.log

https://pastebin.canonical.com/64889/

Although it doesn't have datestamps, based on the configglue-1.0 import errors, you can see that the first run after the last rollout begins at line 19, in which one purchase already had the earnings calculated.

The next day it's closer to half, and after that it's nearly always more than half (some days all purchases have earnings already). But there are still some being updated by the import of the vat_report (generated with --field=captured).

I'm not sure what to make of that... it seems to mean that we're not receiving some capture notifications (which is possible, since we're not guaranteed to receive them - pay doesn't currently retry on failure for notifications).