Songkick are concerned about our API usage

[Mark Tully]

Songkick contacted me today regarding the scope's use of their API:

Sam Rudge <email address hidden>
to me
11:22

Hi,

We’ve noticed a large number of requests to our API coming from an access key assigned to you. All the requests seem to be originating from a single IP resolving back to a Canonical controlled server.

According to our logs we’re seeing over 1000 requests/second at some times, including thousands of requests to single URLs. For example, over the last 24 hours we’ve seen over 30,000 requests for the URL

https://api.songkick.com/api/3.0/events.json?artist_name=t&apikey=###

There are 10 URLs that have been hit over 10,000 times in the last 24 hours.

From my interpretation of the logs, these requests appear to be some sort of auto-complete functionality using stubs of artist names

91.189.92.52 - - [21/Jul/2015:10:48:38 +0100] "GET /api/3.0/events.json?artist_name=calc&apikey=### HTTP/1.1" 200 83 "-" "Python-urllib/2.7" HTTPS:off -
91.189.92.52 - - [21/Jul/2015:10:48:38 +0100] "GET /api/3.0/events.json?artist_name=chro&apikey=### HTTP/1.1" 200 83 "-" "Python-urllib/2.7" HTTPS:off -
91.189.92.52 - - [21/Jul/2015:10:48:38 +0100] "GET /api/3.0/events.json?artist_name=ak&apikey=### HTTP/1.1" 200 32282 "-" "Python-urllib/2.7" HTTPS:off -
91.189.92.52 - - [21/Jul/2015:10:48:38 +0100] "GET /api/3.0/events.json?artist_name=li&apikey=### HTTP/1.1" 200 83 "-" "Python-urllib/2.7" HTTPS:off -
91.189.92.52 - - [21/Jul/2015:10:48:38 +0100] "GET /api/3.0/events.json?artist_name=mitsub&apikey=### HTTP/1.1" 200 83 "-" "Python-urllib/2.7" HTTPS:off -
91.189.92.52 - - [21/Jul/2015:10:48:38 +0100] "GET /api/3.0/events.json?artist_name=F&apikey=### HTTP/1.1" 200 83 "-" "Python-urllib/2.7" HTTPS:off -
91.189.92.52 - - [21/Jul/2015:10:48:38 +0100] "GET /api/3.0/events.json?artist_name=vnc&apikey=### HTTP/1.1" 200 83 "-" "Python-urllib/2.7" HTTPS:off -
91.189.92.52 - - [21/Jul/2015:10:48:38 +0100] "GET /api/3.0/events.json?artist_name=vn&apikey=### HTTP/1.1" 200 83 "-" "Python-urllib/2.7" HTTPS:off -
91.189.92.52 - - [21/Jul/2015:10:48:38 +0100] "GET /api/3.0/events.json?artist_name=shoot&apikey=### HTTP/1.1" 200 83 "-" "Python-urllib/2.7" HTTPS:off -
91.189.92.52 - - [21/Jul/2015:10:48:38 +0100] "GET /api/3.0/events.json?artist_name=stea&apikey=### HTTP/1.1" 200 83 "-" "Python-urllib/2.7" HTTPS:off -
91.189.92.52 - - [21/Jul/2015:10:48:38 +0100] "GET /api/3.0/events.json?artist_name=calc&apikey=### HTTP/1.1" 200 83 "-" "Python-urllib/2.7" HTTPS:off -
91.189.92.52 - - [21/Jul/2015:10:48:38 +0100] "GET /api/3.0/events.json?artist_name=s&apikey=### HTTP/1.1" 200 83 "-" "Python-urllib/2.7" HTTPS:off -
91.189.92.52 - - [21/Jul/2015:10:48:39 +0100] "GET /api/3.0/events.json?artist_name=FAK&apikey=### HTTP/1.1" 200 83 "-" "Python-urllib/2.7" HTTPS:off -
91.189.92.52 - - [21/Jul/2015:10:48:39 +0100] "GET /api/3.0/events.json?artist_name=post&apikey=### HTTP/1.1" 200 83 "-" "Python-urllib/2.7" HTTPS:off -
91.189.92.52 - - [21/Jul/2015:10:48:39 +0100] "GET /api/3.0/events.json?artist_name=libre&apikey=### HTTP/1.1" 200 83 "-" "Python-urllib/2.7" HTTPS:off -

But it’s requesting one, two and three character names which probably return a lot of mostly useless results.

Would you be able to investigate improving this behaviour, I’d suggest adding caching to these requests if possible, they could safely be cached for a few hours. Also you could potentially only send requests for the auto-complete when the artist name reaches a certain length, maybe 3 or 4 characters.

Please let us know if we can assist with your implementation, however, unfortunately, if the app continues to use the API this way, we might have to block it or rate limit it to prevent degradation of service to other users.

-Sam

[David Callé (davidc3)]

Thanks for the heads up Mark, the issue is being addressed by severely limiting songkick "smart" querying from the Dash.

Nevertheless, these queries given as examples are odd, with "mitsub" and "FAK" (FAKE?) being in the top 10. They hint at automated testing of the Dash. After reaching to devs in charge of desktop testing, smartscopes server and Unity7 maintenance, it doesn't seem to come from them.

[Bret Barker (noise)]

We've added some basic logic to skip dash queries less than 4 chars and that are prefixes of common commands. This was released around midnight UTC this morning, 2015-07-27.

[Michi Henning (michihenning)]

Marking as invalid for media scope.