can not match on query parameters listed in the requests.get param keyword

[Mary Bucknell]

I would like to be able to set up different mock responses based on the query parameters. The code that I'm testing using the params keyword to pass the query parameter to the requests.get. If I include the query parameters in the string to be matched, the matching does not occur.

It would also be great if the request_history would include the query parameters specified using the keyword param.

[Jamie Lennox (jamielennox)]

Hi, Thanks for the report.

So there is definitely a way to set up responses based on query paramters but at the moment they are included in the url string, however there are some edge cases to it that boil down to the fact that query strings are not matched by default and the last defined matcher has priority.

So working through an example:

with requests_mock.mock() as m:
    m.get("http://site.example.com", text='2')
    m.get("http://site.example.com?a=b", text='1')

    print requests.get('http://site.example.com?a=b').text
    print requests.get('http://site.example.com?a=c').text

Will print as expected 1 then 2, however if i invert the order the mocks are defined in you'll get 2, 2 because the 2 matcher is handling both query strings first. Notice that 2 is triggered even though a=c is not defined anywhere. This can be unfortunate but it made testing urls with queries like filter or sort order commands easier because it typically doesn't significantly change the response from the url in testing.

To compensate for this there is a flag called complete_qs which you can set that will make it so the response will only match if the query string is matched exactly. So changing the example above:

with requests_mock.mock() as m:
    m.get("http://site.example.com?a=b", text='1', complete_qs=True)
    m.get("http://site.example.com", text='2', complete_qs=True)

    print requests.get('http://site.example.com?a=b').text
    print requests.get('http://site.example.com?a=c').text

notice i have added complete_qs AND i have changed the order they are defined in so that the more general matcher now is defined last (has priority) which would have failed above.

With this setup I get '1' and then a NoMockAddress exception because both matchers will only be triggered when the query string is an exact match.

I'm guessing that you are running into some combination of those problems when defining your responses that looks like you are not matching at all.

So i understand the ordering problem is non-intuitive and should probably be better documented, however do you think it would be better to allow defining query strings like get(url, query={'a', ['b']}) or something? that would be a fairly easy thing to support if it was confusing i would just need to look at what interface requests uses for that and emulate it.

Regarding request_history, that information is present i just call it qs, so printing:

    print m.last_request.qs

above would give me:

    {'a': ['b']}

Is that what you are looking for? I don't remember if the `qs` name was something i made up or something standard, are you suggesting we alias it to `keyword`?

Let me know if that helps and suggestions for the interface. I'm pretty sure the functionality you need is there it just might be intuitive enough.

[Mary Bucknell (msbucknell)]

My issue is a little different all though it is good to understand the ordering issue. In the call that I'm mocking, I'm using the params keyword to specify the query parameters. I have been unable to match on these query parameters. In addition, they do not show up in the request history. This is important for my use case as the function being tested fills in the params based on passed in parameters. Below is an example of the function I'm trying to test using the mock_requests:

def retrieve_lookups(code_uri, params={}):
    """
    :param code_uri: string - The part of the url that identifies what kind of information to lookup. Should start with a slash
    :param params: dict - Any query parameters other than the mimeType that should be sent with the lookup
    :return: list of dictionaries representing the json object returned by the code lookup. Return None if
        the information can not be retrieved
    """
    local_params = dict(params)
    local_params['mimeType'] = 'json'
    resp = session.get(app.config['CODES_ENDPOINT'] + code_uri, params=local_params)
    if resp.status_code == 200:
        lookups = resp.json()
    else:
        #TODO: Log error
        lookups = None
    return lookups

[Jamie Lennox (jamielennox)]

How are you looking them up in request history? params={} is a helper that requests provides that constructs a query string, by the time request_mock gets to work with it they will be part of the request so I can't see how that should affect it.

For example the following example:

import requests
import requests_mock

with requests_mock.mock() as m:
    m.get('http://site.example.com?a=b', text='resp')

    resp = requests.get('http://site.example.com', params={'a': 'b', 'c': 'd'})

    print "url:", m.last_request.url
    print "params/qs:", m.last_request.qs
    print "path:", m.last_request.path
    print "query:", m.last_request.query

produces:

url: http://site.example.com/?a=b&c=d
params/qs: {'a': ['b'], 'c': ['d']}
path: /
query: a=b&c=d

(last_request is just an alias for m.request_history[-1]). So i did have to construct my request_mock with the query string like ?a=b to match - and i'm absolutely willing to make it so you can just pass params={} there like you do with requests. But the response still matched and the query string is available in a number of forms in history.

Can you point me to an example you're having trouble with?

[Mary Bucknell (msbucknell)]

Jammie,

Thank you for looking into this. Providing the example above helped me isolate the problem. There were two.

The first is that I was using the string 'mock://wqpfake.com/test_lookup/endpoint for my url. I would get NoMockAddress: No mock address: GET mock://wqpfake.com/test_lookup/endpoint/test raised. When I changed this to use http://wqpfake.com/test_lookup/endpoint I still wasn't matching on my query parameters. It turns out that I was using (and setting in the code I was testing mimeType=json). If I tried to mock using m.get(self.codes_endpoint + '/test?mimeType=json' it wouldn't match. If I used mimetype=json (all lower case), it would. The returned information for the query parameter was in the m.last_request but with the key value in all lowercase.

It may be worth mentioning both of these issues in the documentation. I get the idea for using mock:// from some blog post about testing that I was reading (probably using standard mocks).

I'd like to offer a word of thanks and encouragement. I much prefer using this as a mocking tool for request rather than the standard python mocks which still confound me after several years of use!

[Mary Bucknell (msbucknell)]

I found something that might be bug. If I'm mocking a request where the query string value has uppercase, the m.last_request.query returns lower all lower case. If I change your example from above a little bit, you can see what happens.

import requests
import requests_mock

with requests_mock.mock() as m:
    m.get('http://site.example.com/test', text='resp')

    resp = requests.get('http://site.example.com/test', params={'a': 'BAD'})

    print "url:", m.last_request.url
    print "params/qs:", m.last_request.qs
    print "path:", m.last_request.path
    print "query:", m.last_request.query

results in
url: http://site.example.com/test?a=BAD
params/qs: {'a': ['bad']}
path: /test
query: a=bad

[Jamie Lennox (jamielennox)]

Oh! I definitely should have thought of that bug [1]. Yes, that's one that I unfortunately can't change without a major version bump because I know some people rely on it. That's a pure bug that something that was used for simply case insensitive matching ended up getting exposed to users.

The workaround is documented here [2]. I do need to make this more prominent, maybe on the README in the repo to suggest people set it.

One day i'll make a 2.0 and this will be the default - but i'm expecting something more major to come along to make that worth it.

The documentation does a lot of mentions of mock://. When i first wrote it i expected many more people to use it via the adapter - i don't know why - and when you mount the adapter you have to give it a protocol, which i chose mock://. Really i need to rewrite all that to just use the mock() syntax and http instead i think as almost everyone is using requests-mock to mock out a real http(s) service.

[1] https://bugs.launchpad.net/requests-mock/+bug/1584008
[2] http://requests-mock.readthedocs.io/en/latest/knownissues.html#case-insensitivity