Add form and multipart_form helpers to request

[Jamie Lennox]

We have a json() helper on last_request, however for form encoded a=b&c=d data and multipart form encoded data we don't have a helper. Whilst there are less useful with APIs they are a common usage for people using requests.

[Ian Cordasco (icordasc)]

Might I ask why? Is it to ensure that requests encoded it properly? That seems more like someone testing requests than their code imo

[Jamie Lennox (jamielennox)]

Sure, i consider that a major advantage of the library is not just stubbing out responses but being able to analyze what you sent. This is something I know i use a lot.

For example we have a big request() wrapper function that adds certain things, after you call it you can do:

last_request = self.requests_mock.last_request

self.assertEqual(val, last_request.headers[key])
self.assertEqual(dict_data, last_request.json())

the JSON one gets used a lot, and it's not that we don't trust that requests has done the right thing it's mostly that by passing some flag or param to a function i invoke the change i expect to receive in the outputted message.

Because I'm mainly using json apis I haven't really used the:

requests.post(url, data={'a': 'b', 'c': 'd'})

format, which sets data to a=b;c=d. This would seem to be just as useful to people as json() because we don't want to enforce ordering on that string and it's annoying to work with. In the test i was doing i was able to use this to say:

self.assertEqual(['b'], last_request.form()['a'])

rather than parse it myself.

[Ian Cordasco (icordasc)]

That makes sense for libraries that might use other libraries that might add things after the fact. In that same vein, what about a multipart_form helper? This would be useful in the case of

    requests.post(url, data={'a': 'b', 'c': 'd'}, files={'part-name': ('filename', open('file.txt', 'rb'))})

From a requests POV that's also a very popular feature. I recognize, however, that we don't use it often in OpenStack.

[Jamie Lennox (jamielennox)]

Yea, I'd be keen for that. Do you have a suggestion how that would look?

There's cgi.parse_multipart which can do some of this, or somehow use cgi.FieldStorage (not sure how this works yet).

I can see a boolean is_multipart property and a multipart_form that returns a Mapping (probably just a dict) of part-name -> (filename, data, content_type)?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/385740

[Jamie Lennox (jamielennox)]

That was harder than expected, the handlers around multipart forms are really bad. I've put up what i think is reasonable at [1] based on top of the other form patch and some basic rearranging. I'm going to make this bug target both a form and multipart_form helper.

[1] https://review.openstack.org/385740

[Ian Cordasco (icordasc)]

There's actually a much more usable version of this in the requests-toolbelt:

    from requests_toolbelt.multipart import decoder

    md = decoder.MultipartDecoder.from_response(resp)

That said, I realize this is something for the request object so you might do something more like

    content = request.body
    content_type = requests.headers.get('Content-Type')

    md = decoder.MultipartDecoder(content, content_type)
    for part in md.parts:
        print(part.content)
        print(part.text)
        print(part.headers)

[Jamie Lennox (jamielennox)]

Ah, i didn't look there. I wasn't thinking about requests doing this sort of decoding so i was searching for python decoders.

So i completely agree that it's a nicer interface - however i'm not sure about taking on toolbelt as a dependency (at least it doesn't have any further dependencies) for something that hasn't been actually requested from a user yet.

Know any lighter way? Or more have any preference for how to expose the multipart_form response so it doesn't expose this underlying implementation at all?

[Ian Cordasco (icordasc)]

I don't particularly know a "lighter" way, short of just copying that code out of the toolbelt. I'm also not certain how you'd want to expose this frankly. I would guess something akin to a dictionary but I'm not sure how well that would work for requests-mock users.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Jamie Lennox (<email address hidden>) on branch: master
Review: https://review.openstack.org/385740
Reason: Moved to https://github.com/jamielennox/requests-mock/pull/36

[OpenStack Infra (hudson-openstack)]

Change abandoned by Jamie Lennox (<email address hidden>) on branch: master
Review: https://review.openstack.org/385324
Reason: Moved to https://github.com/jamielennox/requests-mock/pull/36