Metaclass exception when accessing headers attribute of a requests response

[Marc Abramowitz]

Given the following program:

```
(capturemock)marca@marca-mac2:~/capturemock/requests$ cat test.py
from capturemock import capturemock

@capturemock("requests")
def do_stuff():
    import requests

    resp = requests.get("http://www.google.com/")
    print(resp)
    print(resp.headers)

do_stuff()
```

```
(capturemock)marca@marca-mac2:~/capturemock/requests$ python test.py
<Response [200]>
CaseInsensitiveDict({'alternate-protocol': '80:quic', 'x-xss-protection': '1; mode=block', 'transfer-encoding': 'chunked', 'set-cookie': 'PREF=ID=7e28a73323b44303:FF=0:TM=1385711443:LM=1385711443:S=PLAZXhxGUX2EjpFr; expires=Sun, 29-Nov-2015 07:50:43 GMT; path=/; domain=.google.com, NID=67=ELE4eygYpLp0amsGQzpsg8OjGHrHuf3EVjzChdR9fjkbgRkHqoK1p68w--nDM8r-CdPKNae5llmovNXuNH_vnphKKOfyf7dfXXSsut7mMhebRTpdN_UbljHzjmIaTaqd; expires=Sat, 31-May-2014 07:50:43 GMT; path=/; domain=.google.com; HttpOnly', 'expires': '-1', 'server': 'gws', 'cache-control': 'private, max-age=0', 'date': 'Fri, 29 Nov 2013 07:50:43 GMT', 'p3p': 'CP="This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for more info."', 'content-type': 'text/html; charset=ISO-8859-1', 'x-frame-options': 'SAMEORIGIN'})
```

works fine but if setting CAPTUREMOCK_MODE=1, it fails:

```
(capturemock)marca@marca-mac2:~/capturemock/requests$ CAPTUREMOCK_MODE=1 python test.py
<Response [200]>
Traceback (most recent call last):
  File "test.py", line 12, in <module>
    do_stuff()
  File "/Users/marca/dev/bzr-repos/capturemock/capturemock/__init__.py", line 210, in wrapped_func
    result = func(*funcargs, **funckw)
  File "test.py", line 9, in do_stuff
    print(resp.headers)
  File "/Users/marca/dev/bzr-repos/capturemock/capturemock/pythonclient.py", line 197, in __getattribute__
    return self.__getattr__(attrname)
  File "/Users/marca/dev/bzr-repos/capturemock/capturemock/pythonclient.py", line 90, in __getattr__
    self.captureMockTarget)
  File "/Users/marca/dev/bzr-repos/capturemock/capturemock/pythontraffic.py", line 530, in getAttribute
    return self.getAndRecordRealAttribute(traffic, proxyTarget, attrName, proxy, fullAttrName)
  File "/Users/marca/dev/bzr-repos/capturemock/capturemock/pythontraffic.py", line 574, in getAndRecordRealAttribute
    return self.transformResponse(traffic, realAttr, proxy)
  File "/Users/marca/dev/bzr-repos/capturemock/capturemock/pythontraffic.py", line 595, in transformResponse
    responseText, transformedResponse = traffic.transformResponse(response, proxy)
  File "/Users/marca/dev/bzr-repos/capturemock/capturemock/pythontraffic.py", line 230, in transformResponse
    transformedResponse = self.transformStructure(wrappedValue, self.insertProxy, proxy)
  File "/Users/marca/dev/bzr-repos/capturemock/capturemock/pythontraffic.py", line 242, in transformStructure
    return transformMethod(result, *args)
  File "/Users/marca/dev/bzr-repos/capturemock/capturemock/pythontraffic.py", line 252, in insertProxy
    return result.createProxy(proxy)
  File "/Users/marca/dev/bzr-repos/capturemock/capturemock/pythontraffic.py", line 97, in createProxy
    return proxy.captureMockCreateInstanceProxy(self.name, self.target, self.classDesc)
  File "/Users/marca/dev/bzr-repos/capturemock/capturemock/pythonclient.py", line 94, in captureMockCreateInstanceProxy
    newClass = self.captureMockNameFinder.makeClass(classDesc)
  File "/Users/marca/dev/bzr-repos/capturemock/capturemock/pythonclient.py", line 36, in makeClass
    return self.defineClass(actualClassName, classDefStr)
  File "/Users/marca/dev/bzr-repos/capturemock/capturemock/pythonclient.py", line 16, in defineClass
    self.defineClassLocally(classDefStr)
  File "/Users/marca/dev/bzr-repos/capturemock/capturemock/pythonclient.py", line 49, in defineClassLocally
    exec(classDefStr, self)
  File "<string>", line 1, in <module>
TypeError: Error when calling the metaclass bases
    metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases
```

[Marc Abramowitz (msabramo)]

Added some debug logging for this problem: https://code.launchpad.net/~msabramo/capturemock/print_classDefStr_on_TypeError

[Marc Abramowitz (msabramo)]

and here's a merge request for the change to add extra debug logging:

https://code.launchpad.net/~msabramo/capturemock/print_classDefStr_on_TypeError/+merge/197425

[Marc Abramowitz (msabramo)]

Here's a merge request to add a failing test that shows the core problem -- capturemock recording doesn't work with a subclass of collections.MutableMapping (possibly because it's an abstract base class):

https://code.launchpad.net/~msabramo/capturemock/tests_add_mutable_mapping/+merge/197996

[Geoff Bache (geoff.bache)]

Thanks for the patches and test. I've merged in the latest one, in combination with my patch from earlier. It's fair to say I'd have done it in a more general and complicated way, but this patch fixes the problem in hand and doesn't really have any drawbacks, so I've committed it - we can worry about the general case when we need to. The mutable_mapping test now works, hopefully you can check if the requests test does too.

[Marc Abramowitz (msabramo)]

That's great that the patches and test were useful!

So I can confirm that our patches fixed the recording of mutable_mapping and it also fixes the recording of use of the requests module, which is great!

Unfortunately, **replay** is still broken.

I have submitted a merge request to add a new TextTest called mutable_mapping/replay:

https://code.launchpad.net/~msabramo/capturemock/tests_add_mutable_mapping_replay/+merge/198497

[Marc Abramowitz (msabramo)]

To make it easier to see the problem without TextTest, here's a command-line demonstration. Once there is a recorded mock and you try to replay it, then the error appears:

```
marca@marca-mac2:~/dev/bzr-repos/capturemock$ cat test_mutable_mapping.py
from capturemock import capturemock

@capturemock('moduletomock')
def test():
    from moduletomock import MyDict

    mutable_mapping = MyDict(animal='dog')

    print('mutable_mapping["animal"] = %r' % mutable_mapping["animal"])

test()
marca@marca-mac2:~/dev/bzr-repos/capturemock$ python test_mutable_mapping.py
mutable_mapping["animal"] = 'dog'
marca@marca-mac2:~/dev/bzr-repos/capturemock$ CAPTUREMOCK_MODE=1 python test_mutable_mapping.py
mutable_mapping["animal"] = 'dog'
marca@marca-mac2:~/dev/bzr-repos/capturemock$ python test_mutable_mapping.py
*** TypeError while trying to exec classDefStr = 'class MyDict(InstanceProxy, _abcoll.MutableMapping) : __metaclass__ = ProxyMetaClass'
Traceback (most recent call last):
  File "test_mutable_mapping.py", line 11, in <module>
    test()
  File "/Users/marca/dev/bzr-repos/capturemock/capturemock/__init__.py", line 210, in wrapped_func
    result = func(*funcargs, **funckw)
  File "test_mutable_mapping.py", line 5, in test
    from moduletomock import MyDict
  File "/Users/marca/dev/bzr-repos/capturemock/capturemock/pythonclient.py", line 103, in __getattr__
    self.captureMockTarget)
  File "/Users/marca/dev/bzr-repos/capturemock/capturemock/pythontraffic.py", line 535, in getAttribute
    return self.getAttributeFromReplay(traffic, proxyTarget, attrName, proxy, fullAttrName)
  File "/Users/marca/dev/bzr-repos/capturemock/capturemock/pythontraffic.py", line 552, in getAttributeFromReplay
    return proxy.captureMockCreateClassProxy(fullAttrName, newTarget, classDesc)
  File "/Users/marca/dev/bzr-repos/capturemock/capturemock/pythonclient.py", line 125, in captureMockCreateClassProxy
    classProxy = self.captureMockMakeClass(classDesc, proxyTarget)
  File "/Users/marca/dev/bzr-repos/capturemock/capturemock/pythonclient.py", line 122, in captureMockMakeClass
    return self.captureMockNameFinder.makeClass(classDesc, metaClassName)
  File "/Users/marca/dev/bzr-repos/capturemock/capturemock/pythonclient.py", line 48, in makeClass
    return self.defineClass(actualClassName, classDefStr)
  File "/Users/marca/dev/bzr-repos/capturemock/capturemock/pythonclient.py", line 17, in defineClass
    self.defineClassLocally(classDefStr)
  File "/Users/marca/dev/bzr-repos/capturemock/capturemock/pythonclient.py", line 61, in defineClassLocally
    exec(classDefStr, self)
  File "<string>", line 1, in <module>
TypeError: Error when calling the metaclass bases
    metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases
```

[Marc Abramowitz (msabramo)]

The problem seems to be that in captureMockMakeClass, proxyTargetClass is None, so it skips the code you just added that checks the metaclass and creates an ABCMetaProxy which subclasses abc.ABCMeta. It instead uses ProxyMetaClass, which doesn't subclass abc.ABCMeta.

[Marc Abramowitz (msabramo)]

Attaching patch with very hacky fix. Undoubtedly, you can come up with something cleaner and more general!

[Marc Abramowitz (msabramo)]

Slightly less hacky way to fix replay problem (but still very hacky)...

[Marc Abramowitz (msabramo)]

The above patch makes 3 TextTest tests fail though because sometimes the subclass string between the parentheses does not have a dot in it.

[Marc Abramowitz (msabramo)]

Better (but still pretty hacky) patch. This one passes all TextTest tests.

[Geoff Bache (geoff.bache)]

I just fixed this. And then I noticed your patches here, which Launchpad hadn't mailed me about for some reason...

I think mine is slightly more generic in that it handles multiple base classes and lets the existing mechanism find and import everything. But I had a rather dense line so I included three of yours to make it more readable.

Thanks for your efforts here, hopefully it will work for real now!

[Marc Abramowitz (msabramo)]

Yep, it appears to work now!

Check this out:

https://github.com/msabramo/capturemock_examples

Thanks for all your help and I'm glad that I could be of assistance!

Cheers,
Marc

On Fri, Dec 13, 2013 at 12:37 PM, Geoff Bache <email address hidden> wrote:

> I just fixed this. And then I noticed your patches here, which Launchpad
> hadn't mailed me about for some reason...
>
> I think mine is slightly more generic in that it handles multiple base
> classes and lets the existing mechanism find and import everything. But
> I had a rather dense line so I included three of yours to make it more
> readable.
>
> Thanks for your efforts here, hopefully it will work for real now!
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1256218
>
> Title:
> Metaclass exception when accessing headers attribute of a requests
> response
>
> Status in CaptureMock:
> Fix Committed
>
> Bug description:
> Given the following program:
>
> ```
> (capturemock)marca@marca-mac2:~/capturemock/requests$ cat test.py
> from capturemock import capturemock
>
> @capturemock("requests")
> def do_stuff():
> import requests
>
> resp = requests.get("http://www.google.com/")
> print(resp)
> print(resp.headers)
>
>
> do_stuff()
> ```
>
> ```
> (capturemock)marca@marca-mac2:~/capturemock/requests$ python test.py
> <Response [200]>
> CaseInsensitiveDict({'alternate-protocol': '80:quic',
> 'x-xss-protection': '1; mode=block', 'transfer-encoding': 'chunked',
> 'set-cookie':
> 'PREF=ID=7e28a73323b44303:FF=0:TM=1385711443:LM=1385711443:S=PLAZXhxGUX2EjpFr;
> expires=Sun, 29-Nov-2015 07:50:43 GMT; path=/; domain=.google.com,
> NID=67=ELE4eygYpLp0amsGQzpsg8OjGHrHuf3EVjzChdR9fjkbgRkHqoK1p68w--nDM8r-CdPKNae5llmovNXuNH_vnphKKOfyf7dfXXSsut7mMhebRTpdN_UbljHzjmIaTaqd;
> expires=Sat, 31-May-2014 07:50:43 GMT; path=/; domain=.google.com;
> HttpOnly', 'expires': '-1', 'server': 'gws', 'cache-control': 'private,
> max-age=0', 'date': 'Fri, 29 Nov 2013 07:50:43 GMT', 'p3p': 'CP="This is
> not a P3P policy! See
> http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657for more info."', 'content-type': 'text/html; charset=ISO-8859-1',
> 'x-frame-options': 'SAMEORIGIN'})
> ```
>
> works fine but if setting CAPTUREMOCK_MODE=1, it fails:
>
> ```
> (capturemock)marca@marca-mac2:~/capturemock/requests$
> CAPTUREMOCK_MODE=1 python test.py
> <Response [200]>
> Traceback (most recent call last):
> File "test.py", line 12, in <module>
> do_stuff()
> File "/Users/marca/dev/bzr-repos/capturemock/capturemock/__init__.py",
> line 210, in wrapped_func
> result = func(*funcargs, **funckw)
> File "test.py", line 9, in do_stuff
> print(resp.headers)
> File
> "/Users/marca/dev/bzr-repos/capturemock/capturemock/pythonclient.py", line
> 197, in __getattribute__
> return self.__getattr__(attrname)
> File
> "/Users/marca/dev/bzr-repos/capturemock/capturemock/pythonclient.py", line
> 90, in __getattr__
> self.captureMockTarget)
> File
> "/Users/marca/dev/bzr-repos/capturemock/capturemock/pythontraffic.py", line
> 53...