Modernize txaws.client to use twisted.web.client.Agent

Bug #924459 reported by Drew Smathers
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
txAWS
In Progress
Medium
Drew Smathers

Bug Description

txaws.client current uses HTTPClientFactory but should use twisted.web.client.Agent. An interface should also be exposed to make the producer pluggable.

Related branches

Revision history for this message
Drew Smathers (djfroofy) wrote :

As part of this, I think the receiver protocol class should be pluggable as well.

Revision history for this message
Duncan McGreggor (oubiwann) wrote :

You had me at "pluggable."

Revision history for this message
Drew Smathers (djfroofy) wrote :

Initial commit made in newclient branch. Still very, very inchoate.

Also any coaching on how to mark bugs etc (maybe a link to relevant documents) I would really appreciate. I've used lp off and on and now I think I like it better than github/bitbucket.

Changed in txaws:
status: New → In Progress
assignee: nobody → Drew Smathers (djfroofy)
milestone: none → 0.3
importance: Undecided → Medium
Revision history for this message
Duncan McGreggor (oubiwann) wrote : Re: [Bug 924459] Re: Modernize txaws.client to use twisted.web.client.Agent

On Mon, Feb 6, 2012 at 11:04 AM, Drew Smathers
<email address hidden> wrote:
>
> Also any coaching on how to mark bugs etc (maybe a link to relevant
> documents) I would really appreciate.

I dont think there's anything in place for that yet...

But I've just created a new blueprint for general EC2 improvements:
  https://blueprints.launchpad.net/txaws/+spec/ec2-improvements

And I've linked your bug to it. Feel free to file others as you see
them, and link them to that blueprint.

> I've used lp off and on and now I
> think I like it better than github/bitbucket.

I actually *really* like LP for managing tasks and bugs (and building
PPAs, etc.) but I have to say, that after using git for about 6 months
now, I have a hard time going back to bzr. git is sooo much faster and
I really like having all my branches maintained in a single
repo/working directory. It really simplifies my coding workflow.

But compared to svn or cvs pain, the bzr-git gap is a mere speck --
quite easy to live with.

Revision history for this message
Duncan McGreggor (oubiwann) wrote :

On Mon, Feb 6, 2012 at 11:04 AM, Drew Smathers
<email address hidden> wrote:
>
> Initial commit made in newclient branch. Still very, very inchoate.

Just branched it and took a look at the diff -- looks like you're off
to a good start!

Revision history for this message
Drew Smathers (djfroofy) wrote :

Awesome. Thanks for the feedback.

Revision history for this message
Tristan Seligmann (mithrandi) wrote :

This is kind of off-topic for this ticket, but bzr has supported a colocated branch workflow (single working tree housing a repository and multiple branches) for a while now.

Revision history for this message
Drew Smathers (djfroofy) wrote :

Tristan, yes off topic ;-) But ... that sounds awesome. And I'll look into this later. Any docs you'd recommend reading on the subject?

Revision history for this message
Tristan Seligmann (mithrandi) wrote :

Look at the bzr-colo plugin and associated docs.

Revision history for this message
Drew Smathers (djfroofy) wrote :

Ok ... so I got a little ahead of myself on the newclient branch and went into extending APIs in s3 to support plugging in a response body receiver factor and, per API call to the client, an IBodyProducer. I think those aspects are better reflected in a separate ticket ... really just this one I think: #767205

I will probably also create a new branch for these lower-level changes (ditch newclient) and cherry pick or merge relevant changes into that new branch ... if I can learn how to do that with bzr (I'm mostly fluent only in git right now).
.

One question: How much do we care about backwards compat?? There are some public members (client, factory ...) on BaseQuery object that could be used (are being used internally in txaws in various places not covered by tests apparently), so this causes me some concern. Trying to retain these members with some compatibility seems horribly messy to me, so I'm conflicted as to how to approach this. So, any suggestions?

Revision history for this message
Drew Smathers (djfroofy) wrote :

Also just to note some functionality that broke which needs to be resolved in addition to other things:

$ ./bin/txaws-discover --key $AWS_ACCESS_KEY_ID --secret $AWS_SECRET_ACCESS_KEY --action DescribeRegions --endpoint ec2.ap-southeast-1.amazonaws.com
Unhandled error in Deferred:
Unhandled Error
Traceback (most recent call last):
  File "/Users/dsmathers/Envs/txaws/lib/python2.6/site-packages/Twisted-11.1.0_r33520-py2.6-macosx-10.3-fat.egg/twisted/web/_newclient.py", line 1034, in _bodyDataFinished_CONNECTED
    self._bodyProtocol.connectionLost(reason)
  File "./txaws/client/base.py", line 116, in connectionLost
    d.callback(self._buffer.getvalue())
  File "/Users/dsmathers/Envs/txaws/lib/python2.6/site-packages/Twisted-11.1.0_r33520-py2.6-macosx-10.3-fat.egg/twisted/internet/defer.py", line 368, in callback
    self._startRunCallbacks(result)
  File "/Users/dsmathers/Envs/txaws/lib/python2.6/site-packages/Twisted-11.1.0_r33520-py2.6-macosx-10.3-fat.egg/twisted/internet/defer.py", line 464, in _startRunCallbacks
    self._runCallbacks()
--- <exception caught here> ---
  File "/Users/dsmathers/Envs/txaws/lib/python2.6/site-packages/Twisted-11.1.0_r33520-py2.6-macosx-10.3-fat.egg/twisted/internet/defer.py", line 551, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "./txaws/client/discover/command.py", line 72, in write_error
    print >> self.output, "URL: %s" % query.client.url
exceptions.AttributeError: 'Query' object has no attribute 'client'

Revision history for this message
Drew Smathers (djfroofy) wrote :

N.B. I've made an additional change since proposing for merge. The StringIOBodyReceiver I think is better generalized as StreamingBodyReceiver which can be useful for streaming to file. Example:

fd = open('data.bin', 'w')
query = BaseQuery(..., receiver_factory=lambda: StreamingBodyReceiver(fd, readback=False)

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Related blueprints