simplestreams

Bug #1240838
Comment #3

Comment 3 for bug 1240838

Revision history for this message

Scott Moser (smoser) wrote on 2013-10-17: Re: simplestreams is several orders of magnitude slower at downloading than wget

the "order of magnitude" is very incorrect.

### wget ###
$ time wget http://maas.ubuntu.com/images/ephemeral/daily/precise/20131017/precise-daily-maas-amd64.tar.gz -q
real 0m5.888s
user 0m0.165s
sys 0m1.720s

### python2 with no local proxy ###
$ rm -Rf out.d; time python2 /usr/bin/sstream-mirror --max=1 -vvv "$INDEX_URL" out.d release=precise arch=amd64
<snip>
inserting http://maas.ubuntu.com/images/ephemeral/daily/precise/20131017/precise-daily-maas-amd64.tar.gz to precise/20131017/precise-daily-maas-amd64.tar.gz

real 0m8.092s
user 0m3.852s
sys 0m1.694s

### python3 with local proxy ###
$ rm -Rf out.d; time http_proxy=http://localhost:3128/ python3 /usr/bin/sstream-mirror --max=1 -vvv "$INDEX_URL" out.d release=precise arch=amd64
<snip>
inserting http://maas.ubuntu.com/images/ephemeral/daily/precise/20131017/precise-daily-maas-amd64.tar.gz to precise/20131017/precise-daily-maas-amd64.tar.gz

real 0m6.689s
user 0m3.008s
sys 0m1.160s

### python3 with no local proxy ###
$ rm -Rf out.d; time python3 /usr/bin/sstream-mirror --max=1 -vvv "$INDEX_URL" out.d release=precise arch=amd64
<snip>
inserting http://maas.ubuntu.com/images/ephemeral/daily/precise/20131017/precise-daily-maas-amd64.tar.gz to precise/20131017/precise-daily-maas-amd64.tar.gz

real 0m7.637s
user 0m3.774s
sys 0m1.533s

Additionally, for sanity's sake:
$ cat go.py
#!/usr/bin/python
BUFLEN = 1024 * 10

import urllib2, sys
if len(sys.argv) > 3:
BUFLEN = int(sys.argv[3])

rfp = urllib2.urlopen(sys.argv[1])
with open(sys.argv[2], "wb") as wfp:
   while True:
      buf = rfp.read(BUFLEN)
      wfp.write(buf)
      if len(buf) != BUFLEN:
          break
rfp.close()

$ time ./go.py http://maas.ubuntu.com/images/ephemeral/daily/precise/20131017/precise-daily-maas-amd64.tar.gz out.img

real 0m6.011s
user 0m1.033s
sys 0m1.762s

Very unscientific, but clearly there is no order of magnitude problem in the downloading itself. Note the simplstreams code spends more 'user' time than wget, that is to be expected in python code execution compared to C. Also, simplestreams downloading does not seem significantly different to simple urllib2 example above.

the "order of magnitude" is very incorrect.

### wget ###
$ time wget http://maas.ubuntu.com/images/ephemeral/daily/precise/20131017/precise-daily-maas-amd64.tar.gz -q
real    0m5.888s
user    0m0.165s
sys     0m1.720s

real    0m8.092s
user    0m3.852s
sys     0m1.694s

real    0m6.689s
user    0m3.008s
sys     0m1.160s

real    0m7.637s
user    0m3.774s
sys     0m1.533s

Additionally, for sanity's sake:
$ cat go.py
#!/usr/bin/python
BUFLEN = 1024 * 10

import urllib2, sys
if len(sys.argv) > 3:
   BUFLEN = int(sys.argv[3])

rfp = urllib2.urlopen(sys.argv[1])
with open(sys.argv[2], "wb") as wfp:
   while True:
      buf = rfp.read(BUFLEN)
      wfp.write(buf)
      if len(buf) != BUFLEN:
          break
rfp.close()

$ time ./go.py http://maas.ubuntu.com/images/ephemeral/daily/precise/20131017/precise-daily-maas-amd64.tar.gz out.img

real    0m6.011s
user    0m1.033s
sys     0m1.762s

Very unscientific, but clearly there is no order of magnitude problem in the downloading itself.  Note the simplstreams code spends more 'user' time than wget, that is to be expected in python code execution compared to C.  Also, simplestreams downloading does not seem significantly different to simple urllib2 example above.