Python-boto crashes with Duplicity and Amazon S3

[Andrew Simpson]

Binary package hint: python-boto

Using Duplicity to back up to Amazon S3 with Karmic. Having been using this back up method for long time (more than one year), but has suddenly stopped working in the last few days.

Can verify the problem on another machine with clean install of Karmic. A Debian Stable machine on same network works with Duplicity + Amazon S3, ruling out network problems.

Installing latest Duplicity from source made no difference.

Installing newer python-boto from Lucid (Python-boto-1.9b-1ubuntu3-all.deb) fixes the problem.

Output from Duplicity

-------------------

asimpson@starfish:~$ ./duplicity-amazon.sh
/usr/lib/pymodules/python2.6/boto/connection.py:315: DeprecationWarning: socket.ssl() is deprecated. Use ssl.wrap_socket() instead.
  sslSock = socket.ssl(sock, None, None)
/usr/lib/pymodules/python2.6/boto/connection.py:318: DeprecationWarning: FakeSocket is deprecated, and won't be in 3.x. Use the result of ssl.wrap_socket() directly instead.
  h.sock = httplib.FakeSocket(sock, sslSock)
Reading globbing filelist /home/asimpson/.backup-data/include_file_amazon.txt
Main action: inc
Args: /usr/bin/duplicity -v5 --encrypt-key xxxxxxxx --sign-key xxxxxxxxx --full-if-older-than 1M --include-globbing-filelist /home/asimpson/.backup-data/include_file_amazon.txt --exclude ** --allow-source-mismatch / s3+http://andrews-xxxxxxxxxx
/usr/bin/python 2.6.4 (r264:75706, Dec 7 2009, 18:45:15)
[GCC 4.4.1]
duplicity 0.5.18 (May 20, 2009)
Linux starfish 2.6.31-19-generic #56-Ubuntu SMP Thu Jan 28 01:26:53 UTC 2010 i686
Using temporary directory /tmp/duplicity-TkwpPs-tempdir
Temp has 33609723904 available, backup will use approx 34078720.
Last full backup date: none
Last full backup is too old, forcing full backup
AsyncScheduler: instantiating at concurrency 0
A .
A home
A home/asimpson
A home/asimpson/Documents
A home/asimpson/Documents/Christmas letter 2008.odt
A home/asimpson/Documents/Rachel maths.ods
A home/asimpson/Documents/affordable holidays refund request.odt
Error accessing possibly locked file /lost+found
AsyncScheduler: running task synchronously (asynchronicity disabled)
Traceback (most recent call last):
  File "/usr/bin/duplicity", line 825, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 818, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 792, in main
    full_backup(col_stats)
  File "/usr/bin/duplicity", line 336, in full_backup
    bytes_written = write_multivol("full", tarblock_iter, globals.backend)
  File "/usr/bin/duplicity", line 245, in write_multivol
    (tdp, dest_filename)))
  File "/usr/lib/python2.6/dist-packages/duplicity/asyncscheduler.py", line 148, in schedule_task
    return self.__run_synchronously(fn, params)
  File "/usr/lib/python2.6/dist-packages/duplicity/asyncscheduler.py", line 174, in __run_synchronously
    ret = fn(*params)
  File "/usr/bin/duplicity", line 244, in <lambda>
    async_waiters.append(io_scheduler.schedule_task(lambda tdp, dest_filename: put(tdp, dest_filename),
  File "/usr/bin/duplicity", line 199, in put
    backend.put(tdp, dest_filename)
  File "/usr/lib/python2.6/dist-packages/duplicity/backends/botobackend.py", line 152, in put
    self.bucket = self.conn.create_bucket(self.bucket_name)
  File "/usr/lib/pymodules/python2.6/boto/s3/connection.py", line 277, in create_bucket
    data=data)
  File "/usr/lib/pymodules/python2.6/boto/s3/connection.py", line 305, in make_request
    data, host, auth_path, sender)
  File "/usr/lib/pymodules/python2.6/boto/connection.py", line 419, in make_request
    return self._mexe(method, path, data, headers, host, sender)
  File "/usr/lib/pymodules/python2.6/boto/connection.py", line 348, in _mexe
    connection = self.get_http_connection(host, self.is_secure)
  File "/usr/lib/pymodules/python2.6/boto/connection.py", line 258, in get_http_connection
    return self.refresh_http_connection(host, is_secure)
  File "/usr/lib/pymodules/python2.6/boto/connection.py", line 281, in refresh_http_connection
    self._cache[cached_name].close()
  File "/usr/lib/python2.6/httplib.py", line 712, in close
    self.sock.close() # close it manually... there may be other refs
AttributeError: close

/usr/lib/pymodules/python2.6/boto/connection.py:315: DeprecationWarning: socket.ssl() is deprecated. Use ssl.wrap_socket() instead.
  sslSock = socket.ssl(sock, None, None)
/usr/lib/pymodules/python2.6/boto/connection.py:318: DeprecationWarning: FakeSocket is deprecated, and won't be in 3.x. Use the result of ssl.wrap_socket() directly instead.
  h.sock = httplib.FakeSocket(sock, sslSock)

[Chuck Short (zulcss)]

I am going to mark this as fixed released then, it might be a good candidate for an SRU.

Regards
chuck