azure backend not working with latest Azure Storage SDK for Python

[Rafal Sosinski]

There is a problem using latest version of duplicity (0.7.12) with latest Azure Storage SDK for Python (2.0.0) for access Azure resources

root@restore:~# duplicity --file-to-restore data/Capability --no-encryption azure://backup05162017 /mnt -v9
Using archive dir: /root/.cache/duplicity/851a65c225c23ac6257e3fb3ed9061aa
Using backup name: 851a65c225c23ac6257e3fb3ed9061aa
GPG binary is gpg, version 1.4.20
Import of duplicity.backends.acdclibackend Succeeded
Import of duplicity.backends.azurebackend Succeeded
Import of duplicity.backends.b2backend Succeeded
Import of duplicity.backends.botobackend Succeeded
Import of duplicity.backends.cfbackend Succeeded
Import of duplicity.backends.copycombackend Succeeded
Import of duplicity.backends.dpbxbackend Failed: No module named dropbox
Import of duplicity.backends.gdocsbackend Succeeded
Import of duplicity.backends.giobackend Succeeded
Import of duplicity.backends.hsibackend Succeeded
Import of duplicity.backends.hubicbackend Succeeded
Import of duplicity.backends.imapbackend Succeeded
Import of duplicity.backends.lftpbackend Succeeded
Import of duplicity.backends.localbackend Succeeded
Import of duplicity.backends.mediafirebackend Succeeded
Import of duplicity.backends.megabackend Succeeded
Import of duplicity.backends.multibackend Succeeded
Import of duplicity.backends.ncftpbackend Succeeded
Import of duplicity.backends.onedrivebackend Succeeded
Import of duplicity.backends.par2backend Succeeded
Import of duplicity.backends.pydrivebackend Succeeded
Import of duplicity.backends.rsyncbackend Succeeded
Import of duplicity.backends.ssh_paramiko_backend Succeeded
Import of duplicity.backends.ssh_pexpect_backend Succeeded
Import of duplicity.backends.swiftbackend Succeeded
Import of duplicity.backends.sxbackend Succeeded
Import of duplicity.backends.tahoebackend Succeeded
Import of duplicity.backends.webdavbackend Succeeded
Using temporary directory /tmp/duplicity-omCmzF-tempdir
Backend error detail: Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1546, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1540, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1375, in main
    action = commandline.ProcessCommandLine(sys.argv[1:])
  File "/usr/lib/python2.7/dist-packages/duplicity/commandline.py", line 1140, in ProcessCommandLine
    backup, local_pathname = set_backend(args[0], args[1])
  File "/usr/lib/python2.7/dist-packages/duplicity/commandline.py", line 1015, in set_backend
    globals.backend = backend.get_backend(bend)
  File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 223, in get_backend
    obj = get_backend_object(url_string)
  File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 209, in get_backend_object
    return factory(pu)
  File "/usr/lib/python2.7/dist-packages/duplicity/backends/azurebackend.py", line 53, in __init__
    Exception: %s""" % str(e))
BackendException: Azure backend requires Microsoft Azure Storage SDK for Python (https://pypi.python.org/pypi/azure-storage/).
Exception: cannot import name BlobService

Required package is already installed

root@restore:~# pip freeze
adal==0.4.5
asn1crypto==0.22.0
azure-common==1.1.6
azure-nspkg==2.0.0
azure-storage==0.34.2
certifi==2017.4.17
cffi==1.10.0
cryptography==1.8.1
duplicity===-version
enum34==1.1.6
futures==3.1.1
idna==2.5
ipaddress==1.0.18
isodate==0.5.4
keyring==10.3.2
lockfile==0.12.2
msrest==0.4.8
oauthlib==2.0.2
packaging==16.8
pathlib2==2.2.1
pexpect==4.0.1
ptyprocess==0.5
pycparser==2.17
pycrypto==2.6.1
PyJWT==1.5.0
pyparsing==2.2.0
python-dateutil==2.6.0
requests==2.14.2
requests-oauthlib==0.8.0
scandir==1.5
SecretStorage==2.3.1
six==1.10.0

There is no error when using old version of azure package (0.11.1)

Problem is on Linux (Debian, Ubuntu, RHEL and CentOS)

[Hélder Pinto (helder-pinto)]

There is no error as well when using old version of azure-storage 0.20.3 (coming into Azure SDK 1.0.3).

[Hélder Pinto (helder-pinto)]

This bug is still present in Duplicity v0.7.13.1

[David Chin (david-w-h-chin)]

Bug is still present in Duplicity 0.7.14.

The reason is that azure-storage has changed.

This is using:
* Python 2.7.13
* azure-storage 0.36.0

Based on the code in duplicity/backends/azurebackend.py, try out in ipython:
In [1]: import azure
In [2]: import azure.storage
In [3]: from azure.storage.blob import BlobService
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
<ipython-input-3-7f43c9cb0958> in <module>()
----> 1 from azure.storage.blob import BlobService
ImportError: cannot import name BlobService

The source of azure-storage/azure/storage/blob/__ini__.py show 3 different classes:
* BlockBlobService
* PageBlobService
* AppendBlobService

This is from the current master branch: https://github.com/Azure/azure-storage-python/blob/master/azure/storage/blob/__init__.py

[David Chin (david-w-h-chin)]

The duplicity trunk 0.8 series does seem to work with the current version of azure-storage

[edso (ed.so)]

David,

tried to comment your merge request, but it was gone then. here my comments.. ede/duply.net

>hey David,
>
>i see you removed some legacy compat code there for older azure sdk versions. do they not work >anymore? can we expect every user on every platform to update to the latest azure sdk version?
>
>..ede/duply.net

[David Chin (david-w-h-chin)]

edso:

Yeah: I withdrew it because I realized it may break for people using the older Azure API, and also that I was working on the 0.7.14 tagged release rather than current dev branch. Also, the 0.8 branch works, so I switched to using that rather than my rushed changes.

[Joshua Smith (jiffypopjr)]

I've been testing with the latest public release of duplicity and it no longer works with 0.20.3 of Azure... getting the following:

Could not create Azure container: HTTPSConnectionPool(host='****.blob.core.windows.net', port=443): Max retries exceeded with url: /mycontainer?restype=container (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f8bce7ad2d0>: Failed to establish a new connection: [Errno -2] Name or service not known',))
Using temporary directory /tmp/duplicity-EuESF1-tempdir

[ediaz (ediaz-pk25)]

This will work with

pip install azure-storage==0.20.0

is because of this

https://stackoverflow.com/questions/35558463/how-to-import-azure-blobservice-in-python#35592905

I don't have time to change the phyton code but with time will be easy.

in my debian I use the version of debian.. because the last version fails in some code of duplicity

Regards!

[ediaz (ediaz-pk25)]

I like to add there is some time of solve on the version of 0.8.0 of duplicity

Regards!