Support for non-default S3 regions

[Luke]

Support for Amazon's S3 eastern data centers would be much appreciated!

The connections are already supported in Boto:
http://docs.pythonboto.org/en/latest/ref/s3.html#module-boto.s3.connection

Please expose these S3 features of Boto so that those of us located in non-US/EU locations can take advantage of the better bandwidth of using buckets that are physically closer.

For example, an option like --s3-region with the following options:
ap-northeast-1
ap-southeast-1
EU
sa-east-1
us-west-1
us-west-2

[Ian Gibbs (realflash-uk)]

Is anyone working on this? I would be willing to fix this - it's really annoying. My personal use case is with the UK leaving the EU, I need to backup to eu-west-2 not the default EU region in order to not accidentally export data across borders.

Is there any preference to how it would be implemented - cmd line argument vs specifying the region on the bucket URI, for example?

[Kenneth Loafman (kenneth-loafman)]

No one is working on it that I know of. We would appreciate a merge
request or a patch set, either one.

On Sun, Oct 29, 2017 at 11:35 PM, Ian Gibbs <email address hidden>
wrote:

> Is anyone working on this? I would be willing to fix this - it's really
> annoying. My personal use case is with the UK leaving the EU, I need to
> backup to eu-west-2 not the default EU region in order to not
> accidentally export data across borders.
>
> Is there any preference to how it would be implemented - cmd line
> argument vs specifying the region on the bucket URI, for example?
>
> ** Summary changed:
>
> - S3 APNortheast and APSoutheast connections
> + Support for non-default S3 regions
>
> --
> You received this bug notification because you are subscribed to
> Duplicity.
> https://bugs.launchpad.net/bugs/1003159
>
> Title:
> Support for non-default S3 regions
>
> Status in Duplicity:
> Confirmed
>
> Bug description:
> Support for Amazon's S3 eastern data centers would be much
> appreciated!
>
> The connections are already supported in Boto:
> http://docs.pythonboto.org/en/latest/ref/s3.html#module-
> boto.s3.connection
>
> Please expose these S3 features of Boto so that those of us located in
> non-US/EU locations can take advantage of the better bandwidth of
> using buckets that are physically closer.
>
> For example, an option like --s3-region with the following options:
> ap-northeast-1
> ap-southeast-1
> EU
> sa-east-1
> us-west-1
> us-west-2
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/duplicity/+bug/1003159/+subscriptions
>

[Ian Gibbs (realflash-uk)]

Wilco. Should I do it against 7 or 8?

On 31 October 2017 at 12:12, Kenneth Loafman <email address hidden> wrote:

> No one is working on it that I know of. We would appreciate a merge
> request or a patch set, either one.
>
>
> On Sun, Oct 29, 2017 at 11:35 PM, Ian Gibbs <email address hidden>
> wrote:
>
> > Is anyone working on this? I would be willing to fix this - it's really
> > annoying. My personal use case is with the UK leaving the EU, I need to
> > backup to eu-west-2 not the default EU region in order to not
> > accidentally export data across borders.
> >
> > Is there any preference to how it would be implemented - cmd line
> > argument vs specifying the region on the bucket URI, for example?
> >
> > ** Summary changed:
> >
> > - S3 APNortheast and APSoutheast connections
> > + Support for non-default S3 regions
> >
> > --
> > You received this bug notification because you are subscribed to
> > Duplicity.
> > https://bugs.launchpad.net/bugs/1003159
> >
> > Title:
> > Support for non-default S3 regions
> >
> > Status in Duplicity:
> > Confirmed
> >
> > Bug description:
> > Support for Amazon's S3 eastern data centers would be much
> > appreciated!
> >
> > The connections are already supported in Boto:
> > http://docs.pythonboto.org/en/latest/ref/s3.html#module-
> > boto.s3.connection
> >
> > Please expose these S3 features of Boto so that those of us located in
> > non-US/EU locations can take advantage of the better bandwidth of
> > using buckets that are physically closer.
> >
> > For example, an option like --s3-region with the following options:
> > ap-northeast-1
> > ap-southeast-1
> > EU
> > sa-east-1
> > us-west-1
> > us-west-2
> >
> > To manage notifications about this bug go to:
> > https://bugs.launchpad.net/duplicity/+bug/1003159/+subscriptions
> >
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1003159
>
> Title:
> Support for non-default S3 regions
>
> Status in Duplicity:
> Confirmed
>
> Bug description:
> Support for Amazon's S3 eastern data centers would be much
> appreciated!
>
> The connections are already supported in Boto:
> http://docs.pythonboto.org/en/latest/ref/s3.html#module-
> boto.s3.connection
>
> Please expose these S3 features of Boto so that those of us located in
> non-US/EU locations can take advantage of the better bandwidth of
> using buckets that are physically closer.
>
> For example, an option like --s3-region with the following options:
> ap-northeast-1
> ap-southeast-1
> EU
> sa-east-1
> us-west-1
> us-west-2
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/duplicity/+bug/1003159/+subscriptions
>

[Kenneth Loafman (kenneth-loafman)]

If possible, I'd like to see duplicity use defaults as is, and an option to
specify any other connection by name if needed, rather than use an option
for each endpoint possible.

Ideally, we should use a duplicity_rc file in the appropriate location, but
that is going to be a lot of work.

On Thu, Nov 2, 2017 at 11:09 AM, Ian Gibbs <email address hidden>
wrote:

> Wilco. Should I do it against 7 or 8?
>
> On 31 October 2017 at 12:12, Kenneth Loafman <email address hidden>
> wrote:
>
> > No one is working on it that I know of. We would appreciate a merge
> > request or a patch set, either one.
> >
> >
> > On Sun, Oct 29, 2017 at 11:35 PM, Ian Gibbs <<email address hidden>
> >
> > wrote:
> >
> > > Is anyone working on this? I would be willing to fix this - it's really
> > > annoying. My personal use case is with the UK leaving the EU, I need to
> > > backup to eu-west-2 not the default EU region in order to not
> > > accidentally export data across borders.
> > >
> > > Is there any preference to how it would be implemented - cmd line
> > > argument vs specifying the region on the bucket URI, for example?
> > >
> > > ** Summary changed:
> > >
> > > - S3 APNortheast and APSoutheast connections
> > > + Support for non-default S3 regions
> > >
> > > --
> > > You received this bug notification because you are subscribed to
> > > Duplicity.
> > > https://bugs.launchpad.net/bugs/1003159
> > >
> > > Title:
> > > Support for non-default S3 regions
> > >
> > > Status in Duplicity:
> > > Confirmed
> > >
> > > Bug description:
> > > Support for Amazon's S3 eastern data centers would be much
> > > appreciated!
> > >
> > > The connections are already supported in Boto:
> > > http://docs.pythonboto.org/en/latest/ref/s3.html#module-
> > > boto.s3.connection
> > >
> > > Please expose these S3 features of Boto so that those of us located
> in
> > > non-US/EU locations can take advantage of the better bandwidth of
> > > using buckets that are physically closer.
> > >
> > > For example, an option like --s3-region with the following options:
> > > ap-northeast-1
> > > ap-southeast-1
> > > EU
> > > sa-east-1
> > > us-west-1
> > > us-west-2
> > >
> > > To manage notifications about this bug go to:
> > > https://bugs.launchpad.net/duplicity/+bug/1003159/+subscriptions
> > >
> >
> > --
> > You received this bug notification because you are subscribed to the bug
> > report.
> > https://bugs.launchpad.net/bugs/1003159
> >
> > Title:
> > Support for non-default S3 regions
> >
> > Status in Duplicity:
> > Confirmed
> >
> > Bug description:
> > Support for Amazon's S3 eastern data centers would be much
> > appreciated!
> >
> > The connections are already supported in Boto:
> > http://docs.pythonboto.org/en/latest/ref/s3.html#module-
> > boto.s3.connection
> >
> > Please expose these S3 features of Boto so that those of us located in
> > non-US/EU locations can take advantage of the better bandwidth of
> > using buckets that are physically closer.
> >
> > For example, an option like --s3-region with the following options:
> > ap-northeast-1
> > ap-southeast-1
> > EU
> > sa-east-1
> > us-west-1
> > u...

[Ian Gibbs (realflash-uk)]

That makes sense as a approach. Boto has a constant for the regions but it
is out of date and they aren't maintaining it any more. With AWS still
expanding it makes sense not to hard code a list into Duplicity -
presumably use of the constant in boto is optional for this reason.

I'll make the changes against the 8 branch.

On 2 Nov 2017 19:46, "Kenneth Loafman" <email address hidden> wrote:

If possible, I'd like to see duplicity use defaults as is, and an option to
specify any other connection by name if needed, rather than use an option
for each endpoint possible.

Ideally, we should use a duplicity_rc file in the appropriate location, but
that is going to be a lot of work.

On Thu, Nov 2, 2017 at 11:09 AM, Ian Gibbs <email address hidden>
wrote:

> Wilco. Should I do it against 7 or 8?
>
> On 31 October 2017 at 12:12, Kenneth Loafman <email address hidden>
> wrote:
>
> > No one is working on it that I know of. We would appreciate a merge
> > request or a patch set, either one.
> >
> >
> > On Sun, Oct 29, 2017 at 11:35 PM, Ian Gibbs <<email address hidden>
> >
> > wrote:
> >
> > > Is anyone working on this? I would be willing to fix this - it's
really
> > > annoying. My personal use case is with the UK leaving the EU, I need
to
> > > backup to eu-west-2 not the default EU region in order to not
> > > accidentally export data across borders.
> > >
> > > Is there any preference to how it would be implemented - cmd line
> > > argument vs specifying the region on the bucket URI, for example?
> > >
> > > ** Summary changed:
> > >
> > > - S3 APNortheast and APSoutheast connections
> > > + Support for non-default S3 regions
> > >
> > > --
> > > You received this bug notification because you are subscribed to
> > > Duplicity.
> > > https://bugs.launchpad.net/bugs/1003159
> > >
> > > Title:
> > > Support for non-default S3 regions
> > >
> > > Status in Duplicity:
> > > Confirmed
> > >
> > > Bug description:
> > > Support for Amazon's S3 eastern data centers would be much
> > > appreciated!
> > >
> > > The connections are already supported in Boto:
> > > http://docs.pythonboto.org/en/latest/ref/s3.html#module-
> > > boto.s3.connection
> > >
> > > Please expose these S3 features of Boto so that those of us located
> in
> > > non-US/EU locations can take advantage of the better bandwidth of
> > > using buckets that are physically closer.
> > >
> > > For example, an option like --s3-region with the following options:
> > > ap-northeast-1
> > > ap-southeast-1
> > > EU
> > > sa-east-1
> > > us-west-1
> > > us-west-2
> > >
> > > To manage notifications about this bug go to:
> > > https://bugs.launchpad.net/duplicity/+bug/1003159/+subscriptions
> > >
> >
> > --
> > You received this bug notification because you are subscribed to the bug
> > report.
> > https://bugs.launchpad.net/bugs/1003159
> >
> > Title:
> > Support for non-default S3 regions
> >
> > Status in Duplicity:
> > Confirmed
> >
> > Bug description:
> > Support for Amazon's S3 eastern data centers would be much
> > appreciated!
> >
> > The connections are already supported in Boto:
> > http://docs.pythonboto.org/en/latest/ref/s3.html#modu...

[Kap (kap467)]

Sorry not really aware of all the moving parts here but with this (hypothetical) patch would it then be possible to then use the zone with the storage class "Zone-Infrequent" access?

[Justin Myers (myersjustinc)]

Started to look into this myself, since my NAS (my normal Duplicity destination) is configured to copy my backups to an S3 bucket in `ca-central-1`, and I ran into this bug when seeing whether I could restore directly from that location.

I've only attempted reads so far, but I'm able to interact with this non-default bucket just fine if I use one of the endpoint hostnames listed here: https://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region

This fails for the aforementioned bucket with a 409 (Conflict)

```
$ duplicity collection-status "s3+http://${BUCKET}/${PREFIX}"
```

This succeeds:

```
$ duplicity collection-status "s3://s3.ca-central-1.amazonaws.com/${BUCKET}/${PREFIX}"
```

Inspiration comes from this comment in a related boto issue, since the library itself has trouble with newer non-default regions: https://github.com/boto/boto/issues/2741

(I imagine this would be easier for us to program around if we were using the newer boto3 library instead (https://aws.amazon.com/sdk-for-python/), but this workaround might be useful in the meantime.)