Download of vol1 in validate_encryption_settings() fails when using S3 glacier

[Alexander Hofstätter]

Duplicity version: 0.8.12.1612
ubuntu 20.04 LTS
python 3.8.2

When i try to restart a backup which goes to S3 (via boto3) (use of: --s3-use-deep-archive or --s3-use-glacier) it fails:

Last full backup date: Wed Apr 29 17:45:32 2020
RESTART: Volumes 2 to 2 failed to upload before termination.
         Restarting backup at volume 2.
Attempt 1 failed. ClientError: An error occurred (InvalidObjectState) when calling the GetObject operation: The operation is not valid for the object's storage class
Attempt 2 failed. ClientError: An error occurred (InvalidObjectState) when calling the GetObject operation: The operation is not valid for the object's storage class

This is because a file stored in glacier cannot be downloaded. I dont know if this known?

To solve this: either skip validate_encryption_settings() when using any of this storage backends or introduce an option to skip the download manually (e.g. --skip-download-on-restart)

I fixed it for me with manully skipping the validate_encryption_settings() function.

[Alexander Hofstätter (alexanderhofstaetter)]

Update: Actually, all actions which require read access to the destination are not possible when the objects have a glacier-like storage classe are not possible ...

[Kenneth Loafman (kenneth-loafman)]

When using boto3, glacier restore is not supported yet. You'll need to download the backup to a local drive and restore from there.

[Alexander Hofstätter (alexanderhofstaetter)]

Yes, this is clear as per design of the glacier storage backend. However, the problem ist not the restore action, the problem is that when duplicity wants to restart a backup (because e.g. the upload failed recently) it wants to download the latest archive to verify the use of the same keys.

This is not possible with glacier and therefore if the upload of a backup to glacier fails the backup can never be continued.

[Jack (jack-234521)]

This is not fixed in 8.14 - glacier and glacier deep mode save the signature files in glacier/deep storage class, and --s3-use-ia stores BOTH the manifest and signature files in the IA storage class.

[Kenneth Loafman (kenneth-loafman)]

It seems to be working as it should. When you do the incremental are you specifying the storage class or using IA? We check to see if --s3-use-deep-archive or --s3-use-glacier is used and don't run validate_encryption_settings() if either is set.

[Jack (jack-234521)]

My incremental job uses the same parameters as the full, and it fails on glacier/deep because duplicity tries to pull the remote manifest and can't retrieve it - here's what I run:

duplicity /mnt/backup/src boto3+s3://BUCKETNAME/FOLDERNAME --file-prefix-archive archive-$(hostna
me -f)- --file-prefix-manifest manifest-$(hostname -f)- --file-prefix-signature signature-$(hostname -f)- --s3-use-deep-archive

...and here's the output:
Synchronizing remote metadata to local cache...
Copying signature-backup.HOSTNAME.com-duplicity-full-signatures.20200830T050003Z.sigtar.gpg to local cache.
Attempt 1 failed. ClientError: An error occurred (InvalidObjectState) when calling the GetObject operation: The operation is not valid for the object's storage class

It tries 4 more attempts and then fails completely.

I'm still not following the logic here anyways - wouldn't it be better to store the signature files as as a standard class file, and allow the metadata sync to proceed as usual...rather than just disabling metadata sync for non-standard storage classes? The implementation seems to be inconsistent across AWS storage classes, and I can't get incrementals to work at all with glacier deep.

[Jack (jack-234521)]

Sorry - mixed my words - as you can see it's failing on an attempt to retreive the remote *signatures* file, not the manifest file. The manifest is stored in standard class; the signatures file is stored in DEEP_ARCHIVE class.

[Hartmut (freemlist)]

Any progress on this bug? I have the same issue:
- Run a full backup with parameter --s3-use-deep-archive
- Next day run a incremental backup, which results in the following error:

"Copying duplicity-full-signatures.20220724T100144Z.sigtar.gpg to local cache.
Attempt of get Nr. 1 failed. InvalidObjectState: An error occurred (InvalidObjectState) when calling the GetObject operation: The operation is not valid for the object's storage class"

The bug is reproducible under Ubuntu 20.04.4 LTS with version 0.8.12 and 0.8.23.2

[Kenneth Loafman (kenneth-loafman)]

Yes, this was fixed for boto+s3, but then we changed the default for s3:// from boto to boto3. Boto3 does not support glacier yet, so you may just need to specify boto+s3:// instead.

[Eric (bugcatcher321)]

TLDR;

Simply temporarily restore the signature files from glacier to s3. Once the restore is complete, configure your backup to use the legacy "boto+s3://" and run the incremental backup.

-=-=-=-=-=-

To clarify the issue reported here (recently), there seems to be an edge case where during the last incremental backup the `duplicity-new-signatures.20220720T121233Z.to.20220724T120838Z.sigtar.gz` signature file was not saved to local cache.

However, the signature file is saved to S3 with the glacier storage class.

On the next incremental backup, it of course fails when duplicity attempts to save the signatures file to local cache, but cannot access the glacier file.

For myself, incremental backups performed on/after July 24, 2022 did not save a local copy of the signature file.

I can confirm that my backup was configured with "s3://" and "--s3-use-deep-archive". Thus, my hypothesis is that edge case here is due to performing a one-time successful incremental backup being after the user updated duplicity with s3:// default to boto3, and for whatever reason the signature file was not saved to the local cache (as you mentioned Boto3 does not support glacier yet).

To leave a breadcrumb trail to solve the issue for others, temporarily restore the signature files from glacier to s3. Once the restore is complete, configure your backup to use the legacy "boto+s3://", you will also need to configure "--s3-region-name" and "--s3-endpoint-url" and run the incremental backup.

It will download the signature files to cache, and also successfully save any new signature files from the incremental backup to the local cache. (Just fixed my glacier backups, and can confirm v0.8.23 is working as it should with glacier).