Encryption Feature: Expose 'X-Encrypted-Data' header

[clayg]

We worked pretty hard when implementing encryption to make sure that a user couldn't tell if a Swift Operator chose to use encryption or not. But it turns out some other notable object storage systems always drop something like "x-amz-server-side-encryption" header in the response of data that's been encrypted server-side.

While the scope of the encryption feature as it is implemented today probably *prefers* the use case where the user should not need to know or care if data is encrypted - it doesn't *preclude* the use case where an operator would like to expose the information, or a user might want to know it.

Use-case:

Not all of the object data in swift is encrypted (added to existing cluster), so when I GET/HEAD i'd like to know this information.

Use-Case:

If a Swift deployment/solution is ever certified by ICD503 compliance or something, a user who's data requires this certification or something similar may need to audit to say "Yup, see my data says it's encrypted right there - CHEAAACKAH"

Probably the encryption middleware should add a Backend header in the response anytime it decrypts something (or maybe in the PUT response too?), then by default there'd be no change in behavior, but a keymaster implementation (or any other middleware like swift3) *could* translate the backend header to something user visible or even add a toggle for the translation behavior.

[Janie Richling (jrichli)]

Nice idea. We have in the past looked for any "crypto-meta" to indicate whether encryption was involved in any part. One complication is that you can have the body unencrypted, while having the metadata encrypted. This would happen if a POST is issued while having encryption in the pipeline against an existing unecrypted object. I suppose the encrypted flag that may be added would ensure that all related information is encrypted if the use case is for some sort of compliance.

There are a couple trello cards for encryption follow-up that are about only providing keys if there is a need for them. If changes are made in the keymaaster for this which would help to determine when exactly keys are needed, then this could be a related patch.

[Mahati Chamarthy (mahati-chamarthy)]

Associated Fix: https://review.openstack.org/#/c/348604/

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/619127

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/619127
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=8e95a93858dc215e2d2f7d48e035e6b5f1ab582a
Submitter: Zuul
Branch: master

commit 8e95a93858dc215e2d2f7d48e035e6b5f1ab582a
Author: Tim Burke <email address hidden>
Date: Tue Nov 20 17:21:04 2018 -0800

    s3api: Allow some forms of server-side-encryption

    ...if and only if encryption is enabled. A few things to note about server-side
    encryption:

    - We register whether encryption is present and enabled when the proxy server
      starts up.
    - This is generally considered an operator feature, not a user-facing one. S3
      API users can now learn more about how your cluster is set up than they
      previously could.
    - If encryption is enabled but there are no keymasters in the pipeline, all
      writes will fail with "Unable to retrieve encryption keys."
    - There's still a 'swift.crypto.override' env key that keymasters can set to
      skip encryption, so this isn't a full guarantee that things will be
      encrypted. On the other hand, none of the keymasters in Swift ever set that
      override.

    Note that this *does not* start including x-amz-server-side-encryption
    headers in the response, neither during PUT nor GET. We should only
    send that when we know for sure that the data on disk was encrypted.

    Change-Id: I4c20bca7fedb839628f1b2f8611807631b8bf430
    Related-Bug: 1607116
    Related-Change: Icf28dc57e589f9be20937947095800d7ce57b2f7