rgw trust forwarded https not set

[Samuel Allan]

For use with encryption, the `rgw trust forwarded https` option must be enabled.

> Requests for server-side encryption must be sent over a secure HTTPS connection to avoid sending secrets in plaintext. If a proxy is used for SSL termination, rgw trust forwarded https must be enabled before forwarded requests will be trusted as secure.

https://docs.ceph.com/en/latest/radosgw/encryption/

This option is not currently set by the charm. It should be set, so that SSE-C encryption can work out of the box.

It should be always set if the apache2 proxy with ssl is being used. Not sure the security implications of setting it if ssl is not enabled.

---

Note: a current workaround is to run this:

```
juju config ceph-radosgw config-flags='{"global":{"rgw trust forwarded https": true}}'
```

Which configures /etc/ceph/ceph.conf as required with

```
[global]
rgw trust forwarded https = True
```

[Nobuto Murata (nobuto)]

Here is the config reference:

https://docs.ceph.com/en/latest/radosgw/config-ref/?highlight=trust%20forwarded%20https#confval-rgw_trust_forwarded_https
> Enable this option to trust the Forwarded and X-Forwarded-Proto headers sent by the proxy when determining whether the connection is secure. This is required for some features, such as server side encryption. (Never enable this setting if you do not have a trusted proxy in front of radosgw, or else malicious users will be able to set these headers in any request.)

And the charm has the following for Apache2 as a reverse proxy and TLS termination:
https://github.com/openstack/charm-ceph-radosgw/blob/4484b0f0ed0c3b32947cfaa96025dbcc489c4f14/templates/openstack_https_frontend.conf#L24
> RequestHeader set X-Forwarded-Proto "https"

So I think we can enable the config when TLS is configured by the charm.

[Samuel Allan (samuelallan)]

For reference, this is the output from the minio cli when the trust forwarded https is not enabled:

```
$ mcli --insecure cp mcli_20230526233154.0.0_amd64.deb testbed/mybucket/encrypted-mcli2.deb --encrypt-key "testbed/mybucket/=JkjkEoXp0HC3YYVFm7IKaOoG/18ZvI8bQ/y+ioPNx2s="
mcli: <ERROR> Failed to copy `mcli_20230526233154.0.0_amd64.deb`. Error response code InvalidRequest.

# and debug output for the last request
mcli: <DEBUG> PUT /mybucket/encrypted-mcli2.deb HTTP/1.1
Host: 192.168.151.118
User-Agent: MinIO (linux; amd64) minio-go/v7.0.55 mcli/RELEASE.2023-05-26T23-31-54Z
Content-Length: 9862956
Authorization: AWS4-HMAC-SHA256 Credential=**REDACTED**/20230530/default/s3/aws4_request, SignedHeaders=content-type;host;x-amz-content-sha256;x-amz-date;x-amz-server-side-encryption-customer-algorithm;x-amz-server-side-encryption-customer-key;x-amz-server-side-encryption-customer-key-md5, Signature=**REDACTED**
Content-Type: application/x-debian-package
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20230530T065040Z
X-Amz-Server-Side-Encryption-Customer-Algorithm: AES256
X-Amz-Server-Side-Encryption-Customer-Key: JkjkEoXp0HC3YYVFm7IKaOoG/18ZvI8bQ/y+ioPNx2s=
X-Amz-Server-Side-Encryption-Customer-Key-Md5: DqFKVxJ0Q63IcSfbi3fKnw==
Accept-Encoding: gzip

mcli: <DEBUG> HTTP/1.1 400 Bad Request
Connection: close
Content-Length: 220
Accept-Ranges: bytes
Content-Type: application/xml
Date: Tue, 30 May 2023 06:50:40 GMT
Server: Apache/2.4.52 (Ubuntu)
X-Amz-Request-Id: tx00000b40b01d62d21cfb9-0064759cc0-2d9f-default

<?xml version="1.0" encoding="UTF-8"?><Error><Code>InvalidRequest</Code><BucketName>mybucket</BucketName><RequestId>tx00000b40b01d62d21cfb9-0064759cc0-2d9f-default</RequestId><HostId>2d9f-default-default</HostId></Error>mcli: <DEBUG> TLS Certificate found:
mcli: <DEBUG> >> Expires: 2024-05-29 03:59:43 +0000 UTC
mcli: <DEBUG> Response Time: 25.577889ms

```

And when it is, it works fine:

```
$ mcli --insecure cp mcli_20230526233154.0.0_amd64.deb testbed/mybucket/encrypted-mcli2.deb --encrypt-key "testbed/mybucket/=JkjkEoXp0HC3YYVFm7IKaOoG/18ZvI8bQ/y+ioPNx2s="
...0230526233154.0.0_amd64.deb: 9.41 MiB / 9.41 MiB ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 28.95 MiB/s 0s
```

[Nobuto Murata (nobuto)]

Subscribing ~field-high.

It's a cosmetic issue indeed and it's easy to workaround by passing config-flag. But I personally think that the charmed radosgw deployment has a deviation from the expected deployment by the upstream radosgw and unnecessarily blocks a documented function of radosgw as Amazon SSE-C.

https://docs.ceph.com/en/quincy/radosgw/encryption/#customer-provided-keys
> no special Ceph configuration is needed to support this encryption mode

[Samuel Allan (samuelallan)]

Submitted a patch to address this: https://review.opendev.org/c/openstack/charm-ceph-radosgw/+/884797

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-ceph-radosgw/+/884797
Committed: https://opendev.org/openstack/charm-ceph-radosgw/commit/541ceec4018e311cef1517a62eefa28cd53bc162
Submitter: "Zuul (22348)"
Branch: master

commit 541ceec4018e311cef1517a62eefa28cd53bc162
Author: Samuel Walladge <email address hidden>
Date: Wed May 31 12:23:11 2023 +0930

    Enable rgw trust forwarded https when https proxy

    This option is required for server-side encryption to be allowed
    if radosgw is behind a reverse proxy,
    such as here when certificates are configured and apache2 is running.

    ref. https://docs.ceph.com/en/latest/radosgw/encryption/

    It is safe to always enable when https is configured in the charm,
    because it will be securely behind the reverse proxy in the unit.
    This option must not be enabled when https is not configured in the charm,
    because this would allow clients to spoof headers.

    Closes-Bug: #2021560
    Change-Id: I940f9b2f424a3d98936b5f185bf8f87b71091317

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/quincy.2
Review: https://review.opendev.org/c/openstack/charm-ceph-radosgw/+/884827

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-ceph-radosgw/+/884827
Committed: https://opendev.org/openstack/charm-ceph-radosgw/commit/13664c11d01976d1b711b8bea184ca2d7a55c10e
Submitter: "Zuul (22348)"
Branch: stable/quincy.2

commit 13664c11d01976d1b711b8bea184ca2d7a55c10e
Author: Samuel Walladge <email address hidden>
Date: Wed May 31 12:23:11 2023 +0930

    Enable rgw trust forwarded https when https proxy

    This option is required for server-side encryption to be allowed
    if radosgw is behind a reverse proxy,
    such as here when certificates are configured and apache2 is running.

    ref. https://docs.ceph.com/en/latest/radosgw/encryption/

    It is safe to always enable when https is configured in the charm,
    because it will be securely behind the reverse proxy in the unit.
    This option must not be enabled when https is not configured in the charm,
    because this would allow clients to spoof headers.

    Closes-Bug: #2021560
    Change-Id: I940f9b2f424a3d98936b5f185bf8f87b71091317
    (cherry picked from commit 541ceec4018e311cef1517a62eefa28cd53bc162)