centralized logging is missing due to rgw_enable_ops_log

[Nobuto Murata]

Ceph radosgw has a mechanism to log operations into Ceph itself. The handy feature is not enabled by default so it would be nice if the charm enables it for auditing purposes.

Currently GET operation for example is logged at each radosgw unit.
e.g. /var/log/ceph/ceph-client.rgw.juju-d642a4-0-lxd-1.log

However, the equivalent log cannot be retrieved via the Ceph embedded feature.

# radosgw-admin log list | head
[
 "obj_delete_at_hint.0000000078",
 "obj_delete_at_hint.0000000070",
 "obj_delete_at_hint.0000000104",
 "obj_delete_at_hint.0000000026",
 "obj_delete_at_hint.0000000028",
 "obj_delete_at_hint.0000000040",
 "obj_delete_at_hint.0000000015",
 "obj_delete_at_hint.0000000069",
 "obj_delete_at_hint.0000000095",

# radosgw-admin log show --bucket simplestreams \
    --bucket-id 52b9a1c1-1c52-42b8-85a8-89df0fbcc471.6774.1 \
    --date 2023-06-01-12
error reading log 2023-06-01-12-52b9a1c1-1c52-42b8-85a8-89df0fbcc471.6774.1-simplestreams: (2) No such file or directory

By enabling the feature by hand, it allows to see the log through the radsgw-admin interface.
https://docs.ceph.com/en/quincy/radosgw/config-ref/#confval-rgw_enable_ops_log

$ juju config ceph-radosgw config-flags='{"global":{"rgw enable ops log": true}}'

$ curl -v https://$HOSTNAME/simplestreams/foo

# radosgw-admin log show --bucket simplestreams \
    --bucket-id 52b9a1c1-1c52-42b8-85a8-89df0fbcc471.6774.1 \
    --date 2023-06-01-12 | head
{
 "bucket_id": "52b9a1c1-1c52-42b8-85a8-89df0fbcc471.6774.1",
 "bucket_owner": "1515718f5892428fa97d4641243848ed",
 "bucket": "simplestreams",
 "log_entries": [
     {
         "bucket": "simplestreams",
         "time": "2023-06-01T12:21:17.098687Z",
         "time_local": "2023-06-01T12:21:17.098687+0000",
         "remote_addr": "127.0.0.1",

# radosgw-admin log show --bucket simplestreams \
    --bucket-id 52b9a1c1-1c52-42b8-85a8-89df0fbcc471.6774.1 \
    --date 2023-06-01-12 | grep foo
         "uri": "GET /simplestreams/foo HTTP/1.1",

[Nobuto Murata (nobuto)]

We should additionally set either "rgw remote addr param" or "rgw log http headers" with "HTTP_X_FORWARDED_FOR" otherwise the remote ip is always 127.0.0.1.

$ juju config ceph-radosgw \
    config-flags='{"global":{"rgw enable ops log": true, "rgw remote addr param": "HTTP_X_FORWARDED_FOR"}}'

        {
            "bucket": "simplestreams",
            "time": "2023-06-01T13:10:00.060561Z",
            "time_local": "2023-06-01T13:10:00.060561+0000",
            "remote_addr": "192.168.151.107",
            "object_owner": "1515718f5892428fa97d4641243848ed",
            "user": "anonymous",
            "operation": "get_obj",
            "uri": "GET /simplestreams/foo HTTP/1.1",
            "http_status": "200",
            "error_code": "",
            "bytes_sent": 0,
            "bytes_received": 0,
            "object_size": 0,
            "total_time": 4,
            "user_agent": "curl/7.81.0",
            "referrer": "",
            "trans_id": "tx00000ff5efd6ded2863e0-00647898a8-9207-default",
            "authentication_type": "Local",
            "temp_url": false
        },

$ juju config ceph-radosgw \
    config-flags='{"global":{"rgw enable ops log": true, "rgw log http headers": "http_x_forwarded_for"}}'

        {
            "bucket": "simplestreams",
            "time": "2023-06-01T12:44:27.316514Z",
            "time_local": "2023-06-01T12:44:27.316514+0000",
            "remote_addr": "127.0.0.1",
            "object_owner": "1515718f5892428fa97d4641243848ed",
            "user": "anonymous",
            "operation": "get_obj",
            "uri": "GET /simplestreams/foo HTTP/1.1",
            "http_status": "200",
            "error_code": "",
            "bytes_sent": 0,
            "bytes_received": 0,
            "object_size": 0,
            "total_time": 8,
            "user_agent": "curl/7.81.0",
            "referrer": "",
            "http_x_headers": [
                {
                    "HTTP_X_FORWARDED_FOR": "192.168.151.107"
                }
            ],
            "trans_id": "tx000002dcd7463dff3832f-00647892ab-8c1f-default",
            "authentication_type": "Local",
            "temp_url": false
        }

[Nobuto Murata (nobuto)]

"good-first-bug" to have a config option as "enable-ops-log" or something like that.