ceph-csi units blocked with `Config manifests require the definition of 'fsid'`

[Marian Gasparovic]

When running jobs/integration/validation.py::test_kubelet_extra_config ceph-csi units go to blocked state reporting `Config manifests require the definition of 'fsid'`

Logs - https://oil-jenkins.canonical.com/artifacts/585af18b-43be-4813-bb27-b44b232af98b/index.html

[Adam Dyess (addyess)]

is this a duplicate of https://bugs.launchpad.net/charm-ceph-csi/+bug/2054583

[Kevin W Monroe (kwmonroe)]

Hi @marian, from the logs this looks like your env is a microk8s charm based deployment that has ceph-csi charm units in the model. Ceph/mk8s integration wouldn't use the ceph-csi charm, so perhaps the env is dirty from previous validation runs?

If not, it looks similar to: https://bugs.launchpad.net/charm-ceph-csi/+bug/2035393

Which we determined to be invalid because the actual root cause was an invalid namespace prior to having the ceph manifests deployed: https://bugs.launchpad.net/charm-ceph-csi/+bug/2053265

That was fixed in charmed k8s 1.29+ck1 (released late april 2024). I'm gonna mark this bug as invalid for now and ask that you confirm with charmed-k8s + ceph-csi at 1.29+ck1 (or greater) to weed out any env and/or duplicate fixes. Thank you!

[Jeffrey Chang (modern911)]

ceph-csi node in blocked state before we start k8s-suite.
And we skipped checking ceph-csi status in the sku, can be some configuration issue.

This is not a mk8s deployment, but a charmed k8s 1.30 beta.
And this problem doesn't happen with 1.29/stable.
We might need to tweak config for 1.30 a bit.

[Adam Dyess (addyess)]

Likely resolved with

https://github.com/charmed-kubernetes/ceph-csi-operator/commit/ca9577257f331ecb95ca5ef1fdd94c649dc4f3f4

[Adam Dyess (addyess)]

https://solutions.qa.canonical.com/testruns/da629ee9-0528-4dfe-9e8f-59fe8bf3e538

aight, on this run -- i notice that the charm is failing to get the `ceph fsid` on the ceph-csi units. (same as the kubernetes-control-plane units)

* the charm apt installs the ceph tools
* configures the ceph client using connection details from the ceph-mon units
* runs ceph cli commands on that unit

i see reports in the charm

```
2024-06-25 00:26:51 ERROR unit.ceph-csi/1.juju-log server.go:325 get_ceph_fsid: Failed to get CephFS ID, reporting as empty string
```

which comes from the charm code:
```python
    def get_ceph_fsid(self) -> str:
        """Get the Ceph FSID (cluster ID)"""
        try:
            return self.ceph_cli("fsid").strip()
        except subprocess.SubprocessError:
            logger.error("get_ceph_fsid: Failed to get CephFS ID, reporting as empty string")
            return ""
```

i guess i need more details since either ceph-mon isn't providing the correct connection details, or the cli tools cannot reach the ceph-mon hosts

[Adam Dyess (addyess)]

Thanks so much to @asbalderson in helping me troubleshoot this.

it turned out that ceph-csi charm and ceph-mon charm were using a juju space `public-space` to communicate over the `ceph-client` relation.

Howver, since `ceph-csi` is a subordinate -- the primary `kubernetes-control-plane` didn't have an interface in the `public-space` space and was using an address from the `oam-space` to communicate.

This resulted in the ceph client on the ceph-csi unit not being able to contact the ceph-mon agent

Perhaps this bug could BEST be resolved by providing some indication to the user that the ceph-mon is timing out trying to contact it.