rabbitmq does not finish installing

[Marian Gasparovic]

Sunbeam bootstrap times out because rabbitmq does not finish installing.

In rabbitmg logs there is

Application rabbit exited with reason: {cannot_acquire_startup_lock,{rabbit,start,[normal,[]]}}

Logs and configs
https://oil-jenkins.canonical.com/artifacts/1cbf4758-35f2-48d6-a92b-83d2d5585746/index.html

I saw it twice in manual install as well.

[James Page (james-page)]

From rabbitmq logs:

2023-09-22T18:47:44.446Z [rabbitmq] 2023-09-22 18:47:44.446579+00:00 [info] <0.228.0> Will try to lock with peer discovery backend rabbit_peer_discovery_k8s
2023-09-22T18:47:44.447Z [rabbitmq] 2023-09-22 18:47:44.447602+00:00 [warning] <0.489.0> Description: "Authenticity is not established by certificate path validation"
2023-09-22T18:47:44.447Z [rabbitmq] 2023-09-22 18:47:44.447602+00:00 [warning] <0.489.0> Reason: "Option {verify, verify_peer} and cacertfile/cacerts is missing"
2023-09-22T18:47:44.447Z [rabbitmq] 2023-09-22 18:47:44.447602+00:00 [warning] <0.489.0> 
2023-09-22T18:47:48.451Z [rabbitmq] 2023-09-22 18:47:48.451012+00:00 [error] <0.228.0> Failed to fetch a list of nodes from Kubernetes API: {failed_connect,[{to_address,{"kubernetes.default.svc.cluster.local",443}},
2023-09-22T18:47:48.451Z [rabbitmq] 2023-09-22 18:47:48.451012+00:00 [error] <0.228.0> {inet,[inet],timeout}]}
2023-09-22T18:47:48.452Z [rabbitmq] 2023-09-22 18:47:48.452228+00:00 [warning] <0.491.0> Description: "Authenticity is not established by certificate path validation"
2023-09-22T18:47:48.452Z [rabbitmq] 2023-09-22 18:47:48.452228+00:00 [warning] <0.491.0> Reason: "Option {verify, verify_peer} and cacertfile/cacerts is missing"
2023-09-22T18:47:48.452Z [rabbitmq] 2023-09-22 18:47:48.452228+00:00 [warning] <0.491.0> 
2023-09-22T18:47:52.454Z [rabbitmq] 2023-09-22 18:47:52.454747+00:00 [error] <0.228.0> Failed to lock with peer discovery backend rabbit_peer_discovery_k8s: "{failed_connect,[{to_address,{\"kubernetes.default.svc.cluster.local\",443}},\n {inet,[inet],timeout}]}"

Looks like the certificate verification for the K8S API endpoint used for discovery of peers failed.

[James Page (james-page)]

@marosg - have you seen this issue multiple times in deployments?

[Marian Gasparovic (marosg)]

@james-page I can see two occurrences, both on Sep 25