Requests for certificates hang when using vault charm in Juju Cross Model Relation
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Charm Helpers |
Triaged
|
High
|
Unassigned | ||
vault-charm |
Fix Released
|
High
|
Chris MacNaughton |
Bug Description
We are setting up a kubernetes cluster using Juju running on OpenStack.
Our undercloud juju controller is running vault in a model. We have created an offer on that model to expose it out for cross model relations.
Our overcloud juju controller (that targets OpenStack) has kubernetes-master, kubernetes-worker and etcd charms deployed.
We then created a cross controller, cross model relation for the Overcloud juju controller to consume the undercloud Vault:
```juju consume undercloud:
This shows up on the overcloud juju controller just fine, however, I then try creating a relationship from etcd:certificates to the exposed vault:
``` juju add-relation etcd:certificates vault```
This shows success, however, the charm status is locked in maintenance:
Unit Workload Agent Machine Public address Ports Message
etcd/0* maintenance idle 0 172.16.20.60 Requesting tls certificates.
The etcd unit log shows the following in an endless loop:
2019-01-28 14:12:29 INFO juju-log certificates:10: Initializing Snap Layer
2019-01-28 14:12:29 DEBUG certificates-
2019-01-28 14:12:29 INFO juju-log certificates:10: Invoking reactive handler: reactive/
2019-01-28 14:12:29 INFO juju-log certificates:10: Invoking reactive handler: reactive/
2019-01-28 14:12:29 INFO juju-log certificates:10: Invoking reactive handler: reactive/
2019-01-28 14:12:29 INFO juju-log certificates:10: Invoking reactive handler: reactive/
2019-01-28 14:12:30 INFO juju-log certificates:10: Invoking reactive handler: reactive/
2019-01-28 14:12:30 INFO juju-log certificates:10: Get config refresh.timer for snap core
2019-01-28 14:12:30 INFO juju-log certificates:10: Invoking reactive handler: hooks/relations
2019-01-28 14:12:38 INFO juju-log certificates:10: Reactive main running for hook certificates-
2019-01-28 14:12:38 INFO juju-log certificates:10: Initializing Leadership Layer (is leader)
2019-01-28 14:12:39 INFO juju-log certificates:10: Initializing Snap Layer
2019-01-28 14:12:39 DEBUG certificates-
2019-01-28 14:12:39 INFO juju-log certificates:10: Invoking reactive handler: reactive/
2019-01-28 14:12:39 INFO juju-log certificates:10: Invoking reactive handler: reactive/
2019-01-28 14:12:39 INFO juju-log certificates:10: Invoking reactive handler: reactive/
2019-01-28 14:12:39 INFO juju-log certificates:10: Invoking reactive handler: reactive/
2019-01-28 14:12:40 INFO juju-log certificates:10: Invoking reactive handler: reactive/
2019-01-28 14:12:40 INFO juju-log certificates:10: Get config refresh.timer for snap core
2019-01-28 14:12:40 INFO juju-log certificates:10: Invoking reactive handler: hooks/relations
2019-01-28 14:12:48 INFO juju-log certificates:10: Reactive main running for hook certificates-
2019-01-28 14:12:48 INFO juju-log certificates:10: Initializing Leadership Layer (is leader)
2019-01-28 14:12:48 INFO juju-log certificates:10: Initializing Snap Layer
2019-01-28 14:12:49 DEBUG certificates-
2019-01-28 14:12:49 INFO juju-log certificates:10: Invoking reactive handler: reactive/
2019-01-28 14:12:49 INFO juju-log certificates:10: Invoking reactive handler: reactive/
2019-01-28 14:12:49 INFO juju-log certificates:10: Invoking reactive handler: reactive/
2019-01-28 14:12:49 INFO juju-log certificates:10: Invoking reactive handler: reactive/
2019-01-28 14:12:50 INFO juju-log certificates:10: Invoking reactive handler: reactive/
2019-01-28 14:12:50 INFO juju-log certificates:10: Get config refresh.timer for snap core
2019-01-28 14:12:50 INFO juju-log certificates:10: Invoking reactive handler: hooks/relations
2019-01-28 14:12:57 INFO juju-log certificates:10: Reactive main running for hook certificates-
2019-01-28 14:12:58 INFO juju-log certificates:10: Initializing Leadership Layer (is leader)
2019-01-28 14:12:58 INFO juju-log certificates:10: Initializing Snap Layer
2019-01-28 14:12:58 DEBUG certificates-
2019-01-28 14:12:58 INFO juju-log certificates:10: Invoking reactive handler: reactive/
2019-01-28 14:12:58 INFO juju-log certificates:10: Invoking reactive handler: reactive/
2019-01-28 14:12:59 INFO juju-log certificates:10: Invoking reactive handler: reactive/
2019-01-28 14:12:59 INFO juju-log certificates:10: Invoking reactive handler: reactive/
2019-01-28 14:13:00 INFO juju-log certificates:10: Invoking reactive handler: reactive/
2019-01-28 14:13:00 INFO juju-log certificates:10: Get config refresh.timer for snap core
2019-01-28 14:13:00 INFO juju-log certificates:10: Invoking reactive handler: hooks/relations
2019-01-28 14:13:07 INFO juju-log certificates:10: Reactive main running for hook certificates-
2019-01-28 14:13:08 INFO juju-log certificates:10: Initializing Leadership Layer (is leader)
2019-01-28 14:13:08 INFO juju-log certificates:10: Initializing Snap Layer
The overcloud etcd lxd can access the undercloud vault API at a network level:
ubuntu@
404 page not found
tags: | added: atos |
Changed in vault-charm: | |
assignee: | nobody → Chris MacNaughton (chris.macnaughton) |
milestone: | none → 19.04 |
importance: | Undecided → High |
Changed in vault-charm: | |
status: | New → Confirmed |
Changed in vault-charm: | |
milestone: | 19.04 → 19.07 |
I have reproduced this and am digging in further.
I can see, in the charm logs on Vault, that
2019-02-05 07:48:07 INFO juju-log certificates:3: Processing certificate request from remote- 2ea1e7247f67465 68fd0e43cacfdc7 c0/0 for 10.5.0.11
where the above IP is the etcd unit in the remote model.