kubernetes provider: juju removed existing ingress after controller restart
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Canonical Juju |
Fix Released
|
High
|
Yang Kelvin Liu |
Bug Description
Hi,
Earlier today, on controller restart (3 nodes, jujuds were restarted as close to simultaneously as we could manage) we found that charm-created ingresses for an app was removed. The application uses the k8s-wordpress charm which manually creates k8s ingresses due to LP:1849725.
| https:/
From the controller logs:
machine-0: 21:18:00 DEBUG juju.kubernetes
machine-0: 21:18:00 DEBUG juju.kubernetes
machine-2: 21:18:02 DEBUG juju.kubernetes
machine-2: 21:18:03 DEBUG juju.kubernetes
The controller in question has two separate models with applications named wordpress-k8s. Only the ingress for one of them was lost. The k8s cluster is running v1.16.10. There is no persistent storage configured, so we added the cluster to Juju with --skip-storage.
summary: |
- kubernetes provider: juju removed existing ingresses + kubernetes provider: juju removed existing ingress after controller + restart |
Changed in juju: | |
importance: | Undecided → High |
assignee: | nobody → Thomas Miller (tlmiller) |
milestone: | none → 2.8.1 |
description: | updated |
description: | updated |
description: | updated |
description: | updated |
description: | updated |
Changed in juju: | |
assignee: | Thomas Miller (tlmiller) → Yang Kelvin Liu (kelvin.liu) |
status: | New → Triaged |
status: | Triaged → In Progress |
Changed in juju: | |
status: | In Progress → Fix Committed |
Changed in juju: | |
status: | Fix Committed → Fix Released |
I believe it was because the ingress wasn't able to find the TLS secret:
2020/06/22 22:48:42 [alert] 31082#31082: *98047390 no ssl_certificate _by_lua* defined in server blog.launchpad.net while loading SSL certificate by lua, client: 91.189.91.49, server: 0.0.0.0:443 tls_post_ process_ client_ hello:cert cb error) while loading SSL certificate by lua, client: 91.189.91.49, server: 0.0.0.0:443
2020/06/22 22:48:42 [crit] 31082#31082: *98047390 SSL_do_handshake() failed (SSL: error:1417A179:SSL routines:
This certificate definitely exists and is correct in k8s.
ubuntu@ juju-7cf688- prod-is- external- kubernetes- 9:~$ kubectl get secrets blog-launchpad- net-tls -n prod-launchpad- blog-k8s net-tls Opaque 2 11d
NAME TYPE DATA AGE
blog-launchpad-
https:/ /pastebin. canonical. com/p/fnzVN9Kkh g/