Need to set smtp server and associated tls variables for monitoring-grafana addon

[Drew Freiberger]

We have as a feature request, a need to update the smtp server for the grafana built into the dashboard-addons to be able usable as an alerting mechanism via email reports.

The grafana charm can serve as a basis for what configurations need to be updated.

Is it possible there are attributes exposed via kubernetes that could be used to modify this by the cloud operator?

[Drew Freiberger (afreiberger)]

The follow-up comment on this heapster related issue is probably the answer to how to get smtp variables updated into the pod via image updates.

https://github.com/kubernetes-retired/heapster/issues/1527

[Drew Freiberger (afreiberger)]

It seems that the influxdb-grafana-controller template already sets some GF_* vars, we just need to alter this template to set some more variables like smtp host, port, sslverify

https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/cluster-monitoring/influxdb/influxdb-grafana-controller.yaml#L68

which is pulled from:

https://github.com/juju-solutions/cdk-addons/blob/master/get-addon-templates#L271

Could the get-addon-templates script in the cdk-addons alter the influxdb-grafana-controller.yaml to add below into the env section of the pod file based on charm configs before running it in kubernetes?

GF_SMTP_ENABLED=true"
GF_SMTP_HOST=smtp.example.com:25
<email address hidden>
GF_SMTP_USER=smtpuser
GF_SMTP_PASSWORD=secret
GF_SMTP_SKIP_VERITY=true

[Jay Kuri (jk0ne)]

subscribed ~field-critical
per csanders - this bug is preventing customer from enabling grafana alerting.

[Tim Van Steenburgh (tvansteenburgh)]

The solution proposed in comment #2 won't really work since get-addon-templates runs when the cdk-addons snap is built, so it wouldn't be able to alter the templates at charm run time.

We don't have a good way to customize pieces of the default CDK addon templates while retaining CDK management of those addons. In other words, it's all or nothing. You either accept the defaults, and CDK will make sure those pods are running, and that they get upgraded with new releases of CDK, or you turn off the addons that you want to customize and apply your own yaml. One problem with that is that you'd be responsible for upgrading versions of those addon templates yourself.

I believe the correct long term solution is that we should have a grafana k8s charm, on which you could set this config, which would in turn modify the pod spec. But we don't have that today.

One possible solution that falls between the ideal of the future and the restrictive reality of today is to add charm config on kubernetes-master through which the addons yaml could be customized. It might look something like this:

addon-customisations:
  kubernetes-dashboard: <custom yaml>
  influxdb-grafana-controller: <custom yaml>
  grafana-service: <none>
  ...

So basically one entry in the map for each template that CDK deploys, with optional yaml that would be merged onto the default. This would allow for customisations while retaining the ability for CDK to upgrade the default addon templates over time.

What do you think of this idea? Not sure how quickly we could get this done, but interested in feedback on the approach.

In the meantime, if you want to set this custom grafana config today, here's what you could do:

1) juju run --unit $(juju status|grep kubernetes-master|grep '*'|awk -F'*' '{print $1}') -- "for file in kubernetes-dashboard influxdb-grafana-controller influxdb-service grafana-service heapster-rbac heapster-controller heapster-service; do cat /root/snap/cdk-addons/current/addons/\${file}.yaml;echo '---'; done" > dashboard.yaml
2) edit dashboard.yaml to match desired input
3) juju config kubernetes-master enable-dashboard-addons=false
4) kubectl apply -f dashboard.yaml

[Drew Freiberger (afreiberger)]

Am I correct in assuming we would need to remove the 'cdk-addons: true' labels from the dashboard.yaml to ensure the charm hooks don't reap the manually deployed addons?

[George Kraft (cynerva)]

Yes.