Support pod annotations for sidecar charms

[Tom Haddon]

We filed https://bugs.launchpad.net/juju/+bug/1884177 back in June of 2020 because we wanted to be able to set annotations on pods:

"""
We would like to be able to set pod annotations so that Prometheus will scrape all of the pods in our scale>1 deployment.

If we were at scale=1, service annotations would probably be sufficient, although we'd have to update the config to look at service annotations too.
"""

While this bug is fixed for non-sidecar charms, I don't see a way to be able to set pod annotations for non-sidecar charms.

I did try this as well:
```
juju config mattermost kubernetes-service-annotations="prometheus.io/port:8067 prometheus.io/scrape:true"
```
But then there's no output from (assuming a model named `mattermost`):
```
microk8s.kubectl describe service -n mattermost | grep prometheus
```

[John A Meinel (jameinel)]

I would guess that the 'config' step would depend on whether it is able to create the annotations on the associated pods. I believe for a sidecar charm, it would be possible to get access to the K8s API, but it might be harder to figure out what containers/pods need to be annotated, and it would certainly look differently than a previous pod-spec style charm (which could just include the annotations as part of its pod declaration)

[Canonical Juju QA Bot (juju-qa-bot)]

This Medium-priority bug has not been updated in 60 days, so we're marking it Low importance. If you believe this is incorrect, please update the importance.