Juju container controller MongoDB startup script suppresses all mongod output

[Weii Wang]

One of the containers inside juju k8s controller pod is juju-db which starts a mongodb instance. The current startup script for that container is using bash command substitution to launch the mongodb process.

https://github.com/juju/juju/blob/485a8aa6f85b03c09428ba0c6984a5decc5f81ca/caas/scripts.go#L25

$(mongod ${args})

When the mongod failed to start, or some error occurred causing mongod to exit, command substitution will cause the shell to treat everything printed by mongod as a new command and try to execute that command. Of course, that’s not a command, so the only message left was some gibberish produced by shell when the shell is trying to run mongod’s output as a command. This will cause all the valuable debug information about the mongod to be lost.

The log remained will look like this:
/root/mongo.sh: 6: {"t":{"$date":"2022-07-14T06:18:39.191Z"},"s":"W",: not found

To reproduce this error, first bootstrap juju in k8s, then find the juju controller namespace and pod name:

$ juju version
2.9.32-ubuntu-amd64

$ kubectl get namespaces \
    -o=custom-columns=:.metadata.name | grep controller
controller-test

$ kubectl get pods -n controller-test \
    -o=custom-columns=:.metadata.name | grep controller
controller-0

Then manually send SIGTERM signal to mongod:

$ kubectl exec controller-0 -n controller-test \
    --container mongodb -- bash -c "pkill mongod"

Watch the log of the “failed” container:

$ kubectl logs controller-0 -n controller-test \
    --container mongodb -p
/root/mongo.sh: 6: {"t":{"$date":"2022-07-14T06:18:39.191Z"},"s":"W",: not found

[Weii Wang (weii-wang)]

I have proposed a simple patch for this issue.
https://github.com/juju/juju/pull/14372