Juju container controller MongoDB startup script suppresses all mongod output

Bug #1982832 reported by Weii Wang
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
juju
Fix Released
Medium
Weii Wang

Bug Description

One of the containers inside juju k8s controller pod is juju-db which starts a mongodb instance. The current startup script for that container is using bash command substitution to launch the mongodb process.

https://github.com/juju/juju/blob/485a8aa6f85b03c09428ba0c6984a5decc5f81ca/caas/scripts.go#L25

$(mongod ${args})

When the mongod failed to start, or some error occurred causing mongod to exit, command substitution will cause the shell to treat everything printed by mongod as a new command and try to execute that command. Of course, that’s not a command, so the only message left was some gibberish produced by shell when the shell is trying to run mongod’s output as a command. This will cause all the valuable debug information about the mongod to be lost.

The log remained will look like this:
/root/mongo.sh: 6: {"t":{"$date":"2022-07-14T06:18:39.191Z"},"s":"W",: not found

To reproduce this error, first bootstrap juju in k8s, then find the juju controller namespace and pod name:

$ juju version
2.9.32-ubuntu-amd64

$ kubectl get namespaces \
    -o=custom-columns=:.metadata.name | grep controller
controller-test

$ kubectl get pods -n controller-test \
    -o=custom-columns=:.metadata.name | grep controller
controller-0

Then manually send SIGTERM signal to mongod:

$ kubectl exec controller-0 -n controller-test \
    --container mongodb -- bash -c "pkill mongod"

Watch the log of the “failed” container:

$ kubectl logs controller-0 -n controller-test \
    --container mongodb -p
/root/mongo.sh: 6: {"t":{"$date":"2022-07-14T06:18:39.191Z"},"s":"W",: not found

Weii Wang (weii-wang)
description: updated
Revision history for this message
Weii Wang (weii-wang) wrote :

I have proposed a simple patch for this issue.
https://github.com/juju/juju/pull/14372

Ian Booth (wallyworld)
Changed in juju:
milestone: none → 2.9.33
assignee: nobody → Weii Wang (weii-wang)
importance: Undecided → Medium
status: New → In Progress
Ian Booth (wallyworld)
Changed in juju:
status: In Progress → Fix Committed
Changed in juju:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers