rsyslogd configuration does not support multiple local environments

[Menno Finlay-Smits]

After bootstrapping a fairly simple environment with current master (HEAD at 4e672d2cf5) all machine logs except machine-0 and all unit logs have the following repeated over and over:

INFO juju.worker runner.go:261 start "rsyslog"
DEBUG juju.worker.rsyslog worker.go:86 starting rsyslog worker mode 1 for "machine-1" "menno-local"
ERROR juju.worker runner.go:219 exited "rsyslog": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "juju-generated CA for environment \"rsyslog\"")
INFO juju.worker runner.go:253 restarting "rsyslog" in 3s

To repeat:

  juju switch local
  juju destroy-environment --force --yes local
  juju bootstrap
  juju deploy mysql
  juju deploy mediawiki
  juju add-relation mysql:db mediawiki:db
  # wait for provisioning to finish
  # check logs

The problem is easy to reproduce. It happens each time I run the above.

[Menno Finlay-Smits (menno.smits)]

This problem is only reproducible on my machine so far so I'm lowering the priority.

[Menno Finlay-Smits (menno.smits)]

The cause of this issue is that the rsyslog configuration for another (destroyed) local environment had been left behind in /etc/rsyslog.d on the host machine. The configuration referred to different certs but still listened on the same rsyslog port. That configuration seemed to "win" over the configuration for the active local environment so the certificate used by rsyslogd didn't match the (correct) certificate being used by the client.

Is the rsyslog worker supposed to work with concurrent local environments? At the very least the rsyslog config should get removed once an environment is destroyed.

Changing the bug title to reflect the root cause...

[Paul Gear (paulgear)]

I saw this problem on a new deploy with MAAS as the provider. I'm not sure whether there was a problem with MAAS not releasing the node correctly when the previous environment was torn down, but regardless the machine should have been automatically reinstalled when the node was allocated by juju. If it occurs again I will gather machine logs...

[Paul Gear (paulgear)]

I encountered this again almost straight away; decided to log bug 1417875, because the circumstances were different to this bug.