Jujud rewrites /etc/init/juju-db.conf constantly

[Mark Shuttleworth]

Have noticed constant restarts of mongodb in syslog. It appears that jujud is touching /etc/init/juju-db.conf on a very regular basis (every few seconds). From auditd's audit.log:

type=PATH msg=audit(1406656399.162:214): item=0 name="/etc/init/juju-db.conf" inode=46923851 dev=08:01 mode=0100644 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL
type=SYSCALL msg=audit(1406656412.554:215): arch=c000003e syscall=2 success=yes exit=4 a0=c2100fd600 a1=80000 a2=0 a3=0 items=1 ppid=1 pid=27766 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 ses=4294967295 tty=(none) comm="jujud" exe="/var/lib/juju/tools/1.20.2-trusty-amd64/jujud" key="wtf"
type=CWD msg=audit(1406656412.554:215): cwd="/"
type=PATH msg=audit(1406656412.554:215): item=0 name="/etc/init/juju-db.conf" inode=46923851 dev=08:01 mode=0100644 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL
type=SYSCALL msg=audit(1406656420.566:216): arch=c000003e syscall=2 success=yes exit=4 a0=c2100fc6c0 a1=80000 a2=0 a3=0 items=1 ppid=1 pid=28127 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 ses=4294967295 tty=(none) comm="jujud" exe="/var/lib/juju/tools/1.20.2-trusty-amd64/jujud" key="wtf"
type=CWD msg=audit(1406656420.566:216): cwd="/"
type=PATH msg=audit(1406656420.566:216): item=0 name="/etc/init/juju-db.conf" inode=46923851 dev=08:01 mode=0100644 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL
type=SYSCALL msg=audit(1406656429.850:217): arch=c000003e syscall=2 success=yes exit=4 a0=c2100fd6c0 a1=80000 a2=0 a3=0 items=1 ppid=1 pid=28412 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 ses=4294967295 tty=(none) comm="jujud" exe="/var/lib/juju/tools/1.20.2-trusty-amd64/jujud" key="wtf"
type=CWD msg=audit(1406656429.850:217): cwd="/"
type=PATH msg=audit(1406656429.850:217): item=0 name="/etc/init/juju-db.conf" inode=46923851 dev=08:01 mode=0100644 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL

(this is with sudo auditctl -w /etc/init/juju-db.conf -k wtf -p rwa)

What caused me to be interested in this is that I tried to change the juju-db.conf and my changes were immediately overwritten by something. It appears that something is jujud itself.

How can I stop jujud from overwriting my /etc/init/juju-db.conf changes? Why is it doing that? And if there's a reason for it it do so, how can I change what it writes?

[Mark Shuttleworth (sabdfl)]

jujud is crashing and restarting. Every time it restarts it does:

2014-07-29 18:08:46 INFO juju.mongo mongo.go:169 Ensuring mongo server is running; data directory /var/lib/juju; port 37017
2014-07-29 18:08:46 INFO juju.mongo mongo.go:317 installing juju-mongodb
2014-07-29 18:08:46 INFO juju.utils.apt apt.go:128 Running: [apt-get --option=Dpkg::Options::=--force-confold --option=Dpkg::options::=--force-unsafe-io --assume-yes --quiet install juju-mongodb]
2014-07-29 18:08:46 INFO juju.state.api apiclient.go:250 error dialing "wss://localhost:17070/": websocket.Dial wss://localhost:17070/: dial tcp 127.0.0.1:17070: connection refused
2014-07-29 18:08:46 ERROR juju.worker runner.go:218 exited "api": unable to connect to "wss://localhost:17070/"
2014-07-29 18:08:46 INFO juju.worker runner.go:252 restarting "api" in 3s
2014-07-29 18:08:47 DEBUG juju.mongo mongo.go:260 using mongod: /usr/lib/juju/bin/mongod --version: "db version v2.4.9\nTue Jul 29 18:08:47.314 git version: nogitversion\n"
2014-07-29 18:08:47 INFO juju.state open.go:47 opening state, mongo addresses: ["127.0.0.1:37017"]; entity "machine-0"
2014-07-29 18:08:47 DEBUG juju.state open.go:52 dialing mongo
2014-07-29 18:08:47 DEBUG juju.mongo open.go:87 connection failed, will retry: dial tcp 127.0.0.1:37017: connection refused
2014-07-29 18:08:47 DEBUG juju.mongo open.go:87 connection failed, will retry: dial tcp 127.0.0.1:37017: connection refused
2014-07-29 18:08:48 INFO juju.mongo open.go:95 dialled mongo successfully

I thought that might be rewriting the Upstart juju-db.conf but it doesn't seem that it should:

sudo apt-get --option=Dpkg::Options::=--force-confold --option=Dpkg::options::=--force-unsafe-io --assume-yes --quiet install juju-mongodb
Reading package lists...
Building dependency tree...
Reading state information...
juju-mongodb is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 3 not upgraded.

In other words, apt doesn't reinstall it if it's there already.

[Menno Finlay-Smits (menno.smits)]

Here the machine agent is restarting frequently due to a mgo panic. jujud writes out a new mongo upstart config and restarts mongod every time it starts up - even if it's not necessary - causing the behaviour you're seeing.

This isn't very smart - thumper will fix.

[Dave Cheney (dave-cheney)]

oh -- that is why mongo is getting SIGTERM, from upstart. I was wrong
that it was the OOM killer.

On Wed, Jul 30, 2014 at 10:29 AM, Menno Smits <email address hidden> wrote:
> Here the machine agent is restarting frequently due to a mgo panic.
> jujud writes out a new mongo upstart config and restarts mongod every
> time it starts up - even if it's not necessary - causing the behaviour
> you're seeing.
>
> This isn't very smart - thumper will fix.
>
> --
> You received this bug notification because you are subscribed to juju-
> core.
> Matching subscriptions: MOAR JUJU SPAM!
> https://bugs.launchpad.net/bugs/1349969
>
> Title:
> Jujud rewrites /etc/init/juju-db.conf constantly
>
> Status in juju-core:
> Triaged
> Status in juju-core 1.20 series:
> Triaged
>
> Bug description:
> Have noticed constant restarts of mongodb in syslog. It appears that
> jujud is touching /etc/init/juju-db.conf on a very regular basis
> (every few seconds). From auditd's audit.log:
>
> type=PATH msg=audit(1406656399.162:214): item=0 name="/etc/init/juju-db.conf" inode=46923851 dev=08:01 mode=0100644 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL
> type=SYSCALL msg=audit(1406656412.554:215): arch=c000003e syscall=2 success=yes exit=4 a0=c2100fd600 a1=80000 a2=0 a3=0 items=1 ppid=1 pid=27766 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 ses=4294967295 tty=(none) comm="jujud" exe="/var/lib/juju/tools/1.20.2-trusty-amd64/jujud" key="wtf"
> type=CWD msg=audit(1406656412.554:215): cwd="/"
> type=PATH msg=audit(1406656412.554:215): item=0 name="/etc/init/juju-db.conf" inode=46923851 dev=08:01 mode=0100644 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL
> type=SYSCALL msg=audit(1406656420.566:216): arch=c000003e syscall=2 success=yes exit=4 a0=c2100fc6c0 a1=80000 a2=0 a3=0 items=1 ppid=1 pid=28127 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 ses=4294967295 tty=(none) comm="jujud" exe="/var/lib/juju/tools/1.20.2-trusty-amd64/jujud" key="wtf"
> type=CWD msg=audit(1406656420.566:216): cwd="/"
> type=PATH msg=audit(1406656420.566:216): item=0 name="/etc/init/juju-db.conf" inode=46923851 dev=08:01 mode=0100644 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL
> type=SYSCALL msg=audit(1406656429.850:217): arch=c000003e syscall=2 success=yes exit=4 a0=c2100fd6c0 a1=80000 a2=0 a3=0 items=1 ppid=1 pid=28412 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 ses=4294967295 tty=(none) comm="jujud" exe="/var/lib/juju/tools/1.20.2-trusty-amd64/jujud" key="wtf"
> type=CWD msg=audit(1406656429.850:217): cwd="/"
> type=PATH msg=audit(1406656429.850:217): item=0 name="/etc/init/juju-db.conf" inode=46923851 dev=08:01 mode=0100644 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL
>
> (this is with sudo auditctl -w /etc/init/juju-db.conf -k wtf -p rwa)
>
> What caused me to be interested in this is that I tried to change the
> juju-db.conf and my changes were immediately overwritten by something.
> It appears that something is jujud itself.
>
> How can I stop jujud from overwriting my /etc/init/juju-db.conf
> changes? Why is it doing that? And if there's a reason for it it do
> so, how can I chang...

[Tim Penhey (thumper)]

@Mark, you are right, it is updating the upstart job every time the agent restarts, and it shouldn't. I'm fixing.