rsyslog worker should not add machines that are not ready yet

[John A Meinel]

When using "juju ensure-availability" the new state servers get added as targets for forwarding log messages. However, those seem to be added before the machines have finished booting and setting up rsyslog to receive messages properly.

This causes the source rsyslog to queue up messages while waiting for the target to actually start receiving them.

I believe I even saw forwarding set up for machines that were actually in ERROR state. (though those shouldn't have IP addresses, so I could be wrong.)

It may also be that as we switch over to agents connecting directly to rsyslog and sending their messages we won't be triggering rsyslog to do the queuing itself. However we shouldn't tell agents that they should talk to a target before that target is ready to listen.

[John A Meinel (jameinel)]

So while this is potentially a problem in old versions of Juju, we're moving away from using rsyslog to do the aggregation. With direct connections our queue is just in-memory, and thus we won't fill disk and we already drop messages when full.
When we switch to logging via the API (to the DB) we again won't end up with large rsyslog disk queues.

A healthy environment won't run into this, it was more about preventative measures against failure cascading (we fail to log, causing us to fill up disk, causing other failures, etc.)

We already have the 0.5GB max queue disk space in place, which means this won't every get crazily out of hand.