second rsyslog server definition breaks logs to master

[Matthew Mosesohn]

When a second rsyslog server is configured, no logs are sent to the first server. This is causing issues in our CI tests.

From http://www.rsyslog.com/doc/rsyslog_reliable_forwarding.html :
# start forwarding rule 1
$ActionQueueType LinkedList # use asynchronous processing
$ActionQueueFileName srvrfwd1 # set file name, also enables disk mode
$ActionResumeRetryCount -1 # infinite retries on insert failure
$ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down
*.* @@server1:port
# end forwarding rule 1

# start forwarding rule 2
$ActionQueueType LinkedList # use asynchronous processing
$ActionQueueFileName srvrfwd2 # set file name, also enables disk mode
$ActionResumeRetryCount -1 # infinite retries on insert failure
$ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down
*.* @@server2
# end forwarding rule 2

This should fix deployed nodes logging correctly to both specified rsyslog servers.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/105719

[Matthew Mosesohn (raytrac3r)]

Fix in 105719 works to eventually discard messages sent via TCP to a nonresponsive rsyslog server. The best option is to use UDP to avoid this scenario. After a couple of MB of logs get generated, all old messages get collected and sent to Fuel Master. This will repeat indefinitely, though, distributing logs on and off in spurts. It is still a better option than completely broken logging when 2nd server is broken.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/105719
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=aef7924742d0b10b5f5cb944e917a66a9325d0f6
Submitter: Jenkins
Branch: master

commit aef7924742d0b10b5f5cb944e917a66a9325d0f6
Author: Matthew Mosesohn <email address hidden>
Date: Wed Jul 9 16:10:41 2014 +0400

    Fix logging to multiple remote rsyslog servers

    Change-Id: Ie248d795a1cf0e476bde8b6c2dad1fac0b660f79
    Closes-Bug: #1339659

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/5.0
Review: https://review.openstack.org/114270

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/114458

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/5.0
Review: https://review.openstack.org/114459

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/114458
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=df3ee5b1c3ca29ad8bbf4f570217b0dbe3c81dd7
Submitter: Jenkins
Branch: master

commit df3ee5b1c3ca29ad8bbf4f570217b0dbe3c81dd7
Author: Bogdan Dobrelya <email address hidden>
Date: Fri Aug 15 10:40:41 2014 +0300

    Align ActionQueue rsyslog params in templates

    The changes accepted in fix should be aligned
    with other templates we use for client.conf and
    server.conf then installing and configuring rsyslog.

    Related-bug: #1339659

    Change-Id: I9108b0fb992f3dd5a68f3b9d39d78705e243d9fb
    Signed-off-by: Bogdan Dobrelya <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/114270
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=fd457f904b890e8e5891c3f24fd16cf82a5dd350
Submitter: Jenkins
Branch: stable/5.0

commit fd457f904b890e8e5891c3f24fd16cf82a5dd350
Author: Matthew Mosesohn <email address hidden>
Date: Wed Jul 9 16:10:41 2014 +0400

    Fix logging to multiple remote rsyslog servers

    Change-Id: Ie248d795a1cf0e476bde8b6c2dad1fac0b660f79
    Closes-Bug: #1339659

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/114459
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=d35ded526370de29b620b4de53331f846c057b64
Submitter: Jenkins
Branch: stable/5.0

commit d35ded526370de29b620b4de53331f846c057b64
Author: Bogdan Dobrelya <email address hidden>
Date: Fri Aug 15 10:40:41 2014 +0300

    Align ActionQueue rsyslog params in templates

    The changes accepted in fix should be aligned
    with other templates we use for client.conf and
    server.conf then installing and configuring rsyslog.

    Related-bug: #1339659

    Change-Id: I9108b0fb992f3dd5a68f3b9d39d78705e243d9fb
    Signed-off-by: Bogdan Dobrelya <email address hidden>