puppet logs are missing in /var/log/remote (and UI as well)

[Tatyanka]

{"build_id": "2014-04-30_10-31-20", "mirantis": "yes", "build_number": "170", "ostf_sha": "134765fcb5a07dce0cd1bb399b2290c988c3c63b", "nailgun_sha": "fb676be97794d4b63b8d179e2a8518ad70448f48", "production": "docker", "api": "1.0", "fuelmain_sha": "44954d8ffd74d27219434ec5b676874504b6bf76", "astute_sha": "3cffebde1e5452f5dbf8f744c6525fc36c7afbf3", "release": "5.0", "fuellib_sha": "fd547e5008957183200608124c677c096bf099f9"}

Deploy HA with Centos
When Openstack starts to deploy on node - go to UI and try to find puppet logs

Actual result:
There is no logs from puppet (see screen) But puppet apply is running
sysconfig for puppet looks like:
# The puppetmaster server
#PUPPET_SERVER=puppet

# If you wish to specify the port to connect to do so here
#PUPPET_PORT=8140

# Where to log to. Specify syslog to send log messages to the system log.
#PUPPET_LOG=/var/log/puppet/puppet.log

# You may specify other parameters to the puppet client here
#PUPPET_EXTRA_OPTS=--waitforcert=500

rsyslog condifg looks like:
# file is managed by puppet

### collect puppet logs of all levels in /var/log/puppet/puppet.log
if ($programname == 'puppet-apply' \
 or $programname == 'puppet-user' \
 or $programname == 'puppet-error') \
### use 6 for INFO+ msg levels, 5 for WARN+ etc.
### and $syslogseverity <= '6' \
then -/var/log/puppet/puppet.log
### stop further processing for the matched entries
& ~

root@node-7 ~]# ps afuxww | grep puppet
root 30834 0.0 0.0 103244 852 pts/0 S+ 11:21 0:00 \_ grep puppet
root 7975 9.2 28.2 548852 434072 ? Ss 11:10 1:00 /usr/bin/ruby /usr/bin/puppet apply /etc/puppet/manifests/site.pp --logdest syslog --trace

[Bogdan Dobrelya (bogdando)]

Please provide additional info
1) Logs snapshot (it is important to see how the events flow in the time)
2) Can you see the puppet-apply.log in /var/log/remote/node-* while it is absent in UI?
3) Then (and if) puppet logs would appear in UI?

[Bogdan Dobrelya (bogdando)]

4) What is the /var/log/puppet content for the node-7 (the one which has a puppet running) ?
5) Does ctrl+f5 resolve the UI issue?

[Tatyanka (tatyana-leontovich)]

2) Can you see the puppet-apply.log in /var/log/remote/node-* while it is absent in UI? -
on controllers:
[root@node-7 ~]# ls -all /var/log/puppet/
total 260
drwxr-x---. 2 puppet puppet 4096 May 5 11:11 .
drwxr-xr-x. 20 root root 4096 May 5 11:20 ..
-rw------- 1 root adm 251068 May 5 11:31 puppet.log

On other nodes (compute/ceph/cinder puppet logs show on UI without problem)

 Then (and if) puppet logs would appear in UI? - no
5) - Does ctrl+f5 resolve the UI issue? - no
Al other details I hope you can find in attached snapshot

[Bogdan Dobrelya (bogdando)]

Cannot reproduce it on #180 with Centos HA 1 controller + 1 compute

[Bogdan Dobrelya (bogdando)]

The issue could be related to the recently fixed docker container issues

[Bogdan Dobrelya (bogdando)]

I mean this one #1315865

[Bogdan Dobrelya (bogdando)]

If it is possible to access the env with the issue reproduced on it, please provide an info about
1) status of docker-rsyslog in supervisorctl
2) status of ss -untap | grep 514
while controller nodes are being deployed

[Bogdan Dobrelya (bogdando)]

I found that I have both (1), (2) in the given diagnostic snapshot and it shows:
1) there is no docker started for rsyslog, see ./localhost/commands/ps.txt
2) there is nothing on 514 port to listen for logs, see ./localhost/commands/netstat.txt
So, the source of this issue is an incorrect state of docker container for rsyslog.
The correct state should be like this http://pastebin.com/EtSJASBG

[Bogdan Dobrelya (bogdando)]

Once I cannot reproduce it on #180, I set this one as a dup of 1315865.

[Bogdan Dobrelya (bogdando)]

Apparently reproduced, see details in logs. This time tcpdump captuted pupet logs and local logs for nodes included. Note: both controller and compute nodes are missing pupet-apply logs in /var/log/remote, hence in UI as well.

[Bogdan Dobrelya (bogdando)]

Yes, I was able to reproduce it only after some env deletions and resets. So I set it as a dup