ceph docker containers do not log to /var/log/containers

[John Fulton]

1. The Ceph containers, provided by ceph-docker, that TripleO deploys log to the docker journald socket [1].

2. The OpenStack containers, based on kolla, that TripleO deploys log to /var/log/containers/ [2]

3. The TripleO deployment of the Ceph containers should be set up to log to /var/log/containers/ in order to be consistent.

The solution should be consistent with https://blueprints.launchpad.net/tripleo/+spec/containerized-services-logs

[1] As per https://docs.docker.com/engine/admin/logging/journald :

[heat-admin@overcloud-controller-0 ~]$ sudo docker ps | grep ceph
fc74d7cfcf2c docker.io/ceph/daemon:tag-build-master-jewel-centos-7 "/entrypoint.sh" 27 minutes ago Up 27 minutes ceph-mds-overcloud-controller-0
3843e264b689 docker.io/ceph/daemon:tag-build-master-jewel-centos-7 "/entrypoint.sh" 29 minutes ago Up 29 minutes ceph-mon-overcloud-controller-0
[heat-admin@overcloud-controller-0 ~]$
[heat-admin@overcloud-controller-0 ~]$ sudo journalctl CONTAINER_NAME=ceph-mon-overcloud-controller-0 | tail
Oct 06 18:07:33 overcloud-controller-0 dockerd-current[21341]: 2017-10-06 18:07:33.348867 7f1c401a6700 0 log_channel(cluster) log [INF] : pgmap v905: 1856 pgs: 1856 active+clean; 10280 MB data, 21469 MB used, 26793 GB / 26814 GB avail; 958 B/s rd, 1 op/s
Oct 06 18:07:34 overcloud-controller-0 dockerd-current[21341]: 2017-10-06 18:07:34.507103 7f1c401a6700 0 log_channel(cluster) log [INF] : pgmap v906: 1856 pgs: 1856 active+clean; 10280 MB data, 21469 MB used, 26793 GB / 26814 GB avail; 936 B/s rd, 2 op/s
Oct 06 18:07:52 overcloud-controller-0 dockerd-current[21341]: 2017-10-06 18:07:52.238909 7f1c401a6700 0 log_channel(cluster) log [INF] : pgmap v907: 1856 pgs: 1856 active+clean; 10280 MB data, 21469 MB used, 26793 GB / 26814 GB avail; 703 B/s rd, 0 op/s
Oct 06 18:07:53 overcloud-controller-0 dockerd-current[21341]: 2017-10-06 18:07:53.355183 7f1c401a6700 0 log_channel(cluster) log [INF] : pgmap v908: 1856 pgs: 1856 active+clean; 10280 MB data, 21469 MB used, 26793 GB / 26814 GB avail; 758 B/s rd, 1 op/s
Oct 06 18:08:14 overcloud-controller-0 dockerd-current[21341]: 2017-10-06 18:08:14.313672 7f1c3eb64700 0 mon.overcloud-controller-0@0(leader).data_health(12) update_stats avail 98% total 558 GB, used 10517 MB, avail 548 GB
Oct 06 18:08:17 overcloud-controller-0 dockerd-current[21341]: 2017-10-06 18:08:17.235970 7f1c401a6700 0 log_channel(cluster) log [INF] : pgmap v909: 1856 pgs: 1856 active+clean; 10280 MB data, 21469 MB used, 26793 GB / 26814 GB avail; 513 B/s rd, 0 op/s
Oct 06 18:08:18 overcloud-controller-0 dockerd-current[21341]: 2017-10-06 18:08:18.353905 7f1c401a6700 0 log_channel(cluster) log [INF] : pgmap v910: 1856 pgs: 1856 active+clean; 10280 MB data, 21469 MB used, 26793 GB / 26814 GB avail; 573 B/s rd, 0 op/s
Oct 06 18:08:32 overcloud-controller-0 dockerd-current[21341]: 2017-10-06 18:08:32.222301 7f1c401a6700 0 log_channel(cluster) log [INF] : pgmap v911: 1856 pgs: 1856 active+clean; 10280 MB data, 21469 MB used, 26793 GB / 26814 GB avail; 955 B/s rd, 1 op/s
Oct 06 18:08:33 overcloud-controller-0 dockerd-current[21341]: 2017-10-06 18:08:33.348636 7f1c401a6700 0 log_channel(cluster) log [INF] : pgmap v912: 1856 pgs: 1856 active+clean; 10280 MB data, 21469 MB used, 26793 GB / 26814 GB avail; 887 B/s rd, 1 op/s
Oct 06 18:08:34 overcloud-controller-0 dockerd-current[21341]: 2017-10-06 18:08:34.481434 7f1c401a6700 0 log_channel(cluster) log [INF] : pgmap v913: 1856 pgs: 1856 active+clean; 10280 MB data, 21469 MB used, 26793 GB / 26814 GB avail; 937 B/s rd, 2 op/s
[heat-admin@overcloud-controller-0 ~]$

[2]

(undercloud) [stack@hci-director ~]$ ansible all -m shell -b -a "ls /var/log/containers/"
192.168.1.29 | SUCCESS | rc=0 >>
aodh
ceilometer
glance
gnocchi
heat
horizon
httpd
keystone
neutron
nova
panko
swift

192.168.1.31 | SUCCESS | rc=0 >>
aodh
ceilometer
glance
gnocchi
heat
horizon
httpd
keystone
neutron
nova
panko
swift

192.168.1.26 | SUCCESS | rc=0 >>

192.168.1.27 | SUCCESS | rc=0 >>
ceilometer
neutron
nova

192.168.1.25 | SUCCESS | rc=0 >>

192.168.1.24 | SUCCESS | rc=0 >>
aodh
ceilometer
glance
gnocchi
heat
horizon
httpd
keystone
neutron
nova
panko
swift

(undercloud) [stack@hci-director ~]$

[Juan Antonio Osorio Robles (juan-osorio-robles)]

This could probably be addressed with https://blueprints.launchpad.net/tripleo/+spec/logging-stdout-rsyslog

[Juan Antonio Osorio Robles (juan-osorio-robles)]

Is this still an issue?

[John Fulton (jfulton-org)]

IMO: It's something which would be nice to fix to be consistent but I don't think it's pressing.

[Tom Barron (tpb)]

Note that we don't get the ceph or ganesha logs in SOS reports because they're not in /var/log/containers (and the journal for these isn't captured either).

[John Fulton (jfulton-org)]

As per an update to ceph-ansible 4.x [1] the ceph containers now log to /var/log/ceph [2]

[1] https://github.com/ceph/ceph-ansible/pull/4166/files

[2]

[root@edge0-distributedcomputehci-0 ceph]# ll
total 160
-rw-r--r--. 1 167 167 94109 Aug 28 12:36 ceph-osd.0.log
-rw-r--r--. 1 root root 67842 Aug 28 12:38 ceph-volume.log
[root@edge0-distributedcomputehci-0 ceph]# pwd
/var/log/ceph
[root@edge0-distributedcomputehci-0 ceph]#
[root@edge0-distributedcomputehci-0 ceph]# podman ps | grep ceph
a1d39f0b2357 docker.io/ceph/daemon:latest /opt/ceph-contain... 7 hours ago Up 7 hours ago ceph-mgr-edge0-distributedcomputehci-0
22f05d60e7e7 docker.io/ceph/daemon:latest /opt/ceph-contain... 7 hours ago Up 7 hours ago ceph-osd-0
8e6bb4705f99 docker.io/ceph/daemon:latest /opt/ceph-contain... 7 hours ago Up 7 hours ago ceph-mon-edge0-distributedcomputehci-0
[root@edge0-distributedcomputehci-0 ceph]#

[John Fulton (jfulton-org)]

Wallaby and newer has an option to have ceph log to file during deployment:

https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/features/deployed_ceph.html#ceph-configuration-options