memcached container logs are missing

[Martin André]

The memcached container doesn't have any useful logs.

[heat-admin@overcloud-controller-0 ~]$ sudo docker ps -a | grep memcached
07518cc8d59e 192.168.24.1:8787/master/centos-binary-memcached:tripleo-ci-testing "/bin/bash -c 'source" 3 days ago Up 3 days memcached
98dac9629bd2 192.168.24.1:8787/master/centos-binary-memcached:tripleo-ci-testing "/bin/bash -c 'source" 3 days ago Exited (0) 3 days ago memcached_init_logs
[heat-admin@overcloud-controller-0 ~]$ sudo ls -l /var/log/containers/ | grep memcach
[heat-admin@overcloud-controller-0 ~]$ sudo ls -l /var/log/memcached.log/
total 0
[heat-admin@overcloud-controller-0 ~]$ sudo cat /var/lib/config-data/puppet-generated/memcached/etc/sysconfig/memcached
PORT="11211"
USER="memcached"
MAXCONN="8192"
CACHESIZE="7941"
OPTIONS="-l 192.168.24.12 -U 11211 -t 8 >> /var/log/memcached.log 2>&1"
[heat-admin@overcloud-controller-0 ~]$ sudo docker exec -ti -u root memcached cat /var/log/memcached.log
cat: /var/log/memcached.log: No such file or directory
[heat-admin@overcloud-controller-0 ~]$ sudo docker logs memcached

[Alex Schultz (alex-schultz)]

I don't know if memcached actually logs anything. Looking on the undercloud (not containerized) we have the same thing but there's no actual output or file created.

[centos@undercloud log]$ ps -aux | grep memcache
centos 7111 0.0 0.0 112644 968 pts/0 S+ 17:19 0:00 grep --color=auto memcache
memcach+ 13500 0.0 0.0 339512 6284 ? Ssl Aug15 4:21 /usr/bin/memcached -p 11211 -u memcached -m 3911 -c 8192 -l 127.0.0.1 -U 11211 -t 4 >> /var/log/memcached.log 2>&1
[centos@undercloud log]$ ls -l /var/log/memcached.log
ls: cannot access /var/log/memcached.log: No such file or directory

I'm marking this low because we don't provide -v to the options so it's never logged. We would need to up the verbosity in the memcached configuration to actually have something generated.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/515618

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/515624

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/515627

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/515627
Committed: https://git.openstack.org/cgit/openstack/instack-undercloud/commit/?id=0ef7d39fc67aeb105cfc8054c231a7b956266bc6
Submitter: Zuul
Branch: master

commit 0ef7d39fc67aeb105cfc8054c231a7b956266bc6
Author: Juan Antonio Osorio Robles <email address hidden>
Date: Fri Oct 27 10:35:02 2017 +0300

    Add one level of verbosity for memcached

    This allows us to get SOME logs for it. At least warnings and errors.

    Change-Id: I25e01f659bb3c2a592b5a76163fa118baa846f77
    Related-Bug: #1720183

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/515624
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=15ad21ba08e92b302318bdc34112601ce666ff35
Submitter: Zuul
Branch: master

commit 15ad21ba08e92b302318bdc34112601ce666ff35
Author: Juan Antonio Osorio Robles <email address hidden>
Date: Fri Oct 27 10:32:20 2017 +0300

    Set verbosity by default for memcached

    This sets of one level of verbosity for memcached by default. This
    allows us to see any errors or warnings in the logs.

    Change-Id: Icc6f56c9e0ee6100286d07c8b6660a08baabb241
    Related-Bug: #1720183

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/515618
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=5020f38301a9a0a70f34878196250e24fc639dec
Submitter: Zuul
Branch: master

commit 5020f38301a9a0a70f34878196250e24fc639dec
Author: Juan Antonio Osorio Robles <email address hidden>
Date: Fri Oct 27 10:22:01 2017 +0300

    persist memcached logs in /var/log/containers/memcached/memcached.log

    We used to bind-mount /var/log/memcached.log, but this resulted in the
    file being createdin the memcached container as a directory, since this
    file didn't exist.

    This commit takes the approach of other containers and gets the logs to
    a memcached directory in /var/log/containers.

    Change-Id: I926b65fa557ad56b4faa2be34452b58f7b01247a
    Closes-Bug: #1720183

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/516205

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/516206

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/516270

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/516271

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/516271
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=5f79927c58d342cf29417cef4b6e9111854224af
Submitter: Zuul
Branch: stable/pike

commit 5f79927c58d342cf29417cef4b6e9111854224af
Author: Juan Antonio Osorio Robles <email address hidden>
Date: Fri Oct 27 10:22:01 2017 +0300

    persist memcached logs in /var/log/containers/memcached/memcached.log

    We used to bind-mount /var/log/memcached.log, but this resulted in the
    file being createdin the memcached container as a directory, since this
    file didn't exist.

    This commit takes the approach of other containers and gets the logs to
    a memcached directory in /var/log/containers.

    Change-Id: I926b65fa557ad56b4faa2be34452b58f7b01247a
    Closes-Bug: #1720183
    (cherry picked from commit 5020f38301a9a0a70f34878196250e24fc639dec)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/516270
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=db767bde2f1610812ed460cdc05722f2d567d552
Submitter: Zuul
Branch: stable/pike

commit db767bde2f1610812ed460cdc05722f2d567d552
Author: Juan Antonio Osorio Robles <email address hidden>
Date: Fri Oct 27 10:32:20 2017 +0300

    Set verbosity by default for memcached

    This sets of one level of verbosity for memcached by default. This
    allows us to see any errors or warnings in the logs.

    Change-Id: Icc6f56c9e0ee6100286d07c8b6660a08baabb241
    Related-Bug: #1720183
    (cherry picked from commit 15ad21ba08e92b302318bdc34112601ce666ff35)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/516205
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=85234209307e5917be40ca78342617d003429858
Submitter: Zuul
Branch: master

commit 85234209307e5917be40ca78342617d003429858
Author: Juan Antonio Osorio Robles <email address hidden>
Date: Mon Oct 30 10:00:49 2017 +0200

    Make memcached logs verbosity configurable

    This adds the MemcachedDebug flag which makes memcached logs verbose.
    Also, the service now reacts to the global Debug flag.

    Related-Bug: #1720183
    Change-Id: Ie8120359a12f734aad584b083c8362dd21b70dea

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/516206
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=ca1fc5848661aacbf14b52e33879190c133c8e48
Submitter: Zuul
Branch: master

commit ca1fc5848661aacbf14b52e33879190c133c8e48
Author: Juan Antonio Osorio Robles <email address hidden>
Date: Mon Oct 30 10:04:18 2017 +0200

    Force memcached container log to file

    We were relying on the sysconfig options to set the memcached log file,
    however, this is not happening, as the redirection is being taken as an
    option and ends up being ignored by the memcached command. So instead,
    we set the redirection in the container template.

    Change-Id: Ic94e3fd7884d518eb9558c53acdc6b294823cd0a
    Closes-Bug: #1720183

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/516875

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/516876

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/516876
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=39d4fa32949686054c4b6fd375799a937285775f
Submitter: Zuul
Branch: stable/pike

commit 39d4fa32949686054c4b6fd375799a937285775f
Author: Juan Antonio Osorio Robles <email address hidden>
Date: Mon Oct 30 10:04:18 2017 +0200

    Force memcached container log to file

    We were relying on the sysconfig options to set the memcached log file,
    however, this is not happening, as the redirection is being taken as an
    option and ends up being ignored by the memcached command. So instead,
    we set the redirection in the container template.

    Change-Id: Ic94e3fd7884d518eb9558c53acdc6b294823cd0a
    Closes-Bug: #1720183
    (cherry picked from commit ca1fc5848661aacbf14b52e33879190c133c8e48)

[OpenStack Infra (hudson-openstack)]

Change abandoned by Juan Antonio Osorio Robles (<email address hidden>) on branch: stable/pike
Review: https://review.openstack.org/516875
Reason: you're right

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 7.0.3 release.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/518020

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/518020
Committed: https://git.openstack.org/cgit/openstack/instack-undercloud/commit/?id=de4dd54384a34839d38fc42fd1e35442d34ef810
Submitter: Zuul
Branch: stable/pike

commit de4dd54384a34839d38fc42fd1e35442d34ef810
Author: Juan Antonio Osorio Robles <email address hidden>
Date: Fri Oct 27 10:35:02 2017 +0300

    Add one level of verbosity for memcached

    This allows us to get SOME logs for it. At least warnings and errors.

    Change-Id: I25e01f659bb3c2a592b5a76163fa118baa846f77
    Related-Bug: #1720183
    (cherry picked from commit 0ef7d39fc67aeb105cfc8054c231a7b956266bc6)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 7.0.4 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 8.0.0.0b2 development milestone.