Ubuntu logs are incomplete in snapshot

[Aleksandra Fedorova]

Affects Ubuntu CI gates only

According to astute.log deployment of the node failed:

   2014-04-16T17:03:21 debug: [8443] Node "1" has failed to deploy. There is no more retries for puppet run.

But no errors have been logged to puppet-apply.log of the node.

See https://fuel-jenkins.mirantis.com/job/master_fuellib_review_systest_ubuntu/434/ for details. Logs attached.

[Bogdan Dobrelya (bogdando)]

Already fixed (see dup issue)
Make sure CI gate includes https://review.openstack.org/#/c/85970/ astute fix

[Aleksandra Fedorova (bookwar)]

The patch you link to is merged at 11 Apr. But Fuel CI used fuel-5.0-107 ISO which was build from master yesterday (16 Apr).

This means the patch is already there, but logs are still incomplete.

[Bogdan Dobrelya (bogdando)]

The 2nd patch addressed for the same issue wasn't merged yet https://review.openstack.org/#/c/85975. I hope, once merged - it should resolve the issue completely

[Bogdan Dobrelya (bogdando)]

Logs snapshot now provide puppet logs from the local nodes as well.
Since that, this issue is not a blocker for logs analyzing in CI jobs.
That also allows to compare the "cut" logs from /remote destination vs local ones

[Bogdan Dobrelya (bogdando)]

Almost all Ubuntu CI gate failures are affected by this issue, example https://fuel-jenkins.mirantis.com/job/master_fuellib_review_systest_ubuntu/1251/

Here you can compare the local and remote logs and analyze the possible root cause of such cutting it off

[Bogdan Dobrelya (bogdando)]

As far as I can see all logs is missing in snapshot starting from the "cut moment", so the issue is related to all remote logs delivery not only the puppet ones

[Bogdan Dobrelya (bogdando)]

run_user:run_group should be syslog:syslog for Ubuntu. That is a root cause of missing logs (in Fuel, all syslog templates are for root:root)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/102870

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/5.0
Review: https://review.openstack.org/103076

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/4.1
Review: https://review.openstack.org/103095

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/103076
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=0271711fafb3c558a41e84c37c6dadb571e59619
Submitter: Jenkins
Branch: stable/5.0

commit 0271711fafb3c558a41e84c37c6dadb571e59619
Author: Bogdan Dobrelya <email address hidden>
Date: Thu Jun 26 19:02:02 2014 +0300

    Use syslog:syslog for Ubuntu

    In Fuel, all syslog templates are for root:root. But
    run_user:run_group should be syslog:syslog for Ubuntu.

    poke ci
    Closes-bug: #1308684

    Change-Id: I099560f22b409cf60a6c565af8a1ab13bac48a7c
    Signed-off-by: Bogdan Dobrelya <email address hidden>

[Meg McRoberts (dreidellhasa)]

Should this be documented as a Fixed Issue in the Release Notes? If so, did you fix the owner/group permissions for syslog on Ubuntu or do users need to configure this on the Settings page?

[Bogdan Dobrelya (bogdando)]

AFAIK, the issue was not introduced in the previous releases. It was fixed for 5.0.1 only, other targets are on review. So doc impact seems zero.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/102870
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=ffd87c7a17d2f21e096b9f6462fd9a9f3820cd2b
Submitter: Jenkins
Branch: master

commit ffd87c7a17d2f21e096b9f6462fd9a9f3820cd2b
Author: Bogdan Dobrelya <email address hidden>
Date: Thu Jun 26 19:02:02 2014 +0300

    Use syslog:syslog for Ubuntu

    In Fuel, all syslog templates are for root:root. But
    run_user:run_group should be syslog:syslog for Ubuntu.

    Closes-bug: #1308684

    Change-Id: I099560f22b409cf60a6c565af8a1ab13bac48a7c
    Signed-off-by: Bogdan Dobrelya <email address hidden>

[Bogdan Dobrelya (bogdando)]

Additional fix required for syslog files created by the custom logging rules ,e.g.:
-rw-r--r-- 1 root root 21K Jul 16 00:24 /var/log/alternatives.log
-rw-r--r-- 1 root root 532 Jul 16 00:16 /var/log/boot.log
-rw-r----- 1 root adm 0 Jul 16 00:18 /var/log/ceilometer-all.log
-rw-r----- 1 root adm 0 Jul 16 00:18 /var/log/cinder-all.log
-rw-r--r-- 1 root root 0 Jul 16 00:19 /var/log/corosync.log
-rw-r----- 1 root adm 0 Jul 16 00:18 /var/log/cron.log
-rw-r----- 1 root adm 0 Jul 16 00:18 /var/log/daemon.log
-rw-r----- 1 root adm 0 Jul 16 00:18 /var/log/dashboard.log
-rw-r--r-- 1 root root 478K Jul 16 00:36 /var/log/dpkg.log
-rw-r----- 1 root adm 0 Jul 16 00:18 /var/log/glance-all.log
-rw-r----- 1 root adm 0 Jul 16 08:31 /var/log/haproxy.log
-rw-r----- 1 root adm 0 Jul 16 00:18 /var/log/heat-all.log
-rw-r----- 1 root adm 0 Jul 16 00:18 /var/log/keystone-all.log
-rw-r--r-- 1 root root 1.5M Jul 16 08:35 /var/log/mcollective.log
-rw-r--r-- 1 root root 149 Jul 16 00:24 /var/log/memcached.log
-rw-r----- 1 root adm 0 Jul 16 00:18 /var/log/murano-all.log
-rw-r----- 1 root adm 0 Jul 16 00:18 /var/log/mysqld.log
-rw-r--r-- 1 root root 12M Jul 16 08:35 /var/log/nailgun-agent.log
-rw-r----- 1 root adm 0 Jul 16 00:18 /var/log/neutron-all.log
-rw-r----- 1 root adm 0 Jul 16 00:18 /var/log/nova-all.log
-rw-r----- 1 root adm 0 Jul 16 00:18 /var/log/pacemaker.log
-rw-r----- 1 root adm 0 Jul 16 00:18 /var/log/sahara-all.log
-rw-r----- 1 root adm 0 Jul 16 00:18 /var/log/user.log
-rw-r----- 1 root adm 0 Jul 16 00:18 /var/log/zabbix_server.log

Such files will be locked for syslog once he changed its running user

[Bogdan Dobrelya (bogdando)]

Note that since the first merged patch successfully resolves *remote* logging issues for 5.0.1 and 5.1, we could lower the issue to high (remote logs OK, only its local copies affected). But 4.1.x is still requiring both patches, thus, staying at critical

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/107273

[Bogdan Dobrelya (bogdando)]

How to test the patchset:
At deployed OSt nodes with Ubuntu, the output for "find /var/log/ -type f -size 0 -user root -group adm | xargs -n1 ls -lh"
should be empty

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/5.0
Review: https://review.openstack.org/107294

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/107273
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=c3efecb85c45b9f4987163ea8bff737bf317762e
Submitter: Jenkins
Branch: master

commit c3efecb85c45b9f4987163ea8bff737bf317762e
Author: Bogdan Dobrelya <email address hidden>
Date: Wed Jul 16 12:27:30 2014 +0300

    Update logfile ownership for syslog in Ubuntu

    With $FileGroup:$FileOwner = root:adm, we would have some
    syslog logfiles inaccessible then syslog running user would
    be changed from root to syslog.
    root:adm should not be set for syslog logfiles since
    we use syslog:syslog as drop-provilege-to as well
    (http://www.rsyslog.com/doc/droppriv.html)

    Closes-bug: #1308684

    Change-Id: I4ab98cd77d29d986d202051b7e66c8abb13e4c15
    Signed-off-by: Bogdan Dobrelya <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/107294
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=3315d10e113c3778c1f35c0c90ea88e9da92dcf8
Submitter: Jenkins
Branch: stable/5.0

commit 3315d10e113c3778c1f35c0c90ea88e9da92dcf8
Author: Bogdan Dobrelya <email address hidden>
Date: Wed Jul 16 12:27:30 2014 +0300

    Update logfile ownership for syslog in Ubuntu

    With $FileGroup:$FileOwner = root:adm, we would have some
    syslog logfiles inaccessible then syslog running user would
    be changed from root to syslog.
    root:adm should not be set for syslog logfiles since
    we use syslog:syslog as drop-provilege-to as well
    (http://www.rsyslog.com/doc/droppriv.html)

    Closes-bug: #1308684

    Change-Id: I4ab98cd77d29d986d202051b7e66c8abb13e4c15
    Signed-off-by: Bogdan Dobrelya <email address hidden>

[Bogdan Dobrelya (bogdando)]

If there would be 4.1.2 by any chances, update the 4.1.2 milestone target status as in progress as well

[OpenStack Infra (hudson-openstack)]

Change abandoned by Bogdan Dobrelya (<email address hidden>) on branch: stable/4.1
Review: https://review.openstack.org/103095

[Dennis Dmitriev (ddmitriev)]

Fix released:
api: '1.0'
astute_sha: f655ee86ebf0359b014f00cff63d0aaf15c65308
auth_required: true
build_id: 2014-08-01_10-37-32
build_number: '382'
feature_groups:
- mirantis
fuellib_sha: 5571b86a667e28d4c9770fcce4d32163dee5a710
fuelmain_sha: 7990f5bfa7fea5b74ebf0402b1918109b9bc505b
nailgun_sha: 51f32395eebe2514e78eb7e0a85e694826be40d6
ostf_sha: 15f3be5fbafb7a8c7075b5077a5074a50e679c25
production: docker
release: '5.1'

[OpenStack Infra (hudson-openstack)]

Change abandoned by Bogdan Dobrelya (<email address hidden>) on branch: stable/4.1
Review: https://review.openstack.org/103095

[Denis Meltsaykin (dmeltsaykin)]

CI problems are not relevant anymore. Closing as Won't fix.