Logs are missing in /var/log/remote/node-X.test.domain.local

[Vladimir Khlyunev]

ISO 263
Snapshot https://product-ci.infra.mirantis.net/job/8.0.system_test.ubuntu.ha_neutron_destructive/65/artifact/logs/fail_error_ha_neutron_check_alive_rabbit-fuel-snapshot-2015-12-07_03-48-43.tar.xz

Steps
1) Deploy any cluster
2) Generate diag.snapshot and open it
or
2) Go to /var/log/remote/node-X.test.domain.local

Result:
Only puppet-apply log and bootstrap logs are presented.

This is blocker for debugging swarm jobs.

[Alexandr Kostrikov (akostrikov-mirantis)]

@bpiotrowski can help with that.

[Dmitry Teselkin (teselkin-d)]

It's not a network issue (https://bugs.launchpad.net/fuel/+bug/1523897)

[Ivan Suzdal (isuzdal)]

Seems like wrong fromhost regex in rsyslog
Before:
3433.140055678:7f6915822700: Filter: check for property 'fromhost' (value 'node-8.domain.tld') regex '^[0-9]': FALSE
After:
3490.784190932:7f6917ce7700: Filter: check for property 'fromhost' (value 'node-8.domain.tld') regex '^node-[0-9].*': TRUE

[Ivan Suzdal (isuzdal)]

There is request which should fix this issue
https://review.openstack.org/#/c/254867/

[Bogdan Dobrelya (bogdando)]

Marked as regression. The current match pattern have been working since the 3.1 release, so something get this broken in the 8.0

[Bogdan Dobrelya (bogdando)]

AFAICT, the /var/log/remote/node* destinations have been created by Fuel shotgun as symlinks. And real logs come from the IP addresses based sources.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/254867
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=d71c9b864bfd0931a01ed85fac743d8bad4c1744
Submitter: Jenkins
Branch: master

commit d71c9b864bfd0931a01ed85fac743d8bad4c1744
Author: Ivan Suzdal <email address hidden>
Date: Tue Dec 8 19:55:08 2015 +0300

    Use IP based regex for rsyslog

    From rsyslogd documentation:

      fromhost
        hostname of the system the message was received from
        (in a relay chain, this is the system immediately in
        front of us and not necessarily the original sender).
        This is a DNS-resolved name, except if that is not
        possible or DNS resolution has been disabled.

    So it might be either hostname or IP address, however,
    regex filter will work for IP address only.

    This commit forces IP usage of IP addresses only.

    Closes-Bug: #1523488

    Change-Id: I46e00f0273671a92126ce29a5bcebac257bedf03

[Alexandr Kostrikov (akostrikov-mirantis)]

Used it in test and logs are present for all services.