Astute, RPC Consumer, Assassin logs not available on UI

[Anastasia Palkina]

1. Create new environment by default
2. Enable Ceilometer on Settings tab
3. Add 1 controller+mongo, 1 cinder
4. Open "Logs" tab
5. Select Astute, RPC Consumer, Assassin logs. Error appears: Unable to fetch logs. (see screen)
But these logs available on master-node

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  openstack_version: "2015.1.0-8.0"
  api: "1.0"
  build_number: "264"
  build_id: "264"
  fuel-nailgun_sha: "0e09dce510927f2cc490b898e5fe3f813bd791be"
  python-fuelclient_sha: "f033192b84263f0e699458a4274289a5198ae7e4"
  fuel-agent_sha: "660c6514caa8f5fcd482f1cc4008a6028243e009"
  fuel-nailgun-agent_sha: "a33a58d378c117c0f509b0e7badc6f0910364154"
  astute_sha: "48fd58676debcc85951db68df6d77c22daa55e52"
  fuel-library_sha: "ab7e51f345ffb7c256e0f61addcf86553d7c3867"
  fuel-ostf_sha: "23b7ae2a1a57de5a3e1861ffb7805394ca339cc2"
  fuel-mirror_sha: "6534117233a5bdc51d7d47361bc7d511e4b11e6f"
  fuelmenu_sha: "fcb15df4fd1a790b17dd78cf675c11c279040941"
  shotgun_sha: "a0bd06508067935f2ae9be2523ed0d1717b995ce"
  network-checker_sha: "a3534f8885246afb15609c54f91d3b23d599a5b1"
  fuel-upgrade_sha: "1e894e26d4e1423a9b0d66abd6a79505f4175ff6"
  fuelmain_sha: "26adf12c320936a97a9b0a84169a6e58c530e848"

[Vladimir Sharshov (vsharshov)]

Partial fix for Astute: https://review.openstack.org/#/c/254985/

[Vladimir Sharshov (vsharshov)]

Also i can confirm logs problem with RPC Consumer and Assassin logs

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/254985
Committed: https://git.openstack.org/cgit/openstack/fuel-astute/commit/?id=81d9319d4432ae675a412a8c81f1199c66796f5b
Submitter: Jenkins
Branch: master

commit 81d9319d4432ae675a412a8c81f1199c66796f5b
Author: Vladimir Sharshov (warpc) <email address hidden>
Date: Wed Dec 9 01:29:49 2015 +0300

    Setup correct log patch for Astute

    Correct path: '/var/log/astute/astute.log'

    Change-Id: I0dce97acaff3be8f0b90d92721c876dc9c6374f4
    Partial-Bug: #1523476

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/257277

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/257277
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=6417bca00a4efc721b2d75dda9ba55f2995c2bb2
Submitter: Jenkins
Branch: master

commit 6417bca00a4efc721b2d75dda9ba55f2995c2bb2
Author: Vladimir Sharshov (warpc) <email address hidden>
Date: Mon Dec 14 14:07:37 2015 +0300

    Setup correct log path and debug level for Astute

    Correct log path: '/var/log/astute/astute.log'
    Correct log level: 'debug'

    Change-Id: Ifc05b35369691353a631328531dea36c8d7a55d8
    Partial-Bug: #1523476
    Closes-Bug: #1524368

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/257882

[Vladimir Sharshov (vsharshov)]

We need systemd expert to continue.

In nailgun containers on master node 2 services:

- receiverd;
- assassind.

Do not setup to write logs on follow paths:

- /var/log/nailgun/receiverd.log;
- /var/log/nailgun/assassind.log.

[Dmitry Teselkin (teselkin-d)]

This question was discussed in [1]. It should be resolved on component's side, since there is python logger that can do the work.

[1] https://bugs.launchpad.net/fuel/+bug/1525153

[Matthew Mosesohn (raytrac3r)]

Python logging to file needs to be done by python team.

[Matthew Mosesohn (raytrac3r)]

Just clarifying here.

Comment 9 is incredibly misleading. Maybe @vsharshov you would like to rephrase it?

As far as I understand, we need to write logs directly from the daemons to /var/log/nailgun/${service_name}.log
The list of impacted logs are:
assassind.log
oswl_flavor_collectord.log
oswl_image_collectord.log
oswl_keystone_user_collectord.log
oswl_tenant_collectord.log
oswl_vm_collectord.log
oswl_volume_collectord.log
receiverd.log

supervisord did this for us previously, but now services are logging to stdout and that information gets lost in the ethers because we run systemd without rsyslog in the nailgun container.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/267000

[Ihor Kalnytskyi (ikalnytskyi)]

FYI:

There's another patch that is intended to fix the same problem from Ivan Suzdal -

  https://review.openstack.org/#/c/255272/4

However, I don't like it patch. Despite the fact it fixes the issue on systemd side, it fixes it ugly - by wrapping services in bash process, and redirecting stderr/stdout into hardcoded file. It's done because apparently systemd doesn't support redirecting of standard output streams to files. The approach they recommend is to configure logging on application side.

So I think that should be fixed on application side, besides that would be convenient to setup output log filename through some settings.yaml (do not hardcode it somewhere).

[Mikhail Samoylov (msamoylov)]

Hi.
In addition
We need in nailgun docker container '/var/log/nailgun/statsenderd.log' file.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/268217

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ivan Suzdal (<email address hidden>) on branch: master
Review: https://review.openstack.org/255272
Reason: should be abandoned

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/267000
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=31b3dbb7ae4598436f49db38f7aa58d7dfb5e049
Submitter: Jenkins
Branch: master

commit 31b3dbb7ae4598436f49db38f7aa58d7dfb5e049
Author: Artem Roma <email address hidden>
Date: Wed Jan 13 17:51:46 2016 +0200

    Add logging setup for receiverd and assassind

    Since all services inside containers are now started via systemd we need
    add logging handling which in past was done by supervisord. Thus such
    settings are added for rpc receiver and assassin services. Loggers are
    set up independently from nailgun root logger as writing to files are
    required for mentioned components.

    Change-Id: I21136108d426531f572940dfc9f75fe153111d41
    Partial-Bug: #1523476

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/268217
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=45950e02988bd3092168d1a0bdf2acc198bb6eff
Submitter: Jenkins
Branch: master

commit 45950e02988bd3092168d1a0bdf2acc198bb6eff
Author: Artem Roma <email address hidden>
Date: Fri Jan 15 18:21:17 2016 +0200

    Add handling of logs for fuel statistics collectors and sender

    Separate logger added for statistic module. Logs for collecting process
    of each oswl resource now are written to separate files. This is
    possible because collecting for particular resource is started in
    separate process by systemd and so logs separation is achieved by
    just using filehandler with specific name.

    Change-Id: I35a1d03441639821233d068fbde57a251d39a721
    Closes-Bug: #1523476

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/274616

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/274617

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/274616
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=a3723d868b078c13cc8c07e97f42fd004538f330
Submitter: Jenkins
Branch: stable/8.0

commit a3723d868b078c13cc8c07e97f42fd004538f330
Author: Artem Roma <email address hidden>
Date: Wed Jan 13 17:51:46 2016 +0200

    Add logging setup for receiverd and assassind

    Since all services inside containers are now started via systemd we need
    add logging handling which in past was done by supervisord. Thus such
    settings are added for rpc receiver and assassin services. Loggers are
    set up independently from nailgun root logger as writing to files are
    required for mentioned components.

    Change-Id: I21136108d426531f572940dfc9f75fe153111d41
    Partial-Bug: #1523476

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/274617
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=90f030f506ef44e28cd5ce463f3f35d47b3f37a8
Submitter: Jenkins
Branch: stable/8.0

commit 90f030f506ef44e28cd5ce463f3f35d47b3f37a8
Author: Artem Roma <email address hidden>
Date: Fri Jan 15 18:21:17 2016 +0200

    Add handling of logs for fuel statistics collectors and sender

    Separate logger added for statistic module. Logs for collecting process
    of each oswl resource now are written to separate files. This is
    possible because collecting for particular resource is started in
    separate process by systemd and so logs separation is achieved by
    just using filehandler with specific name.

    Change-Id: I35a1d03441639821233d068fbde57a251d39a721
    Closes-Bug: #1523476

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/275770

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/275777

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/275770
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=571173588774be5232adf08d74bf40d5775a6131
Submitter: Jenkins
Branch: master

commit 571173588774be5232adf08d74bf40d5775a6131
Author: Artem Roma <email address hidden>
Date: Wed Feb 3 17:41:49 2016 +0200

    Fix logic of prepare_logger function

    Logic of adding file handler for logger being setting up fixed.

    Change-Id: I027440083eeb4ea23a5316ad3f9a38824e1aed1b
    Closes-Bug: #1523476

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/275777
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=ad8f5cb8f3584b74360f8fdd99ea047a5f48e4af
Submitter: Jenkins
Branch: stable/8.0

commit ad8f5cb8f3584b74360f8fdd99ea047a5f48e4af
Author: Artem Roma <email address hidden>
Date: Wed Feb 3 17:41:49 2016 +0200

    Fix logic of prepare_logger function

    Logic of adding file handler for logger being setting up fixed.

    Change-Id: I027440083eeb4ea23a5316ad3f9a38824e1aed1b
    Closes-Bug: #1523476

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/276168

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/276215

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/276215
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=5f435007a7d84e2fffd0bd0f41cfaf718dbc68c1
Submitter: Jenkins
Branch: stable/8.0

commit 5f435007a7d84e2fffd0bd0f41cfaf718dbc68c1
Author: Artem Roma <email address hidden>
Date: Thu Feb 4 13:18:55 2016 +0200

    Move logging set up to receiverd.run function

    Logging set up moved to rpc submodule entry point - receiverd.run
    function. This fixes issue when logging fails inside receiver.py module
    due it was not properly set.

    Change-Id: I2a1a0db15d3d3d3f5d63c69c9adc3d2557acc10b
    Closes-Bug: #1523476

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/276168
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=88cd958712dc5f5238a87f65979b77b6266a9f8b
Submitter: Jenkins
Branch: master

commit 88cd958712dc5f5238a87f65979b77b6266a9f8b
Author: Artem Roma <email address hidden>
Date: Thu Feb 4 13:18:55 2016 +0200

    Move logging set up to receiverd.run function

    Logging set up moved to rpc submodule entry point - receiverd.run
    function. This fixes issue when logging fails inside receiver.py module
    due it was not properly set.

    Change-Id: I2a1a0db15d3d3d3f5d63c69c9adc3d2557acc10b
    Closes-Bug: #1523476

[Nastya Urlapova (aurlapova)]

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "529"
  build_id: "529"
  fuel-nailgun_sha: "baec8643ca624e52b37873f2dbd511c135d236d9"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "658be72c4b42d3e1436b86ac4567ab914bfb451b"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "e2d79330d5d708796330fac67722c21f85569b87"
  fuel-ostf_sha: "3bc76a63a9e7d195ff34eadc29552f4235fa6c52"
  fuel-mirror_sha: "fb45b80d7bee5899d931f926e5c9512e2b442749"
  fuelmenu_sha: "e071216cb214e34b4d861478033425ee6a54a3be"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "a43cf96cd9532f10794dce736350bf5bed350e9d"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "a365f05b903368225da3fea9aa42afc1d50dc9b4"

[ElenaRossokhina (esolomina)]

Verified on fuel-9.0-91-2016-03-21_02-00-00.iso

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ivan Suzdal (<email address hidden>) on branch: master
Review: https://review.openstack.org/257882