Nova-compute logs are not present on Fuel master node

[Ilya Shakhat]

Detailed bug description:
  There are no nova-compute logs on Fuel master node

[root@fuel ~]# ls -al /var/log/remote/node-2.domain.tld/ | grep nova
[root@fuel ~]#

Steps to reproduce:
 1. Enable debug logging for OpenStack services
 2. Deploy
 3. Check the presence of logs at Fuel master node

Expected results:
 Nova-compute logs should be present at Fuel master node

Actual result:
 Logs are not present

Reproducibility:
 always

Workaround:
 N/A

Impact:
 debuggability, support

Description of the environment:
 Operation system: Ubuntu 14.04
 Versions of components:
[root@fuel ~]# fuel2 fuel-version
api: '1'
auth_required: true
feature_groups: []
openstack_version: mitaka-9.0
release: '9.0'

 Reference architecture: MOS
 Network model: no matter
 Related projects installed: none

[Maksim Malchuk (mmalchuk)]

tested with latest stable 9.0 ISO#297:

[root@fuel ~]# shotgun2 short-report
cat /etc/fuel_build_id:
 297
cat /etc/fuel_build_number:
 297
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 mitaka-9.0
rpm -qa | egrep 'fuel|astute|network-checker|nailgun|packetary|shotgun':
 fuel-release-9.0.0-1.mos6344.noarch
 fuel-provisioning-scripts-9.0.0-1.mos8680.noarch
 fuel-mirror-9.0.0-1.mos133.noarch
 fuel-openstack-metadata-9.0.0-1.mos8680.noarch
 python-fuelclient-9.0.0-1.mos313.noarch
 fuel-9.0.0-1.mos6344.noarch
 fuel-utils-9.0.0-1.mos8337.noarch
 fuel-nailgun-9.0.0-1.mos8680.noarch
 rubygem-astute-9.0.0-1.mos742.noarch
 fuel-library9.0-9.0.0-1.mos8337.noarch
 network-checker-9.0.0-1.mos72.x86_64
 fuel-agent-9.0.0-1.mos276.noarch
 fuel-ui-9.0.0-1.mos2676.noarch
 fuel-setup-9.0.0-1.mos6344.noarch
 nailgun-mcagents-9.0.0-1.mos742.noarch
 fuel-misc-9.0.0-1.mos8337.noarch
 python-packetary-9.0.0-1.mos133.noarch
 fuelmenu-9.0.0-1.mos269.noarch
 fuel-bootstrap-cli-9.0.0-1.mos276.noarch
 fuel-migrate-9.0.0-1.mos8337.noarch
 shotgun-9.0.0-1.mos88.noarch
 fuel-notify-9.0.0-1.mos8337.noarch
 fuel-ostf-9.0.0-1.mos933.noarch
[root@fuel ~]# ls -al /var/log/remote/node-2.domain.tld/ | grep nova
-rw-------. 1 root root 6039592 May 9 21:20 nova-api.log
-rw-------. 1 root root 6723 May 9 17:01 nova-cert.log
-rw-------. 1 root root 13820 May 9 17:01 nova-conductor.log
-rw-------. 1 root root 8068 May 9 17:01 nova-consoleauth.log
-rw-------. 1 root root 2371 May 9 13:42 nova-novncproxy.log
-rw-------. 1 root root 198190 May 9 21:19 nova-scheduler.log
[root@fuel ~]#

[Maksim Malchuk (mmalchuk)]

Please provide the ISO version and/or diagnostic snapshot.

[Ilya Shakhat (shakhat)]

@Maksim: ISO 245.
It looks like you've checked controller logs, pleases check logs from compute node.

[Maciej Relewicz (rlu)]

on 257 it's ok, controller and compute nova logs are available in /var/log/remote/node-*.test.domain.local

[root@nailgun ~]# fuel nodes
id | status | name | cluster | ip | mac | roles | pending_roles | online | group_id
---|----------|------------------|---------|------------|-------------------|------------|---------------|--------|---------
2 | ready | Untitled (35:46) | 1 | 10.109.0.4 | 64:51:98:e6:35:46 | compute | | True | 1
1 | ready | Untitled (e7:95) | 1 | 10.109.0.6 | 64:50:c1:a1:e7:95 | controller | | True | 1
4 | discover | Untitled (1f:c6) | None | 10.109.0.5 | 64:87:7f:13:1f:c6 | | | False | None
3 | discover | Untitled (c7:11) | None | 10.109.0.3 | 64:bf:04:ef:c7:11 | | | False | None
[root@nailgun ~]# ll /var/log/remote/node-1.test.domain.local/ | grep nova
-rw-------. 1 root root 101063 May 10 12:53 nova-api.log
-rw-------. 1 root root 707736 May 10 12:29 nova-api.log.1.gz
-rw-------. 1 root root 710343 May 8 20:00 nova-api.log.2.gz
-rw-------. 1 root root 709583 May 7 03:29 nova-api.log.3.gz
-rw-------. 1 root root 1174 May 5 11:22 nova-cert.log
-rw-------. 1 root root 1480 May 5 11:22 nova-conductor.log
-rw-------. 1 root root 1181 May 5 11:22 nova-consoleauth.log
-rw-------. 1 root root 24831 May 5 11:22 nova-manage.log
-rw-------. 1 root root 1194 May 5 11:22 nova-novncproxy.log
-rw-------. 1 root root 1595452 May 10 12:51 nova-scheduler.log
[root@nailgun ~]# ll /var/log/remote/node-2.test.domain.local/ | grep nova
-rw-------. 1 root root 3743800 May 10 12:52 nova-compute.log
-rw-------. 1 root root 406157 May 9 04:59 nova-compute.log.1.gz
[root@nailgun ~]# shotgun2 short-report
cat /etc/fuel_build_id:
 257
cat /etc/fuel_build_number:
 257
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 mitaka-9.0
rpm -qa | egrep 'fuel|astute|network-checker|nailgun|packetary|shotgun':
 fuel-release-9.0.0-1.mos6342.noarch
 fuel-misc-9.0.0-1.mos8329.noarch
 fuel-mirror-9.0.0-1.mos133.noarch
 shotgun-9.0.0-1.mos88.noarch
 fuel-openstack-metadata-9.0.0-1.mos8671.noarch
 fuel-notify-9.0.0-1.mos8329.noarch
 fuel-ostf-9.0.0-1.mos928.noarch
 fuel-provisioning-scripts-9.0.0-1.mos8671.noarch
 python-fuelclient-9.0.0-1.mos313.noarch
 fuel-9.0.0-1.mos6342.noarch
 fuel-utils-9.0.0-1.mos8329.noarch
 fuel-nailgun-9.0.0-1.mos8671.noarch
 rubygem-astute-9.0.0-1.mos741.noarch
 fuel-library9.0-9.0.0-1.mos8329.noarch
 network-checker-9.0.0-1.mos72.x86_64
 fuel-agent-9.0.0-1.mos273.noarch
 fuel-ui-9.0.0-1.mos2676.noarch
 fuel-setup-9.0.0-1.mos6342.noarch
 nailgun-mcagents-9.0.0-1.mos741.noarch
 python-packetary-9.0.0-1.mos133.noarch
 fuelmenu-9.0.0-1.mos269.noarch
 fuel-bootstrap-cli-9.0.0-1.mos273.noarch
 fuel-migrate-9.0.0-1.mos8329.noarch

[Maksim Malchuk (mmalchuk)]

Ilya, all logs in place:

[root@fuel ~]# ls -la /var/log/remote/node-{4,5}*/nova-compute.log
-rw-------. 1 root root 1214817 May 11 10:30 /var/log/remote/node-4.domain.tld/nova-compute.log
-rw-------. 1 root root 1215784 May 11 10:31 /var/log/remote/node-5.domain.tld/nova-compute.log
[root@fuel ~]#

So a bug didn't reproduced.

[Ilya Shakhat (shakhat)]

Maksim, on MOS 9 ISO 309 with 1 controller and 1 compute I get:

[root@fuel ~]# fuel node | grep compute
1 | ready | compute_0 | 1 | 10.20.0.3 | 52:54:00:75:54:7c | compute | | True | 1

[root@fuel ~]# find /var/log/remote/ -name nova*
/var/log/remote/node-2.domain.tld/nova-api.log
/var/log/remote/node-2.domain.tld/nova-cert.log
/var/log/remote/node-2.domain.tld/nova-manage.log
/var/log/remote/node-2.domain.tld/nova-scheduler.log
/var/log/remote/node-2.domain.tld/nova-conductor.log
/var/log/remote/node-2.domain.tld/nova-novncproxy.log
/var/log/remote/node-2.domain.tld/nova-consoleauth.log

Also there are no /var/log/nova-all.log at the compute node.

[Maksim Malchuk (mmalchuk)]

Could you please provide the diagnostic snapshot, may be the problem with deployment.
Is nova services work with your configuration?

[Maksim Malchuk (mmalchuk)]

The bug is reproduced only when OpenStack debug is enabled.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/318332

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/318332
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=f989647cc5e6cb4a17db57f5c5d82eb92676efd7
Submitter: Jenkins
Branch: master

commit f989647cc5e6cb4a17db57f5c5d82eb92676efd7
Author: Maksim Malchuk <email address hidden>
Date: Wed May 18 16:11:57 2016 +0300

    Remove duplicate 'default_log_levels' declaration

    Duplicate declaration 'default_log_levels' for the '::nova::logging'
    produces an issue when nova-compute logs not created on the compute
    nodes in case the 'debug' option is enabled during deployment. This
    commit removes this duplication because it already declared in the
    globals. Also, this commit contains some styling cleanups.

    Closes-Bug: #1579072
    Change-Id: Id24d742a6c7d9f1240bba7991959119d19084e94
    Signed-off-by: Maksim Malchuk <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/318534

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/318534
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=af290c4f20d17278e55713d468dfa708ea17ccc4
Submitter: Jenkins
Branch: stable/mitaka

commit af290c4f20d17278e55713d468dfa708ea17ccc4
Author: Maksim Malchuk <email address hidden>
Date: Wed May 18 16:11:57 2016 +0300

    Remove duplicate 'default_log_levels' declaration

    Duplicate declaration 'default_log_levels' for the '::nova::logging'
    produces an issue when nova-compute logs not created on the compute
    nodes in case the 'debug' option is enabled during deployment. This
    commit removes this duplication because it already declared in the
    globals. Also, this commit contains some styling cleanups.

    Closes-Bug: #1579072
    Change-Id: Id24d742a6c7d9f1240bba7991959119d19084e94
    Signed-off-by: Maksim Malchuk <email address hidden>
    (cherry picked from commit f989647cc5e6cb4a17db57f5c5d82eb92676efd7)

[ElenaRossokhina (esolomina)]

verified on iso #417-mos:
nova* logs are created with enabled/disabled checkbox 'OpenStack debug logging'
cat /etc/fuel_build_id:
 417
cat /etc/fuel_build_number:
 417
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 mitaka-9.0
rpm -qa | egrep 'fuel|astute|network-checker|nailgun|packetary|shotgun':
 fuel-release-9.0.0-1.mos6347.noarch
 fuel-bootstrap-cli-9.0.0-1.mos284.noarch
 fuel-migrate-9.0.0-1.mos8398.noarch
 fuel-mirror-9.0.0-1.mos137.noarch
 fuel-notify-9.0.0-1.mos8398.noarch
 nailgun-mcagents-9.0.0-1.mos746.noarch
 python-fuelclient-9.0.0-1.mos316.noarch
 fuelmenu-9.0.0-1.mos270.noarch
 fuel-9.0.0-1.mos6347.noarch
 fuel-utils-9.0.0-1.mos8398.noarch
 fuel-setup-9.0.0-1.mos6347.noarch
 fuel-library9.0-9.0.0-1.mos8398.noarch
 shotgun-9.0.0-1.mos90.noarch
 fuel-agent-9.0.0-1.mos284.noarch
 fuel-ui-9.0.0-1.mos2706.noarch
 fuel-ostf-9.0.0-1.mos934.noarch
 fuel-misc-9.0.0-1.mos8398.noarch
 python-packetary-9.0.0-1.mos137.noarch
 fuel-nailgun-9.0.0-1.mos8709.noarch
 rubygem-astute-9.0.0-1.mos746.noarch
 fuel-provisioning-scripts-9.0.0-1.mos8709.noarch
 network-checker-9.0.0-1.mos72.x86_64
 fuel-openstack-metadata-9.0.0-1.mos8709.noarch