Fuel for OpenStack

neutron openvswitch agent logs in master node are in no match

Series 6.0.x
Bug #1402641

Bug #1402641 reported by Ilya Shakhat on 2014-12-15

This bug affects 1 person

	Status	Importance	Assigned to	Milestone
Fuel for OpenStack	Fix Released	High	Sergey Kolekonov	Fuel for OpenStack 6.0-updates
6.0.x	Fix Committed	High	Alexander Nevenchannyy	Fuel for OpenStack 6.0-updates
6.1.x	Fix Released	High	Sergey Kolekonov	Fuel for OpenStack 6.1

Bug Description

There are no logs from openvswitch agent inside /var/log/remote/. But there is series of files **NO MATCH**.log with content from the agent.

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.0"
  api: "1.0"
  build_number: "49"
  build_id: "2014-12-09_22-41-06"
  astute_sha: "16b252d93be6aaa73030b8100cf8c5ca6a970a91"
  fuellib_sha: "2c99931072d951301d395ebd5bf45c8d401301bb"
  ostf_sha: "a9afb68710d809570460c29d6c3293219d3624d4"
  nailgun_sha: "22bd43b89a17843f9199f92d61fc86cb0f8772f1"
  fuelmain_sha: "3aab16667f47dd8384904e27f70f7a87ba15f4ee"

Tags:

Ilya Shakhat (shakhat) on 2014-12-15

Changed in fuel:
assignee:	nobody → Fuel Library Team (fuel-library)
importance:	Undecided → Critical
milestone:	none → 6.0

Łukasz Oleś (loles) on 2014-12-15

Changed in fuel:
assignee:	Fuel Library Team (fuel-library) → Łukasz Oleś (loles)

Łukasz Oleś (loles) on 2014-12-15

Changed in fuel:
status:	New → In Progress

Revision history for this message

Mike Scherbakov (mihgen) wrote on 2014-12-16:

Folks, is it really Critical? If you can still find logging in other file, and it doesn't really affect functionality - High is the maximum which can be given to the bug.

Revision history for this message

Mike Scherbakov (mihgen) wrote on 2014-12-16:

Moved to High according to my previous comment.

Changed in fuel:
importance:	Critical → High

Andrew Woodward (xarses) on 2014-12-18

summary:

- No logs from neutron openvswitch agent
+ neutron openvswitch agent logs in master node are in no match

Revision history for this message

Mike Scherbakov (mihgen) wrote on 2014-12-22:

Too late for 6.0, so targeting 6.0.1

Changed in fuel:
milestone:	6.0 → 6.0.1
status:	In Progress → Confirmed

Łukasz Oleś (loles) on 2014-12-22

Changed in fuel:
status:	Confirmed → In Progress

Revision history for this message

Łukasz Oleś (loles) wrote on 2015-01-09:

Ok, I finally had time to look into it.

It looks like the problem is on the OpenStack side and it has the same cause as in this thread http://lists.openstack.org/pipermail/openstack-dev/2014-December/052869.html .

To save remote logs we use rsyslog and to find correct log file name we use variable "%syslogtag" which usually looks like this:
"neutron-openvswitch-agent" but for some reason for some logs it looks like this: "2015-01-09". It's just a date and we can't find the correct file to store it.

On fuel side we can make some ugly dirty hacks to find correct file but I think it should be fixed in OpenStack.

Revision history for this message

Łukasz Oleś (loles) wrote on 2015-01-09: Re: [Bug 1402641] Re: neutron openvswitch agent logs in master node are in no match

I will check it and let you know on monday

On Fri, Jan 9, 2015 at 10:43 AM, Aleksandr Didenko
<email address hidden> wrote:
> Łukasz, please try to add this into neutron.conf DEFAULT sections:
>
> use_syslog_rfc_format = true
>
> and see if it helps. It looks like we've lost this option during neutron
> puppet module update.
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/1402641
>
> Title:
> neutron openvswitch agent logs in master node are in no match
>
> Status in Fuel: OpenStack installer that works:
> Confirmed
> Status in Fuel for OpenStack 6.0.x series:
> Won't Fix
> Status in Fuel for OpenStack 6.1.x series:
> Confirmed
>
> Bug description:
> There are no logs from openvswitch agent inside /var/log/remote/. But
> there is series of files **NO MATCH**.log with content from the agent.
>
> VERSION:
> feature_groups:
> - mirantis
> production: "docker"
> release: "6.0"
> api: "1.0"
> build_number: "49"
> build_id: "2014-12-09_22-41-06"
> astute_sha: "16b252d93be6aaa73030b8100cf8c5ca6a970a91"
> fuellib_sha: "2c99931072d951301d395ebd5bf45c8d401301bb"
> ostf_sha: "a9afb68710d809570460c29d6c3293219d3624d4"
> nailgun_sha: "22bd43b89a17843f9199f92d61fc86cb0f8772f1"
> fuelmain_sha: "3aab16667f47dd8384904e27f70f7a87ba15f4ee"
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/fuel/+bug/1402641/+subscriptions

--
Łukasz Oleś

Revision history for this message

Aleksandr Didenko (adidenko) wrote on 2015-01-09:

Actually I've checked our manifests already and this option is already there so it should be configured in neutron.conf, so it's not the case.

Revision history for this message

Łukasz Oleś (loles) wrote on 2015-01-09:

yeah, it's there :(

On Fri, Jan 9, 2015 at 2:41 PM, Aleksandr Didenko <email address hidden> wrote:
> Actually I've checked our manifests already and this option is already
> there so it should be configured in neutron.conf, so it's not the case.
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/1402641
>
> Title:
> neutron openvswitch agent logs in master node are in no match
>
> Status in Fuel: OpenStack installer that works:
> Confirmed
> Status in Fuel for OpenStack 6.0.x series:
> Won't Fix
> Status in Fuel for OpenStack 6.1.x series:
> Confirmed
>
> Bug description:
> There are no logs from openvswitch agent inside /var/log/remote/. But
> there is series of files **NO MATCH**.log with content from the agent.
>
> VERSION:
> feature_groups:
> - mirantis
> production: "docker"
> release: "6.0"
> api: "1.0"
> build_number: "49"
> build_id: "2014-12-09_22-41-06"
> astute_sha: "16b252d93be6aaa73030b8100cf8c5ca6a970a91"
> fuellib_sha: "2c99931072d951301d395ebd5bf45c8d401301bb"
> ostf_sha: "a9afb68710d809570460c29d6c3293219d3624d4"
> nailgun_sha: "22bd43b89a17843f9199f92d61fc86cb0f8772f1"
> fuelmain_sha: "3aab16667f47dd8384904e27f70f7a87ba15f4ee"
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/fuel/+bug/1402641/+subscriptions

--
Łukasz Oleś

Revision history for this message

Bogdan Dobrelya (bogdando) wrote on 2015-01-09:

We have to make sure the following Oslo.log patches present in the log.py of affected neutron-ovs-agent as well:
https://github.com/openstack/oslo.log/commit/3f7855bdccfa93e879f050dd8eb7013ec121a73c
https://github.com/openstack/oslo.log/commit/4c2d53089795ad463b086526dc391435b2a9c611

Revision history for this message

Ilya Shakhat (shakhat) wrote on 2015-01-23:

#10

@Bogdan, the code is present and correct, however config file on compute misses option "use_syslog_rfc_format"

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-01-23: Fix proposed to fuel-library (master)

#11

Fix proposed to branch: master
Review: https://review.openstack.org/149674

Changed in fuel:
status:	Confirmed → In Progress

Sergey Kolekonov (skolekonov) on 2015-01-23

tags:

added: neutron

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-02-12: Fix merged to fuel-library (master)

#12

Reviewed: https://review.openstack.org/149674
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=2dc151b32f144085acbc0a938aba1d24c0797e16
Submitter: Jenkins
Branch: master

commit 2dc151b32f144085acbc0a938aba1d24c0797e16
Author: Sergey Kolekonov <email address hidden>
Date: Fri Jan 23 19:17:13 2015 +0300

Fix OVS agent logging for compute nodes

Currently the Neutron logging configuration option "use_syslog_rfc_format"
is used only for controlles but it's also required for compute nodes

Change-Id: I2c953608a87f4bb31a245909ac04796436caaca2
Closes-bug: #1402641

Changed in fuel:
status:	In Progress → Fix Committed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-02-12: Fix proposed to fuel-library (stable/6.0)

#13

Fix proposed to branch: stable/6.0
Review: https://review.openstack.org/155411

Revision history for this message

Anastasia Palkina (apalkina) wrote on 2015-02-16:

#14

Verified on ISO #126

"build_id": "2015-02-15_22-54-44", "ostf_sha": "f9c37d0876141e1550eb4e703a8e500cd463282f", "build_number": "126", "release_versions": {"2014.2-6.1": {"VERSION": {"build_id": "2015-02-15_22-54-44", "ostf_sha": "f9c37d0876141e1550eb4e703a8e500cd463282f", "build_number": "126", "api": "1.0", "nailgun_sha": "1e3a40dd8a17abe1d38f42da1e0dc1a6d4572666", "production": "docker", "python-fuelclient_sha": "61431ed16fc00039a269424bdbaa410277eff609", "astute_sha": "1f87a9b9a47de7498b4061d15a8c7fb9435709d5", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "2054229e275d08898b5d079a6625ffcc79ae23b8", "fuellib_sha": "7f8d4382abfcd4338964182ebfea1d539f963e66"}}}, "auth_required": true, "api": "1.0", "nailgun_sha": "1e3a40dd8a17abe1d38f42da1e0dc1a6d4572666", "production": "docker", "python-fuelclient_sha": "61431ed16fc00039a269424bdbaa410277eff609", "astute_sha": "1f87a9b9a47de7498b4061d15a8c7fb9435709d5", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "2054229e275d08898b5d079a6625ffcc79ae23b8", "fuellib_sha": "7f8d4382abfcd4338964182ebfea1d539f963e66"