OTLP profiler is not working when using multiple share backends

Bug #2067609 reported by Sylvan Le Deunff
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Shared File Systems Service (Manila)
Fix Released
Medium
Sylvan Le Deunff

Bug Description

Description
===========
osprofiler driver for opentelemetry (OTLP) fails to export traces when manila service is ran with oslo_service.service.ProcessLauncher (this is the case of API and share).

The root cause seems to be related to the fact that profiler is initialized before forking. Everything works well if profiler is initialized in start method (ie after oslo_service.service.ProcessLauncher create a forks for the child process)

Steps to reproduce
==================

* Configure a jaeger instance (e.g with docker-compose https://paste.openstack.org/show/bAagyVkZZ56YAMIdGzr4/)
* Ensure installed version of osprofiler >= 4.0.0 to have new OTLP driver
* Install python packages required by OTLP driver: opentelemetry-sdk opentelemetry-exporter-otlp
* Add osprofiler middleware to api-paste.ini config file (https://paste.openstack.org/show/bRL1xrarCm05rNWMD3Xl/)
* Configure manila to use OTLP driver for profiler (e.g: https://paste.openstack.org/show/bCjt0HWhh6r5HLkPjD7Q/)
* Start manila components (api, scheduler, share)
* Create a share by calling manila API with tracing info (e.g with curl: https://paste.openstack.org/show/bi742a3EFZzh047EdMyG/)

Expected result
===============
Open jaeger and find a trace that span from manila-api to manila-share.

Actual result
=============
API and shares does not produce traces... But scheduler do!

Environment
===========
1. manila==16.0.0

2. Custom NetApp share driver

3. StandaloneNetworkPlugin

Vida Haririan (vhariria)
Changed in manila:
importance: Undecided → Medium
Vida Haririan (vhariria)
Changed in manila:
assignee: nobody → Sylvan Le Deunff (sylvanld)
assignee: Sylvan Le Deunff (sylvanld) → nobody
Revision history for this message
Vida Haririan (vhariria) wrote :
Changed in manila:
assignee: nobody → Sylvan Le Deunff (sylvanld)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to manila (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/manila/+/920873

Changed in manila:
status: New → In Progress
Changed in manila:
milestone: none → dalmatian-2
Changed in manila:
status: In Progress → Fix Committed
Vida Haririan (vhariria)
Changed in manila:
status: Fix Committed → In Progress
Revision history for this message
Vida Haririan (vhariria) wrote :

See related comments in manila-dalmatian-m2-bugsquash ethercal https://ethercalc.net/o2qsmjpagsfb

Changed in manila:
milestone: dalmatian-2 → dalmatian-3
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to manila (master)

Reviewed: https://review.opendev.org/c/openstack/manila/+/920873
Committed: https://opendev.org/openstack/manila/commit/c4144bd5a37e591501d00de7725dc9de6110af8a
Submitter: "Zuul (22348)"
Branch: master

commit c4144bd5a37e591501d00de7725dc9de6110af8a
Author: Sylvan Le Deunff <email address hidden>
Date: Fri May 31 10:58:00 2024 +0200

    Initialize profiler after ProcessLauncher fork

    Closes-Bug: #2067609
    Change-Id: I988779cea3e57a846a7f4a542dd0c6efee2009a8

Changed in manila:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/manila 19.0.0.0rc1

This issue was fixed in the openstack/manila 19.0.0.0rc1 release candidate.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.