[RFE] Ironic support for OSprofiler

[Ramamani Yeleswarapu]

Short summary:
- OSProfiler is a cross service profiler.
- It is already supported and used in several OpenStack projects:
   Cinder, Glance, Heat, Nova, etc.

More about the spec is available here: https://review.openstack.org/#/c/103825/

In Ironic, OSProfiler will allow tracing of Ironic operations by providing output about how much time was spent on API and RPC calls.

More details about how OSprofiler works:
https://github.com/openstack/osprofiler/blob/master/README.rst

[Ramamani Yeleswarapu (ramamani-yeleswarapu)]

I am working on this [RFE].

[Tang Chen (tangchen)]

Hi Ramamani,

Are you still working on this RFE ? Since it has been 3 months that no update here, I'd like to take it over and continue to work on it, if you don't mind.

Thanks. :)

[Shobha Ranganathan (shobha-ranganathan)]

Hi Tang,

Ramamani is actively working on this. She will be updating this RFE with patch very soon

Shobha

[Ramamani Yeleswarapu (ramamani-yeleswarapu)]

Hi Tang Chen,

I have been working on a patch for this and I am about to post it.

Regards,
Ramamani.

[Ramamani Yeleswarapu (ramamani-yeleswarapu)]

Patch submitted: https://review.openstack.org/#/c/337987/

[Tang Chen (tangchen)]

Hi Ramamani, Shobha，

Sure, thanks very much. :)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/358142

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ramamani Yeleswarapu (<email address hidden>) on branch: master
Review: https://review.openstack.org/337987
Reason: Submitted a new patch with bug id, documentation: https://review.openstack.org/#/c/358142/

[Ramamani Yeleswarapu (ramamani-yeleswarapu)]

Hi Tovin, thanks rebasing the patch. I am very much interested in merging this patch; I would appreciate any help in testing this patch. Thanks!

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/358142
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=3773f17403bb02b372673643780b9619151d22ea
Submitter: Jenkins
Branch: master

commit 3773f17403bb02b372673643780b9619151d22ea
Author: Ramamani Yeleswarapu <email address hidden>
Date: Thu Aug 18 16:55:56 2016 -0700

    Enable OSProfiler support in Ironic

    This patch does the following:
    * Adds osprofiler wsgi middleware
      This middleware is used for 2 things:
      - It checks that person who wants to trace is trusted and knows
        secret HMAC key.
      - It starts tracing in case of proper trace headers
        and adds first wsgi trace point, with info about HTTP request.

    * Adds initialization of osprofiler at start of service
      - Initialize and set an oslo.messaging based notifier instance
        to osprofiler, which will be used to send notifications to Ceilometer.

    * Traces HTTP/RPC/DB API calls and SQL requests

    NOTE to test this patch:
    1) Make the following changes in localrc to configure DevStack to enable
       OSProfiler:

       enable_plugin panko https://git.openstack.org/openstack/panko
       enable_plugin ceilometer https://git.openstack.org/openstack/ceilometer
       enable_plugin osprofiler https://git.openstack.org/openstack/osprofiler

       # Enable the following services
       CEILOMETER_NOTIFICATION_TOPICS=notifications,profiler
       ENABLED_SERVICES+=,ceilometer-acompute,ceilometer-acentral
       ENABLED_SERVICES+=,ceilometer-anotification,ceilometer-collector
       ENABLED_SERVICES+=,ceilometer-alarm-evaluator,ceilometer-alarm-notifier
       ENABLED_SERVICES+=,ceilometer-api

    NOTE: the order of enabling plugins matters.

    2) Run stack.sh. Once DevStack environment is setup, enable profiler options
       in ironic.conf and restart ironic services:
       [profiler]
       enabled = true
       hmac_keys = SECRET_KEY
       trace_sqlalchemy = true

    3) Use openstackclient and run baremetal command with
       --os-profile SECRET_KEY
       [--profile can be used, but it is deprecated.]
       For example, the following will cause the <trace-id> to be printed
       after node list:

       $ openstack --os-profile SECRET_KEY baremetal node list
       .....
       .....
       Trace ID: <trace-id>
       Display trace with command:
       osprofiler trace show --html <trace-id>

    4) The trace results can be saved using this command:

       $ osprofiler trace show --html <trace-id> --out trace.html

    OSprofiler spec: https://review.openstack.org/#/c/103825/

    Co-Authored-By: Tovin Seven <email address hidden>
    Co-Authored-By: Hieu LE <email address hidden>

    Partial-Bug: #1560704

    Change-Id: Icd3d7c62cf7442de8a77fc67f119ae9b03725f02

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/490675

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/494784

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/490675
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=f7f7750baf17889c1ab7861635a6021bf778c141
Submitter: Jenkins
Branch: master

commit f7f7750baf17889c1ab7861635a6021bf778c141
Author: Ramamani Yeleswarapu <email address hidden>
Date: Thu Aug 3 16:15:28 2017 -0700

    Enable OSProfiler support in Ironic - follow-up

    Fixes nits, updates documentation and releasenotes.

    This is a follow-up to commit 3773f17403bb02b372673643780b9619151d22ea.

    Change-Id: I4082bbb9f8c09eaf4569317eb1387bf9a36ef254
    Partial-Bug: #1560704

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/494784
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=2a6b2169b6b070a5cf97eca0246e520a8d400f0b
Submitter: Jenkins
Branch: stable/pike

commit 2a6b2169b6b070a5cf97eca0246e520a8d400f0b
Author: Ramamani Yeleswarapu <email address hidden>
Date: Thu Aug 3 16:15:28 2017 -0700

    Enable OSProfiler support in Ironic - follow-up

    Fixes nits, updates documentation and releasenotes.

    This is a follow-up to commit 3773f17403bb02b372673643780b9619151d22ea.

    Change-Id: I4082bbb9f8c09eaf4569317eb1387bf9a36ef254
    Partial-Bug: #1560704
    (cherry picked from commit 59cb133bf2cb83d738007666f29d690ebec9cf96)