FC CI jobs are failing with" error: 'staticmethod' object is not callable"

[Patrick East]

CI jobs are failing with errors in c-vol like:

2016-02-04 02:28:20.299 ERROR cinder.volume.manager [req-e98fdf55-2320-4524-a870-28d151d18028 tempest-DeleteServersTestJSON-79559941] [635f0f-54568c79-5462-4c4e-a2cd-6cf4ec090757s] Validate volume connection failed (error: 'staticmethod' object is not callable).
2016-02-04 02:28:20.299 ERROR oslo_messaging.rpc.dispatcher [req-e98fdf55-2320-4524-a870-28d151d18028 tempest-DeleteServersTestJSON-79559941] Exception during message handling: Bad or unexpected response from the storage volume backend API: Validate volume connection failed (error: 'staticmethod' object is not callable).
2016-02-04 02:28:20.299 20374 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-02-04 02:28:20.299 20374 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 143, in _dispatch_and_reply
2016-02-04 02:28:20.299 20374 ERROR oslo_messaging.rpc.dispatcher executor_callback))
2016-02-04 02:28:20.299 20374 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 189, in _dispatch
2016-02-04 02:28:20.299 20374 ERROR oslo_messaging.rpc.dispatcher executor_callback)
2016-02-04 02:28:20.299 20374 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2016-02-04 02:28:20.299 20374 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2016-02-04 02:28:20.299 20374 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 117, in wrapper
2016-02-04 02:28:20.299 20374 ERROR oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2016-02-04 02:28:20.299 20374 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/cinder/cinder/volume/manager.py", line 1351, in initialize_connection
2016-02-04 02:28:20.299 20374 ERROR oslo_messaging.rpc.dispatcher raise exception.VolumeBackendAPIException(data=err_msg)
2016-02-04 02:28:20.299 20374 ERROR oslo_messaging.rpc.dispatcher VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Validate volume connection failed (error: 'staticmethod' object is not callable).
2016-02-04 02:28:20.299 20374 ERROR oslo_messaging.rpc.dispatcher
2016-02-04 02:28:20.301 ERROR oslo_messaging._drivers.common [req-e98fdf55-2320-4524-a870-28d151d18028 tempest-DeleteServersTestJSON-79559941] Returning exception Bad or unexpected response from the storage volume backend API: Validate volume connection failed (error: 'staticmethod' object is not callable). to caller
2016-02-04 02:28:20.301 ERROR oslo_messaging._drivers.common [req-e98fdf55-2320-4524-a870-28d151d18028 tempest-DeleteServersTestJSON-79559941] ['Traceback (most recent call last):\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 143, in _dispatch_and_reply\n executor_callback))\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 189, in _dispatch\n executor_callback)\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 117, in wrapper\n return f(*args, **kwargs)\n', ' File "/opt/stack/new/cinder/cinder/volume/manager.py", line 1351, in initialize_connection\n raise exception.VolumeBackendAPIException(data=err_msg)\n', "VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Validate volume connection failed (error: 'staticmethod' object is not callable).\n"]

Full log: http://openstack-logs.purestorage.com/19/275919/1/check/PureFCDriver-tempest-dsvm-volume-multipath/21e6a5e/logs/screen-c-vol.txt.gz

[Sean McGinnis (sean-mcginnis)]

From my CI, this appears to have happened after 2016-02-03 02:50 before it started failing at 2016-02-03 11:59 central time.

http://oslogs.compellent.com/dell-sc-fc-269624-14/logs/screen-c-vol.log.txt.gz

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/276425

[Patrick East (patrick-east)]

Looks like the problem is osprofiler, pinning to <1.0.0 makes the issue go away. The patch up to remove the static decorators also fixes it.

[Sean McGinnis (sean-mcginnis)]

May be related to osprofiler change https://github.com/openstack/osprofiler/commit/50832595d6e70db722cf26eb581277535ba2068d - but I haven't been able to dig in enough to fully understand. It does appear the staticmethod decorator is being incorrectly handled by the tracing though.

[Dina Belova (dbelova)]

It's not a duplicate... The issue you're linking to is fixed with the https://github.com/openstack/osprofiler/commit/50832595d6e70db722cf26eb581277535ba2068d

[Sean McGinnis (sean-mcginnis)]

That commit is actually in the 1.0.0 release. So that either doesn't handle this correctly, or there is a different path that needs a similar fix.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/277120

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sean McGinnis (<email address hidden>) on branch: master
Review: https://review.openstack.org/277120
Reason: Official global update pushed out, abandoning this one.

[Sean McGinnis (sean-mcginnis)]

Requirements update in https://review.openstack.org/#/c/276761/ skips 1.0.0 and global update https://review.openstack.org/#/c/277125/ fixes Cinder.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ivan Kolodyazhny (<email address hidden>) on branch: master
Review: https://review.openstack.org/276425

[Sean McGinnis (sean-mcginnis)]

Issue fixed in Cinder by the 1.1.0 release of osprofiler.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/278596

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/278596
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=f55db194876e08ef4671d31c52b8665fb540c00e
Submitter: Jenkins
Branch: master

commit f55db194876e08ef4671d31c52b8665fb540c00e
Author: Eric Harney <email address hidden>
Date: Wed Feb 10 14:27:49 2016 -0500

    Profiler: make it possible to run without loading osprofiler

    This makes it possible to run Cinder without having
    osprofiler installed. This is primarily to make it easier
    to debug issues and ensure that osprofiler is not
    introducing problems when debugging CI failures.

    Note that this requires disabling osprofiler in api-paste.ini
    as well.

    Related-Bug: #1541996
    Change-Id: If00451524b87b949a2d5888a1157184f64cd0d59