snmp pollster is not thread safe

[Mehdi Abaakouk]

Hi,

the snmp pollster uses pysnmp, more precisely oneliner.CommandGenerator() object.

This one is not threads safe. Currently we use one object for all query within a process.

On juno, this was resulting to the following backtrace because on eventlet:

;RuntimeError: Second simultaneous write on fileno 21 detected. Unless you really know what you're doing, make sure that only one greenthread can write any particular socket. Consider using a pools.Pool. If yo
u do know what you're doing and want to disable this error, call eventlet.debug.hub_prevent_multiple_readers(False) - MY THREAD=<built-in method switch of GreenThread object at 0x2da5050>; THAT THREAD=FdListener
('write', 21, <built-in method switch of GreenThread object at 0x2da50f0>, <built-in method throw of GreenThread object at 0x2da50f0>)
2015-11-19 08:08:48.844 25809 ERROR ceilometer.hardware.pollsters.generic Traceback (most recent call last):

Now the issue is completely hidden because nothing warn us when we write with two threads into the same socket.

Cheers,

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/311057

[ZhiQiang Fan (aji-zqfan)]

please submit changes to stable branch, master branch doesn't have such issue IIUC

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/311087

[Mehdi Abaakouk (sileht)]

Complete backtrace on a liberty installation: https://bugzilla.redhat.com/attachment.cgi?id=1096715

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/311057
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=c898ecd921123eab61a8f88e61686ab4edf82b35
Submitter: Jenkins
Branch: master

commit c898ecd921123eab61a8f88e61686ab4edf82b35
Author: Mehdi Abaakouk <email address hidden>
Date: Fri Apr 29 10:30:13 2016 +0200

    Fix concurrency issue with snmp pollsters

    pysnmp oneliner.CommandGenerator() is not thread safe.

    This change creates one oneliner.CommandGenerator() per query to fix
    the issue.

    Closes-bug: #1576577

    Change-Id: Ibf32c068abed5d7198150bc206987f050a093cd8

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/311087
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=fe66169a2e1f774617f60cc5902d84b110efafec
Submitter: Jenkins
Branch: stable/liberty

commit fe66169a2e1f774617f60cc5902d84b110efafec
Author: Mehdi Abaakouk <email address hidden>
Date: Fri Apr 29 10:30:13 2016 +0200

    Fix concurrency issue with snmp pollsters

    pysnmp oneliner.CommandGenerator() is not thread safe.

    This change creates one oneliner.CommandGenerator() per query to fix
    the issue.

    Closes-bug: #1576577

    Change-Id: Ibf32c068abed5d7198150bc206987f050a093cd8
    (cherry picked from commit c898ecd921123eab61a8f88e61686ab4edf82b35)

[Davanum Srinivas (DIMS) (dims-v)]

This issue was fixed in the openstack/ceilometer 5.0.4 release.

[Davanum Srinivas (DIMS) (dims-v)]

This issue was fixed in the openstack/ceilometer 7.0.0.0b2 development milestone.