gnocchi-api hangs with api.workers = 2 and CEPH

[Alejandro Comisario]

Hi, on gnocchi/master integrated with ceilometer-collector/kilo when configuring gnocchi with CEPH as backend with two api workers, the first two parallel writes to CEPH hangs on line :

https://github.com/openstack/gnocchi/blob/master/gnocchi/storage/ceph.py#L129

Changing api workers = 1 works ok , but loosing the possibility to scale on handling more requests.

gnocchi.conf example:

[DEFAULT]
debug = True
verbose = True
log_file = /var/log/gnocchi/gnocchi.log
[api]
port = 8041
host = 0.0.0.0
workers = 2
[archive_policy]
[database]
[indexer]
url = mysql://gnocchi:NOTgnocchi@mysql/gnocchi?charset=utf8
[keystone_authtoken]
signing_dir = /var/cache/gnocchi
auth_uri = http://kstn:5000/v2.0
auth_url = http://kstn:35357/v2.0
project_domain_id = default
project_name = service
project_name = admin
password = MYSUPERPASSWD
username = cloudadmin
auth_plugin = password
memcached_servers = memcache2:11211,memcache1:11211
memcache_security_strategy = ENCRYPT
memcache_secret_key = LE9_s0kyh7Z_qNsmljOT
[metricd]
[oslo_policy]
[statsd]
[storage]
driver = ceph
metric_processing_delay = 5
ceph_pool = gnocchi
ceph_username = gnocchi
ceph_keyring = /etc/ceph/ceph.client.gnocchi.keyring
ceph_conffile = /etc/ceph/ceph.conf
file_basepath = /var/lib/gnocchi
file_basepath_tmp = ${file_basepath}/tmp

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/227218

[Alejandro Comisario (alejandro-f)]

how can we test this so it can be merged ?

[Julien Danjou (jdanjou)]

If you clone master there are instructions at https://review.openstack.org/#/c/227218/ that indicate how to download the patchset. Or run git review -d 227218 to download it.

[Alejandro Comisario (alejandro-f)]

will do!
On Sep 26, 2015 13:00, "Julien Danjou" <email address hidden> wrote:

> If you clone master there are instructions at
> https://review.openstack.org/#/c/227218/ that indicate how to download
> the patchset. Or run git review -d 227218 to download it.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1499115
>
> Title:
> gnocchi-api hangs with api.workers = 2 and CEPH
>
> Status in Gnocchi:
> In Progress
>
> Bug description:
> Hi, on gnocchi/master integrated with ceilometer-collector/kilo when
> configuring gnocchi with CEPH as backend with two api workers, the
> first two parallel writes to CEPH hangs on line :
>
>
> https://github.com/openstack/gnocchi/blob/master/gnocchi/storage/ceph.py#L129
>
> Changing api workers = 1 works ok , but loosing the possibility to
> scale on handling more requests.
>
> gnocchi.conf example:
>
> [DEFAULT]
> debug = True
> verbose = True
> log_file = /var/log/gnocchi/gnocchi.log
> [api]
> port = 8041
> host = 0.0.0.0
> workers = 2
> [archive_policy]
> [database]
> [indexer]
> url = mysql://gnocchi:NOTgnocchi@mysql/gnocchi?charset=utf8
> [keystone_authtoken]
> signing_dir = /var/cache/gnocchi
> auth_uri = http://kstn:5000/v2.0
> auth_url = http://kstn:35357/v2.0
> project_domain_id = default
> project_name = service
> project_name = admin
> password = MYSUPERPASSWD
> username = cloudadmin
> auth_plugin = password
> memcached_servers = memcache2:11211,memcache1:11211
> memcache_security_strategy = ENCRYPT
> memcache_secret_key = LE9_s0kyh7Z_qNsmljOT
> [metricd]
> [oslo_policy]
> [statsd]
> [storage]
> driver = ceph
> metric_processing_delay = 5
> ceph_pool = gnocchi
> ceph_username = gnocchi
> ceph_keyring = /etc/ceph/ceph.client.gnocchi.keyring
> ceph_conffile = /etc/ceph/ceph.conf
> file_basepath = /var/lib/gnocchi
> file_basepath_tmp = ${file_basepath}/tmp
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/gnocchi/+bug/1499115/+subscriptions
>

[Alejandro Comisario (alejandro-f)]

The patch works as expected on master

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/227218
Committed: https://git.openstack.org/cgit/openstack/gnocchi/commit/?id=ed339314d17094c5d9c0308b91c4f944c0157d0a
Submitter: Jenkins
Branch: master

commit ed339314d17094c5d9c0308b91c4f944c0157d0a
Author: Mehdi Abaakouk <email address hidden>
Date: Thu Sep 24 12:47:20 2015 +0200

    rest: use a fork friendly app with werkzeug

    Unlike many wsgi servers like apache mod_wsgi or uwsgi, werkzeug creates
    the wsgi application object before forking workers.

    This have some issues, storage/indexer connection are created
    and leaves on the main process for no reason.
    And the ceph storage driver just doesn't support to be
    forked once the connection to ceph have been established.

    This change creates a class that lazy load the application for
    werkzeug and fixes these issues.

    Closes-bug: #1499115

    Change-Id: I398e289767e2e267466bcf882a5bb013620949d0