Default neutron-ns-metadata-proxy threads setting is too low (newton)

Bug #1711215 reported by Michael Johnson
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
High
Unassigned

Bug Description

In the older neutron-ns-metadata-proxy, in the newton release, the number of threads is fixed at 100. This is a drop from the previous default setting of 1000 as a side effect of changing the number of wsgi threads [1].

This is causing failures at sites with a large number of instances using deployment tools (instance cloud-init logs):
2017-08-01 15:44:36,773 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [119/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Request timed out. (timeout=17.0)]
2017-08-01 15:44:37,775 - DataSourceEc2.py[CRITICAL]: Giving up on md from ['http://169.254.169.254/2009-04-04/meta-data/instance-id'] after 120 seconds

Setting the value, for neutron-ns-metadata-proxy only, back up to 1000 resolves this issue.
It should also be noted that in the Ocata forward version of the neutron-ns-metadata-proxy the default value is 1024 [2].

I am going to propose a patch for stable/newton that sets the default thread count for the neutron-ns-metadata-proxy back up to 1000.

[1] https://github.com/openstack/neutron/blob/master/releasenotes/notes/config-wsgi-pool-size-a4c06753b79fee6d.yaml
[2] https://github.com/openstack/neutron/blob/master/neutron/agent/metadata/driver.py#L44

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/494313

Changed in neutron:
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/newton)

Reviewed: https://review.openstack.org/494313
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=69752ca678be674dcbcde6c5e4874c44843a28d8
Submitter: Jenkins
Branch: stable/newton

commit 69752ca678be674dcbcde6c5e4874c44843a28d8
Author: Michael Johnson <email address hidden>
Date: Wed Aug 16 13:06:42 2017 -0700

    Return neutron-ns-metadata-proxy threads to 1000

    A wsgi threads change [1] also changed the number of threads the
    neutron-ns-metadata-proxy process has down to 100. At some sites this
    low setting is leading to failures.
    This patch increases only the neutron-ns-metadata-proxy default thread count
    back up to 1000 to resolve these failures.

    This is not a cherry pick from Ocata as this code was removed in Ocata
    in favor of using HAProxy with a setting of 1024 [2].

    [1] https://review.openstack.org/278007/
    [2] https://review.openstack.org/431691

    Change-Id: I59cb05e165c6daa4ce0a1b636450f66fb699e88e
    Closes-Bug: #1711215

tags: added: in-stable-newton
Changed in neutron:
status: New → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.