Neutron Metadata Agents backlog value too low and differs from Neutron default

[Ian Cordasco]

Neutron Metadata Agents works roughly like this: The service runs on a Unix Domain Socket. It runs as many threads as cpu_count/2. Each thread has a maximum number of connections that it can accept. The service attempts to talk to Neutron API via RPC but (only on) Kilo and Liberty will fall back to HTTP if necessary.

By default (since Kilo) Neutron has that number set to 4096. Since kilo in OpenStack Ansible the number has been 128.

At high throughput on a large OpenStack Ansible cloud, keeping this value arbitrarily low (1/32nd of it's default value) causes RPC connections from Neutron Metadata Agent to fail and fallback to HTTP. Setting this value to even half of the upstream default (on only 2 of the 5 metadata agents containers) causes this fallback to stop happening.

OpenStack Ansible Master: https://github.com/openstack/openstack-ansible-os_neutron/blob/9898d7188341fe4a340a9f4f299620f205f026cb/defaults/main.yml#L303
Neutron Master: https://github.com/openstack/neutron/blob/dd4f1253c951d78a5b497680dfb31317ba469a58/neutron/agent/metadata/config.py#L112

OpenStack Ansible Mitaka: https://github.com/openstack/openstack-ansible-os_neutron/blob/dc7fd56d233e5ece1940b9344b3fee059f6597b9/defaults/main.yml#L287
Neutron Mitaka: https://github.com/openstack/neutron/blob/13438bad4136c6f5ff4f37c43cb151ec77a8ffd3/neutron/agent/metadata/config.py#L113

OpenStack Ansible Liberty: https://github.com/openstack/openstack-ansible/blob/ac596657132e4fbb05ac581e802e55c68a58da57/playbooks/roles/os_neutron/defaults/main.yml#L159
Neutron Liberty: https://github.com/openstack/neutron/blob/5067c7b49a5701d01c7277699ca5a93885913065/neutron/agent/metadata/config.py#L133

OpenStack Ansible Kilo: https://github.com/openstack/openstack-ansible/blob/98f2b2fd255952f4438f58de03a2f225d0664f2c/playbooks/roles/os_neutron/defaults/main.yml#L233
Neutron Kilo: https://github.com/openstack/neutron/blob/363eeb06104662ee38aeed04af043899379f6ab8/neutron/agent/metadata/config.py#L129

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/328441

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/328442

[Bjoern (bjoern-t)]

Liberty please too

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/328441
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_neutron/commit/?id=357e5432fe70f237e6ef7d84b7224634fac4fd25
Submitter: Jenkins
Branch: master

commit 357e5432fe70f237e6ef7d84b7224634fac4fd25
Author: Ian Cordasco <email address hidden>
Date: Fri Jun 10 13:07:53 2016 -0500

    Use default metadata_backlog value from Neutron

    Change-Id: If8e24446e2b6df3efc330d197265871e8e98253c
    Closes-bug: 1591297

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: liberty
Review: https://review.openstack.org/328502

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: kilo
Review: https://review.openstack.org/328503

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/328442
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_neutron/commit/?id=58abc3243274680c572b31f5b662f9caf56efd4d
Submitter: Jenkins
Branch: stable/mitaka

commit 58abc3243274680c572b31f5b662f9caf56efd4d
Author: Ian Cordasco <email address hidden>
Date: Fri Jun 10 13:07:53 2016 -0500

    Use default metadata_backlog value from Neutron

    Change-Id: If8e24446e2b6df3efc330d197265871e8e98253c
    Closes-bug: 1591297
    (cherry picked from commit 357e5432fe70f237e6ef7d84b7224634fac4fd25)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/328503
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=23cf197223dc5b60bc9223b8180db46db43e2897
Submitter: Jenkins
Branch: kilo

commit 23cf197223dc5b60bc9223b8180db46db43e2897
Author: Ian Cordasco <email address hidden>
Date: Fri Jun 10 15:05:08 2016 -0500

    Use default metadata_backlog value from Neutron

    (cherry picked from commit f7ff05e01396a594c735e1fed82669a3e5db7b45)
    Change-Id: If8e24446e2b6df3efc330d197265871e8e98253c
    Closes-bug: 1591297

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/328502
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=f7ff05e01396a594c735e1fed82669a3e5db7b45
Submitter: Jenkins
Branch: liberty

commit f7ff05e01396a594c735e1fed82669a3e5db7b45
Author: Ian Cordasco <email address hidden>
Date: Fri Jun 10 15:05:08 2016 -0500

    Use default metadata_backlog value from Neutron

    (cherry picked from the os-keystone role 58abc3243274680c572b31f5b662f9caf56efd4d)
    Change-Id: If8e24446e2b6df3efc330d197265871e8e98253c
    Closes-bug: 1591297

[Davanum Srinivas (DIMS) (dims-v)]

This issue was fixed in the openstack/openstack-ansible 12.0.15 release.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/openstack-ansible-os_neutron 13.1.4 release.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/openstack-ansible 11.2.17 release.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/openstack-ansible-os_neutron 14.0.0.0b2 development milestone.