"rpc_response_max_timeout" configuration variable not present in metadata-agent

[Takashi Kajinami]

We have observed that the following Traceback is logged in metadata-agent.log when metadata-agent encounters MessagingTimeout.

~~~
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 397, in get
    return self._queues[msg_id].get(block=True, timeout=timeout)
  File "/usr/lib/python3.6/site-packages/eventlet/queue.py", line 322, in get
    return waiter.wait()
  File "/usr/lib/python3.6/site-packages/eventlet/queue.py", line 141, in wait
    return get_hub().switch()
  File "/usr/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 298, in switch
    return self.greenlet.switch()
queue.Empty
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/neutron_lib/rpc.py", line 157, in call
    return self._original_context.call(ctxt, method, **kwargs)
  File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/client.py", line 181, in call
    transport_options=self.transport_options)
  File "/usr/lib/python3.6/site-packages/oslo_messaging/transport.py", line 129, in _send
    transport_options=transport_options)
  File "/usr/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 646, in send
    transport_options=transport_options)
  File "/usr/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 634, in _send
    call_monitor_timeout)
  File "/usr/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 523, in wait
    message = self.waiters.get(msg_id, timeout=timeout)
  File "/usr/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 401, in get
    'to message ID %s' % msg_id)
oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID <id>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 2197, in __getattr__
    return self._get(name)
  File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 2631, in _get
    value, loc = self._do_get(name, group, namespace)
  File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 2649, in _do_get
    info = self._get_opt_info(name, group)
  File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 2849, in _get_opt_info
    raise NoSuchOptError(opt_name, group)
oslo_config.cfg.NoSuchOptError: no such option rpc_response_max_timeout in group [DEFAULT]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/neutron/agent/metadata/agent.py", line 88, in __call__
    instance_id, tenant_id = self._get_instance_and_tenant_id(req)
  File "/usr/lib/python3.6/site-packages/neutron/agent/metadata/agent.py", line 184, in _get_instance_and_tenant_id
    skip_cache=skip_cache)
  File "/usr/lib/python3.6/site-packages/neutron/agent/metadata/agent.py", line 169, in _get_ports
    skip_cache=skip_cache)
  File "/usr/lib/python3.6/site-packages/neutron/common/cache_utils.py", line 122, in __call__
    return self.func(target_self, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/neutron/agent/metadata/agent.py", line 146, in _get_ports_for_remote_address
    ip_address=remote_address)
  File "/usr/lib/python3.6/site-packages/neutron/agent/metadata/agent.py", line 113, in _get_ports_from_server
    return self.plugin_rpc.get_ports(self.context, filters)
  File "/usr/lib/python3.6/site-packages/neutron/agent/metadata/agent.py", line 71, in get_ports
    return cctxt.call(context, 'get_ports', filters=filters)
  File "/usr/lib/python3.6/site-packages/neutron_lib/rpc.py", line 173, in call
    self._original_context.timeout * 2, self.get_max_timeout())
  File "/usr/lib/python3.6/site-packages/neutron_lib/rpc.py", line 133, in get_max_timeout
    return cls._max_timeout or _get_rpc_response_max_timeout()
  File "/usr/lib/python3.6/site-packages/neutron_lib/rpc.py", line 95, in _get_rpc_response_max_timeout
    return TRANSPORT.conf.rpc_response_max_timeout
  File "/usr/lib/python3.6/site-packages/oslo_messaging/_drivers/common.py", line 508, in __getattr__
    value = getattr(self._conf, name)
  File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 2201, in __getattr__
    raise NoSuchOptError(name)
oslo_config.cfg.NoSuchOptError: no such option rpc_response_max_timeout in group [DEFAULT]
~~~

A similar issue has been reported in https://bugs.launchpad.net/neutron/+bug/1880934 and fixed already. We need the same fix for metadata-agent.

Note: This issue was found in stable/train deployment with ml2+ovs

[Takashi Kajinami (kajinamit)]

I have proposed a fix https://review.opendev.org/c/openstack/neutron/+/782354

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 18.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/788766

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/neutron/+/789005

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/neutron/+/789006

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/788766
Committed: https://opendev.org/openstack/neutron/commit/b16925989211594275ddd5f6de30a47b298b0c00
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit b16925989211594275ddd5f6de30a47b298b0c00
Author: Takashi Kajinami <email address hidden>
Date: Tue Mar 23 11:46:34 2021 +0900

    Provide the rpc_response_max_timeout parameter to metadata-agent

    The rpc_response_max_timeout parameter is used in comminucation over
    messaging queue, thus should be available for metadata-agent which
    communicate with neutron-server over messaging queue.

    Change-Id: I4826edf16bbdf338af6e050b15b4ecce9f8ca6ae
    Closes-Bug: #1920842
    (cherry picked from commit 7f5711f77b79fdaffd91f6ef299783b058d18ace)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/789006
Committed: https://opendev.org/openstack/neutron/commit/63d48754f5f8d288acedc6df62386196edf295a8
Submitter: "Zuul (22348)"
Branch: stable/train

commit 63d48754f5f8d288acedc6df62386196edf295a8
Author: Takashi Kajinami <email address hidden>
Date: Tue Mar 23 11:46:34 2021 +0900

    Provide the rpc_response_max_timeout parameter to metadata-agent

    The rpc_response_max_timeout parameter is used in comminucation over
    messaging queue, thus should be available for metadata-agent which
    communicate with neutron-server over messaging queue.

    Change-Id: I4826edf16bbdf338af6e050b15b4ecce9f8ca6ae
    Closes-Bug: #1920842
    (cherry picked from commit 7f5711f77b79fdaffd91f6ef299783b058d18ace)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/789005
Committed: https://opendev.org/openstack/neutron/commit/ce2a1dc8a0f9b91761be64a5d405f16de390e1f3
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit ce2a1dc8a0f9b91761be64a5d405f16de390e1f3
Author: Takashi Kajinami <email address hidden>
Date: Tue Mar 23 11:46:34 2021 +0900

    Provide the rpc_response_max_timeout parameter to metadata-agent

    The rpc_response_max_timeout parameter is used in comminucation over
    messaging queue, thus should be available for metadata-agent which
    communicate with neutron-server over messaging queue.

    Change-Id: I4826edf16bbdf338af6e050b15b4ecce9f8ca6ae
    Closes-Bug: #1920842
    (cherry picked from commit 7f5711f77b79fdaffd91f6ef299783b058d18ace)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 15.3.4 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 17.1.2 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 16.4.0 release.