gate-tempest-dsvm-largeops sometimes fail

[Mehdi Abaakouk]

Before oslo.messaging 1.5 the rpc.call timeout (by default the rpc_response_timeout) was never respected, but since this issue is fixed sometimes we have timeout issue in the large jobs.

for example we see two rpc.call that timeout in:
http://logs.openstack.org/73/109373/21/check/gate-tempest-dsvm-largeops-src-oslo.messaging/152ec27/logs/screen-n-cpu-1.txt.gz?level=INFO

I have tracked the msg id 42ed8c906547434fa6f63088d33340fc:

the rpc.call is done here:

  2015-01-27 14:55:41.816 6907 DEBUG oslo_messaging._drivers.amqpdriver [-] MSG_ID is 42ed8c906547434fa6f63088d33340fc _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:378

Exactly, one minute later we receive the timeout for this both message:

2015-01-27 14:56:41.823 6907 TRACE nova.compute.manager MessagingTimeout: Timed out waiting for a reply to message ID 42ed8c906547434fa6f63088d33340fc

If we check a bit later (3 seconds), oslo.messaging received the response a bit too late:

2015-01-27 14:56:44.744 6907 INFO oslo_messaging._drivers.amqpdriver [-] No calling threads waiting for msg_id : 42ed8c906547434fa6f63088d33340fc
2015-01-27 14:56:44.744 6907 INFO oslo_messaging._drivers.amqpdriver [-] No calling threads waiting for msg_id : 42ed8c906547434fa6f63088d33340fc

Perhaps we need to increase the rpc_timeout_response for this jobs or by default.

We can note that we between 320 and 640 rpc.call that run concurrently in this jobs, that represent many greenthread to schedule.

Cheers,

[Li Ma (nick-ma-z)]

Seems I also have the same problem.

http://logs.openstack.org/22/145722/3/check/gate-tempest-dsvm-largeops-src-oslo.messaging/633f627/logs/screen-n-cpu-1.txt.gz?level=ERROR

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/151979

[OpenStack Infra (hudson-openstack)]

Change abandoned by Li Ma (<email address hidden>) on branch: master
Review: https://review.openstack.org/151979
Reason: don't need it.

[Mehdi Abaakouk (sileht)]

Doesn't occurs anymore.