Too many open files for nova-compute in Wallaby upgrade

[YG Kumar]

Hi,

We have upgraded to Wallaby and in our computes we are observing nova-compute service is down.
The logs seem to indicate "Too many open files". Can someone help us with this issue.

-----------
:~# lsof -P -M -l -n -d '^cwd,^err,^ltx,^mem,^mmap,^pd,^rtd,^txt' -p 2394860 -a | awk '{if (NR>1) print}' | wc -l
1031
--------

The above process is a nova-compute process

The OS version is Ubuntu 20.04. The hypervisor is KVM

I am attaching the lsof output for this process. Please check that.

Thanks

[sean mooney (sean-k-mooney)]

marking this as a duplicate of https://bugs.launchpad.net/oslo.messaging/+bug/1949964

based on comment 6
https://bugs.launchpad.net/oslo.messaging/+bug/1949964/comments/6

setting

[oslo_messaging_rabbit]
heartbeat_in_pthread = False

seams to be a workaround but its not clear exactly why.

this should be set to false for all nova services except for the nova-api
the nova-api should have it set to true if it is running under uwsgi or apache mod_wsgi.

[YG Kumar (ygk-kmr)]

Thanks Sean for the response.

Can you let me know in which configuration file/files exactly this change has to be made in Wallaby ?

[YG Kumar (ygk-kmr)]

Also, can someone let me know, how to find the versions of oslo_messaging and amqp in OSA setups ?

[YG Kumar (ygk-kmr)]

I see more eventpoll FDs:

-----
nova-comp 263082 nova 38u a_inode 0,14 0 12904 [eventpoll]
nova-comp 263082 nova 39u sock 0,8 0t0 4231344080 protocol: TCP
nova-comp 263082 nova 40u a_inode 0,14 0 12904 [eventpoll]
nova-comp 263082 nova 41u sock 0,8 0t0 4232482200 protocol: TCP
nova-comp 263082 nova 42u a_inode 0,14 0 12904 [eventpoll]
nova-comp 263082 nova 43u sock 0,8 0t0 4232841823 protocol: TCP
-----

Why are there so many sockets opened and many eventpolls ?

[YG Kumar (ygk-kmr)]

Can someone respond ?