Removing reading file process in a subprocess

Bug #2009962 reported by Junfeng Li
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
StarlingX
Fix Released
Low
Junfeng Li

Bug Description

Brief Description
--------------------

"Too many open files" error was seen during controller 1 startup related to lldpcli execution in a subproess. It is likely caused by reading OS release file during subprocess being called.

Severity
---------
Minor

Steps to Reproduce
---------
Unknown

Expected Behavior
------------------
Controller startup succeeds

Actual Behavior
----------------
Controller startup failed

Reproducibility
---------------
No

System Configuration
--------------------
AIO-DX

Timestamp/Logs
--------------
sysinv 2023-03-02 13:56:08.153 75460 ERROR sysinv.agent.lldp.manager [-] [Errno 24] Too many open files: OSError: [Errno 24] Too many open files
2023-03-02 13:56:08.153 75460 ERROR sysinv.agent.lldp.manager Traceback (most recent call last):
2023-03-02 13:56:08.153 75460 ERROR sysinv.agent.lldp.manager File "/usr/lib/python3/dist-packages/sysinv/agent/lldp/manager.py", line 75, in _call_drivers_and_return_array
2023-03-02 13:56:08.153 75460 ERROR sysinv.agent.lldp.manager ret = ret + method()
2023-03-02 13:56:08.153 75460 ERROR sysinv.agent.lldp.manager File "/usr/lib/python3/dist-packages/sysinv/agent/lldp/drivers/lldpd/driver.py", line 317, in lldp_neighbours_list
2023-03-02 13:56:08.153 75460 ERROR sysinv.agent.lldp.manager p = subprocess.Popen(["lldpcli", "-f", get_json_mode(), "show", "neighbor",
2023-03-02 13:56:08.153 75460 ERROR sysinv.agent.lldp.manager File "/usr/lib/python3/dist-packages/eventlet/green/subprocess.py", line 58, in __init__
2023-03-02 13:56:08.153 75460 ERROR sysinv.agent.lldp.manager subprocess_orig.Popen.__init__(self, args, 0, *argss, **kwds)
2023-03-02 13:56:08.153 75460 ERROR sysinv.agent.lldp.manager File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-03-02 13:56:08.153 75460 ERROR sysinv.agent.lldp.manager self._execute_child(args, executable, preexec_fn, close_fds,
2023-03-02 13:56:08.153 75460 ERROR sysinv.agent.lldp.manager File "/usr/lib/python3.9/subprocess.py", line 1722, in _execute_child
2023-03-02 13:56:08.153 75460 ERROR sysinv.agent.lldp.manager errpipe_read, errpipe_write = os.pipe()
2023-03-02 13:56:08.153 75460 ERROR sysinv.agent.lldp.manager OSError: [Errno 24] Too many open files

Workaround
----------
Possible lock/unlock the controller again from active controller

Changed in starlingx:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to config (master)

Reviewed: https://review.opendev.org/c/starlingx/config/+/876986
Committed: https://opendev.org/starlingx/config/commit/52323722885ec2db7e4a5b3e56e47af37a2fab6c
Submitter: "Zuul (22348)"
Branch: master

commit 52323722885ec2db7e4a5b3e56e47af37a2fab6c
Author: Junfeng (Shawn) Li <email address hidden>
Date: Thu Mar 9 10:15:34 2023 -0500

    Removing reading file process in a subprocess

    Details: This commit is to enhance the execution of lldpcli
    in the subprocess.

    The existing way is reading the OS release file during subprocess
    execution. This file reading requires an extra file descriptor.

    The commit is to only take "json0" value for Debian OS as
    moving forward only Debian OS is supported.

    Test Plan:

    PASS: upgrade from 21.12 to 22.12 with this commit
    PASS: fresh installation with this commit

    Closes-Bug: 2009962

    Signed-off-by: Junfeng (Shawn) Li <email address hidden>
    Change-Id: I1b35ab07775bbaee14a8c20513c398efcfa1d6e7

Changed in starlingx:
status: In Progress → Fix Released
Ghada Khalil (gkhalil)
Changed in starlingx:
importance: Undecided → Low
tags: added: stx.9.0 stx.config
Changed in starlingx:
assignee: nobody → Junfeng Li (jli14)
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.