neutron

neutron-servers have high memory usage

Bug #2046510 reported by LIU Yulong on 2023-12-15

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	neutron	Expired	High	Unassigned

Bug Description

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
neutron 11686 6.8 7.3 29205516 28869568 ? S Nov30 1438:32 /usr/bin/python2 /usr/bin/neutron-server --config-file /usr/share/neutron/neutron-dist.conf --config-dir /usr/share/neutron/server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini --config-dir /etc/neutron/conf.d/common --config-dir /etc/neutron/conf.d/neutron-server --log-file /var/log/neutron/server.log
neutron 11678 5.3 5.9 23972284 23636560 ? S Nov30 1117:52 /usr/bin/python2 /usr/bin/neutron-server --config-file /usr/share/neutron/neutron-dist.conf --config-dir /usr/share/neutron/server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini --config-dir /etc/neutron/conf.d/common --config-dir /etc/neutron/conf.d/neutron-server --log-file /var/log/neutron/server.log
neutron 11677 23.3 5.9 23925788 23590656 ? S Nov30 4882:29 /usr/bin/python2 /usr/bin/neutron-server --config-file /usr/share/neutron/neutron-dist.conf --config-dir /usr/share/neutron/server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini --config-dir /etc/neutron/conf.d/common --config-dir /etc/neutron/conf.d/neutron-server --log-file /var/log/neutron/server.log
neutron 11694 21.7 5.9 23909852 23574128 ? S Nov30 4550:55 /usr/bin/python2 /usr/bin/neutron-server --config-file /usr/share/neutron/neutron-dist.conf --config-dir /usr/share/neutron/server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini --config-dir /etc/neutron/conf.d/common --config-dir /etc/neutron/conf.d/neutron-server --log-file /var/log/neutron/server.log
neutron 11693 16.4 5.9 23624384 23288660 ? S Nov30 3431:10 /usr/bin/python2 /usr/bin/neutron-server --config-file /usr/share/neutron/neutron-dist.conf --config-dir /usr/share/neutron/server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini --config-dir /etc/neutron/conf.d/common --config-dir /etc/neutron/conf.d/neutron-server --log-file /var/log/neutron/server.log
neutron 11696 4.2 5.5 22357296 22021572 ? S Nov30 878:51 /usr/bin/python2 /usr/bin/neutron-server --config-file /usr/share/neutron/neutron-dist.conf --config-dir /usr/share/neutron/server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini --config-dir /etc/neutron/conf.d/common --config-dir /etc/neutron/conf.d/neutron-server --log-file /var/log/neutron/server.log

You may see the RSS reachs 28869568KB, it's about 20G+ memory.

Openstack version: stable/rocky

Revision history for this message

LIU Yulong (dragon889) wrote on 2023-12-15:

Download full text (6.1 KiB)

top shows:

top shows:

top - 11:13:01 up 36 days, 20:42,  1 user,  load average: 10.93, 13.75, 14.93
Tasks: 471 total,   8 running, 463 sleeping,   0 stopped,   0 zombie
%Cpu(s): 25.7 us,  2.1 sy,  0.0 ni, 72.0 id,  0.0 wa,  0.0 hi,  0.1 si,  0.0 st
KiB Mem : 51.0/39435072+[|||||||||||||||||||||||||||||||||||||||||||||||||||                                                 ]
KiB Swap:  0.0/0        [                                                                                                    ]

PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                        
 11697 neutron   20   0  979460 643656   2256 R 100.0  0.2   8453:23 neutron-server                                                                                  
 11701 neutron   20   0 1035752 698500   2260 R  99.0  0.2   8423:27 neutron-server                                                                                  
 11707 neutron   20   0 1005804 670000   2256 R  98.7  0.2   8570:03 neutron-server                                                                                  
 11700 neutron   20   0  921912 586112   2260 R  98.3  0.1   8412:28 neutron-server                                                                                  
 11679 neutron   20   0 8301404   7.6g   2308 R  83.4  2.0   4433:41 neutron-server                                                                                  
 11703 neutron   20   0 1017524 681620   2260 S  66.1  0.2   8450:51 neutron-server                                                                                  
 11709 neutron   20   0 1033128 697196   2256 S  45.5  0.2   8478:17 neutron-server                                                                                  
 11708 neutron   20   0 1015596 679736   2260 S  36.2  0.2   8500:55 neutron-server                                                                                  
 11705 neutron   20   0  929300 593500   2260 S  26.2  0.2   8491:52 neutron-server                                                                                  
 11738 neutron   20   0  919496 583680   2256 S  12.3  0.1   8557:00 neutron-server                                                                                  
 11691 neutron   20   0 1036120 699824   2320 R  10.6  0.2   4192:20 neutron-server                                                                                  
 11699 neutron   20   0  994560 658760   2260 S  10.3  0.2   8484:49 neutron-server                                                                                  
 11711 neutron   20   0  914588 578624   2232 S   8.6  0.1   8470:54 neutron-server                                                                                  
 11702 neutron   20   0  960064 624136   2260 S   8.0  0.2   8349:48 neutron-server                                                                                  
 11716 neutron   20   0 1034316 696896   2256 S   7.6  0.2   8473:55 neutron-server                                                                                  
 11687 neutron   20   0  596348 261216   2900 R   6.6  0.1   3595:17 neutron-server                                                                                  
 11682 neutron   20   0  771564 435840   2308 S   6.0  0.1   1268:09 neutron-server                                                                                  
 11698 neutron   20   0 1098636 762836   2260 S   2.3  0.2   8432:21 neutron-server                                                                                  
 11704 neutron   20   0 1070744 734944   2260 S   2.0  0.2   8529:55 neutron-server                                                                                  
 11710 neutron   20   0  991648 655844   2256 S   2.0  0.2   8533:53 neutron-server                                                                                  
 11678 neutron   20   0   22.9g  22.5g   2308 S   1.7  6.0   1117:57 neutron-server                                                                                  
 11689 neutron   20   0  600456 264580   2308 S   1.7  0.1   2118:31 neutron-server                                                                                  
 11706 neutron   20   0 1027472 691540   2256 S   1.7  0.2   8424:00 neutron-server                                                                                  
 11715 neutron   20   0 1042316 705700   2256 S   1.7  0.2   8544:04 neutron-server                                                                                  
 11713 neutron   20   0 1037936 700512   2256 S   1.3  0.2   8525:36 neutron-server                                                                                  
 11680 neutron   20   0 1367716   1.0g   2900 S   1.0  0.3   3182:37 neutron-server                                                                                  
 11685 neutron   20   0 8726108   8.0g   2316 S   1.0  2.1   3894:04 neutron-server                                                                                  
 11690 neutron   20   0  883208 547480   2308 S   1.0  0.1 823:50.83 neutron-server                                                                                  
 11692 neutron   20   0  585372 249644   2308 S   1.0  0.1   1692:47 neutron-server                                                                                  
 11712 neutron   20   0 1020736 684512   2256 S   1.0  0.2   8443:05 neutron-server                                                                                  
 11714 neutron   20   0  976472 640636   2232 S   1.0  0.2   8565:23 neutron-server                                                                                  
 11662 neutron   20   0  946420 616752   8404 S   0.7  0.2   8283:01 neutron-server                                                                                  
 11681 neutron   20   0 3669000   3.2g   2320 S   0.7  0.8   1134:06 neutron-server                                                                                  
 11695 neutron   20   0  599992 264236   2308 S   0.7  0.1   4186:04 neutron-server

Revision history for this message

LIU Yulong (dragon889) wrote on 2023-12-15 (last edit on 2023-12-15):

High memory usage will exist forever without reduction.

Revision history for this message

LIU Yulong (dragon889) wrote on 2023-12-15 (last edit on 2023-12-15):

In our test env we can see almost all neutron-server workers can have high memory usage:

10G for rpc worker:

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
neutron 100412 8.2 3.8 10389244 10052552 ? R Dec13 210:54 neutron-server: rpc worker (/usr/bin/python2 /usr/bin/neutron-server --config-file /usr/share/neutron/neutron-dist.conf --config-dir /usr/share/neutron/server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini --config-dir /etc/neutron/conf.d/common --config-dir /etc/neutron/conf.d/neutron-server --log-file /var/log/neutron/server.log)

8G for API worker:

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
neutron 100406 2.6 3.1 8635096 8298684 ? S Dec13 67:18 neutron-server: api worker (/usr/bin/python2 /usr/bin/neutron-server --config-file /usr/share/neutron/neutron-dist.conf --config-dir /usr/share/neutron/server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini --config-dir /etc/neutron/conf.d/common --config-dir /etc/neutron/conf.d/neutron-server --log-file /var/log/neutron/server.log)

6G for periodic worker:

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
neutron 100466 6.1 2.4 6701964 6363964 ? S Dec13 157:53 neutron-server: periodic worker (/usr/bin/python2 /usr/bin/neutron-server --config-file /usr/share/neutron/neutron-dist.conf --config-dir /usr/share/neutron/server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini --config-dir /etc/neutron/conf.d/common --config-dir /etc/neutron/conf.d/neutron-server --log-file /var/log/neutron/server.log)

Only "rpc state report workers" have average and consistent memory usage.

Revision history for this message

yatin (yatinkarel) wrote on 2023-12-18:

Thanks @LIU for the bug report.

Rocky is quite old and EOL release, would it be possible for you to confirm if you see same issue with newer OpenStack releases?

Found an old bug for memory leak https://bugs.launchpad.net/neutron/+bug/1942179 which was fixed in rocky, can you also check if that is already included in your environment?

How/When you observed this issue? like after initial deployment, after upgrade, after particular scale usage, etc?

How memory consumption increases? like what's the memory utilization just after neutron restart, vs utilization after some hours/days of usage, utilization increase with/without requests/operations for neutron?

Also please do share your environment/deployment details including neutron configurations and usage details.

Marking the bug as INCOMPLETE for now.

Changed in neutron:
status:	New → Incomplete

Revision history for this message

Rodolfo Alonso (rodolfo-alonso-hernandez) wrote on 2023-12-18:

Yeah, I was going to point to the same LP bug Yatin commented. That seems related to the issue reported. The memory consumption happens when trying to access to a non-existing resource in the API route. For example, as commented in the LP#1942179 description:
GET http://10.120.0.30:9696/v2.0/blabla

Revision history for this message

LIU Yulong (dragon889) wrote on 2023-12-19 (last edit on 2023-12-19):

@yatin @Rodolfo
I noticed that bug #1942179, I analysised the log which had no result of such none-exists API call. And more, this issue is not only neutron-API, but also RPC and periodic worker.

@yating
Rocky is quite old and EOL release, would it be possible for you to confirm if you see same issue with newer OpenStack releases?

We have no such deployment, but IMO, the API/woker/services related codes are basically same, so this issue should exist in newer releases.

How/When you observed this issue? like after initial deployment, after upgrade, after particular scale usage, etc?

Almost every deployments have such issue.

Memory useage before restart:
https://paste.openstack.org/show/brvSXpKW7Cci7zazR2Le/

Memory useage after restart:
https://paste.openstack.org/show/baeD8TvsAAn4bSGk3C4T/

I will paste these neutron-server processes after running few hours.

Changed in neutron:
importance:	Undecided → High

Revision history for this message

LIU Yulong (dragon889) wrote on 2023-12-19 (last edit on 2023-12-19):

Same neutron-server processes after running few hours:
https://paste.openstack.org/show/bQiVDsmWC7CygumT8STy/

here are some examples:

yet after restart:
neutron 114273 17.2 0.0 514960 176780 ? S 11:53 0:13 neutron-server: rpc worker (/usr/bin/python2 /usr/bin/neutron-server --config-file /usr/share/neutron/neutron-dist.conf --config-dir /usr/share/neutron/server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini --config-dir /etc/neutron/conf.d/common --config-dir /etc/neutron/conf.d/neutron-server --log-file /var/log/neutron/server.log)

few hours after restart:
neutron 114273 31.2 2.7 7486396 7150056 ? S 11:53 11:53 neutron-server: rpc worker (/usr/bin/python2 /usr/bin/neutron-server --config-file /usr/share/neutron/neutron-dist.conf --config-dir /usr/share/neutron/server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini --config-dir /etc/neutron/conf.d/common --config-dir /etc/neutron/conf.d/neutron-server --log-file /var/log/neutron/server.log)

Its memory usage is about 7Gb. And it is a RPC work, not API worker.

Revision history for this message

Liu Xie (liushy) wrote on 2023-12-20:

Maybe it is a matter for Python with glibc malloc in large scale, you could try using jemalloc instead:

https://opendev.org/opendev/system-config/commit/d7c0be958df516c70eb7955e1783da991c3d36fb

Revision history for this message

Launchpad Janitor (janitor) wrote on 2024-02-18:

[Expired for neutron because there has been no activity for 60 days.]

Changed in neutron:
status:	Incomplete → Expired

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.