glusterfs high memory consumption

Bug #1792369 reported by Andrea
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
glusterfs (Ubuntu)
New
Undecided
Unassigned

Bug Description

I'm running a ctdb cluster. glusterfs is having high memory consumtion on one node.

the cluster share a CIFS share by ADS member. The activity done during this memory raising consumption is seting ACL for ownership. This is done by and from a windows machine from a system administrator. The share over with is doing this operation, has a lot of tiny files and directory trees all over a 1.5TB size.

this is a top view:

top - 13:06:02 up 2 days, 20:53, 1 user, load average: 1.15, 1.15, 1.13
Tasks: 186 total, 1 running, 108 sleeping, 0 stopped, 0 zombie
%Cpu(s): 2.1 us, 1.4 sy, 0.0 ni, 96.0 id, 0.0 wa, 0.0 hi, 0.3 si, 0.2 st
KiB Mem : 32577820 total, 376400 free, 28124128 used, 4077292 buff/cache
KiB Swap: 3999996 total, 3638524 free, 361472 used. 3932796 avail Mem

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
 1321 root 20 0 23.159g 0.022t 4864 S 16.3 72.9 361:14.21 glusterfs
 1543 root 20 0 1377412 66204 4944 S 12.6 0.2 291:52.28 glusterfsd
 2099 domusr 20 0 4387892 3.445g 190476 D 6.0 11.1 70:18.96 smbd
 1897 root -2 0 340252 229928 213384 S 1.7 0.7 49:26.85 ctdbd

here follows ubuntu-bug view report

== ApportVersion =================================
2.20.9-0ubuntu7.3

== Architecture =================================
amd64

== Date =================================
Thu Sep 13 13:02:40 2018

== Dependencies =================================
adduser 3.116ubuntu1
apt 1.6.3ubuntu0.1
apt-utils 1.6.3ubuntu0.1
attr 1:2.4.47-2build1
bzip2 1.0.6-8.1
ca-certificates 20180409
debconf 1.5.66
debconf-i18n 1.5.66
dmsetup 2:1.02.145-4.1ubuntu3
dpkg 1.19.0.5ubuntu2
e2fsprogs 1.44.1-1
fdisk 2.31.1-0.4ubuntu3.1
file 1:5.32-2ubuntu0.1
fuse 2.9.7-1ubuntu1
gcc-8-base 8-20180414-1ubuntu2
glusterfs-common 3.13.2-1build1
gpgv 2.2.4-1ubuntu1.1
ibverbs-providers 17.1-1
libacl1 2.2.52-3build1
libacl1-dev 2.2.52-3build1
libaio1 0.3.110-5
libapt-inst2.0 1.6.3ubuntu0.1
libapt-pkg5.0 1.6.3ubuntu0.1
libattr1 1:2.4.47-2build1
libattr1-dev 1:2.4.47-2build1
libaudit-common 1:2.8.2-1ubuntu1
libaudit1 1:2.8.2-1ubuntu1
libblkid1 2.31.1-0.4ubuntu3.1
libbz2-1.0 1.0.6-8.1
libc-dev-bin 2.27-3ubuntu1
libc6 2.27-3ubuntu1
libc6-dev 2.27-3ubuntu1
libcap-ng0 0.7.7-3.1
libcom-err2 1.44.1-1
libdb5.3 5.3.28-13.1ubuntu1
libdevmapper-event1.02.1 2:1.02.145-4.1ubuntu3
libdevmapper1.02.1 2:1.02.145-4.1ubuntu3
libexpat1 2.2.5-3
libext2fs2 1.44.1-1
libfdisk1 2.31.1-0.4ubuntu3.1
libffi6 3.2.1-8
libfuse2 2.9.7-1ubuntu1
libgcc1 1:8-20180414-1ubuntu2
libgcrypt20 1.8.1-4ubuntu1.1
libgmp10 2:6.1.2+dfsg-2
libgnutls30 3.5.18-1ubuntu1
libgpg-error0 1.27-6
libgpm2 1.20.7-5
libhogweed4 3.4-1
libibverbs1 17.1-1
libicu60 60.2-3ubuntu3
libidn2-0 2.0.4-1.1build2
liblocale-gettext-perl 1.07-3build2
liblvm2app2.2 2.02.176-4.1ubuntu3
liblz4-1 0.0~r131-2ubuntu3
liblzma5 5.2.2-1.3
libmagic-mgc 1:5.32-2ubuntu0.1
libmagic1 1:5.32-2ubuntu0.1
libmount1 2.31.1-0.4ubuntu3.1
libncursesw5 6.1-1ubuntu1.18.04
libnettle6 3.4-1
libnl-3-200 3.2.29-0ubuntu3
libnl-route-3-200 3.2.29-0ubuntu3
libp11-kit0 0.23.9-2
libpam-modules 1.1.8-3.6ubuntu2
libpam-modules-bin 1.1.8-3.6ubuntu2
libpam0g 1.1.8-3.6ubuntu2
libpcre3 2:8.39-9
libpython-stdlib 2.7.15~rc1-1
libpython2.7 2.7.15~rc1-1
libpython2.7-minimal 2.7.15~rc1-1
libpython2.7-stdlib 2.7.15~rc1-1
librdmacm1 17.1-1
libreadline5 5.2+dfsg-3build1
libreadline7 7.0-3
libseccomp2 2.3.1-2.1ubuntu4
libselinux1 2.7-2build2
libsemanage-common 2.7-2build2
libsemanage1 2.7-2build2
libsepol1 2.7-1
libsmartcols1 2.31.1-0.4ubuntu3.1
libsqlite3-0 3.22.0-1
libss2 1.44.1-1
libssl1.1 1.1.0g-2ubuntu4.1
libstdc++6 8-20180414-1ubuntu2
libsystemd0 237-3ubuntu10.3
libtasn1-6 4.13-2
libtext-charwidth-perl 0.04-7.1
libtext-iconv-perl 1.7-5build6
libtext-wrapi18n-perl 0.06-7.1
libtinfo5 6.1-1ubuntu1.18.04
libudev1 237-3ubuntu10.3
libunistring2 0.9.9-0ubuntu1
liburcu6 0.10.1-1
libuuid1 2.31.1-0.4ubuntu3.1
libxml2 2.9.4+dfsg1-6.1ubuntu1.2
libzstd1 1.3.3+dfsg-2ubuntu1
linux-libc-dev 4.15.0-34.37
manpages 4.15-1
manpages-dev 4.15-1
mime-support 3.60ubuntu1
mount 2.31.1-0.4ubuntu3.1
openssl 1.1.0g-2ubuntu4.1
passwd 1:4.5-1ubuntu1
perl-base 5.26.1-6ubuntu0.2
psmisc 23.1-1
python 2.7.15~rc1-1
python-asn1crypto 0.24.0-1
python-certifi 2018.1.18-2
python-chardet 3.0.4-1
python-cryptography 2.1.4-1ubuntu1.2
python-enum34 1.1.6-2
python-idna 2.6-1
python-ipaddress 1.0.17-1
python-jwt 1.5.3+ds1-1
python-minimal 2.7.15~rc1-1
python-openssl 17.5.0-1ubuntu1
python-pkg-resources 39.0.1-2
python-prettytable 0.7.2-3
python-requests 2.18.4-2
python-six 1.11.0-2
python-urllib3 1.22-1
python2.7 2.7.15~rc1-1
python2.7-minimal 2.7.15~rc1-1
readline-common 7.0-3
sed 4.4-2
tar 1.29b-2
ubuntu-keyring 2018.02.28
util-linux 2.31.1-0.4ubuntu3.1
uuid-runtime 2.31.1-0.4ubuntu3.1
xfsprogs 4.9.0+nmu1ubuntu2
xz-utils 5.2.2-1.3
zlib1g 1:1.2.11.dfsg-0ubuntu2

== DistroRelease =================================
Ubuntu 18.04

== Ec2AMI =================================
ami-f3bcb218

== Ec2AMIManifest =================================
(unknown)

== Ec2AvailabilityZone =================================
eu-central-1a

== Ec2InstanceType =================================
t3.2xlarge

== Ec2Kernel =================================
unavailable

== Ec2Ramdisk =================================
unavailable

== Package =================================
glusterfs-client 3.13.2-1build1

== PackageArchitecture =================================
amd64

== ProblemType =================================
Bug

== ProcCpuinfoMinimal =================================
processor : 7
vendor_id : GenuineIntel
cpu family : 6
model : 85
model name : Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz
stepping : 4
microcode : 0x2000043
cpu MHz : 2500.000
cache size : 33792 KB
physical id : 0
siblings : 8
core id : 3
cpu cores : 4
model : 85
model name : Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz
stepping : 4
microcode : 0x2000043
cpu MHz : 2500.000
cache size : 33792 KB
physical id : 0
siblings : 8
core id : 3
cpu cores : 4
apicid : 7
initial apicid : 7
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves ida arat pku ospke
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf
bogomips : 5000.00
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:

== ProcEnviron =================================
TERM=xterm-256color
PATH=(custom, no user)
LANG=C.UTF-8
SHELL=/bin/bash

== ProcVersionSignature =================================
Ubuntu 4.15.0-1020.20-aws 4.15.18

== SourcePackage =================================
glusterfs

== Tags =================================
 bionic ec2-images

== Uname =================================
Linux 4.15.0-1020-aws x86_64

== UpgradeStatus =================================
No upgrade log present (probably fresh install)

Revision history for this message
Tamas (tzseller) wrote :

any news regarding this?
we are experiencing high memory usage on ubuntu 18.04 too.
machine has about 256G ram, with ubuntu 18.04 bionic
after 1.5day glusterfs mount uses about 40% of available memory.

KiB Mem : 26403080+total, 2028824 free, 22798067+used, 34021312 buff/cache
KiB Swap: 66994172 total, 66517500 free, 476672 used. 34135424 avail Mem

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
13324 mysql 20 0 0.170t 0.137t 0.027t S 192.5 55.7 6215:50 mysqld
 3613 root 20 0 0.102t 0.101t 6928 S 0.0 40.9 538:31.30 glusterfs

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.