/usr/lib/accountsservice/accounts-daemon :: memory and CPU time leak

[netikras]

netikras@netikras-netbook ~/received/accountsservice-0.6.20 $ cat /etc/lsb-release
DISTRIB_ID=LinuxMint
DISTRIB_RELEASE=16
DISTRIB_CODENAME=petra
DISTRIB_DESCRIPTION="Linux Mint 16 Petra"

uname -a
Linux netikras-netbook 3.11.0-12-generic #19-Ubuntu SMP Wed Oct 9 16:12:00 UTC 2013 i686 i686 i686 GNU/Linux

Nothing. Using my netbook as usual. Did not really notice when this started since earlier today did not feel any significant slowdowns.

That's what happened:
  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
13155 root 20 0 61424 25m 2948 R 99,6 1,3 1:06.01 accounts-daemon

netikras@netikras-netbook /tmp $ while :; do ps aux | grep accounts|grep -v grep;echo; sleep 2; done
root 13155 93.6 26.1 555660 519912 pts/0 Sl 01:11 3:15 /usr/lib/accountsservice/accounts-daemon

root 13155 92.7 26.1 555660 519912 pts/0 Sl 01:11 3:15 /usr/lib/accountsservice/accounts-daemon

root 13155 91.9 26.1 555660 519912 pts/0 Sl 01:11 3:15 /usr/lib/accountsservice/accounts-daemon

accounts-daemon is using lots o CPU cycles and memory. Stats above are after restarting daemon for several times. Before this memory usage was >60%; CPU - 100%.

In file attached you should be able to see there's a loop checking for something repeatedly. Not sure this is the cause though, but feels like it..

It's the first time I've noticed this problem, but I often leave my computer running unattended so I cannot tell if it's really the first time it happened.

[netikras (netikras)]

Also...

netikras-netbook ~ # /usr/lib/accountsservice/accounts-daemon --version
accounts-daemon 0.6.34

[netikras (netikras)]

daemon is on the loose again:

top - 09:44:42 up 9:33, 10 users, load average: 3,41, 2,95, 2,02
Tasks: 224 total, 2 running, 222 sleeping, 0 stopped, 0 zombie
%Cpu(s): 14,6 us, 16,2 sy, 0,0 ni, 68,9 id, 0,0 wa, 0,1 hi, 0,3 si, 0,0 st
KiB Mem: 1988524 total, 1875556 used, 112968 free, 3044 buffers
KiB Swap: 6835196 total, 173080 used, 6662116 free, 377868 cached

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
13155 root 20 0 1025m 989m 1648 R 99,8 50,9 15:38.64 accounts-daemon

netikras@netikras-netbook /tmp $ while :; do ps aux | grep accounts|grep -v grep;echo; sleep 2; done
root 13155 3.3 50.9 1049988 1012944 pts/0 Rl 01:11 17:03 /usr/lib/accountsservice/accounts-daemon

root 13155 3.3 50.9 1049988 1012944 pts/0 Rl 01:11 17:05 /usr/lib/accountsservice/accounts-daemon

root 13155 3.3 50.9 1049988 1012944 pts/0 Rl 01:11 17:07 /usr/lib/accountsservice/accounts-daemon

root 13155 3.3 50.9 1049988 1012944 pts/0 Rl 01:11 17:09 /usr/lib/accountsservice/accounts-daemon

root 13155 3.3 50.9 1049988 1012944 pts/0 Rl 01:11 17:11 /usr/lib/accountsservice/accounts-daemon

The same daemon I've started yesterday.

[netikras (netikras)]

After posting the comment above accounts-daemon did calm down a little bit:

while :; do ps -eo user,pid,pcpu,pmem,rss,args|sort -nrk5|head -1; echo; sleep 2; done
root 13155 3.3 50.9 1012388 /usr/lib/accountsservice/accounts-daemon

root 13155 3.3 50.9 1012388 /usr/lib/accountsservice/accounts-daemon

But I still want my 1G of RAM back! I have only 2G on my netbook. If this daemon goes wild during my presentation today it will be not very nice...

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Jonathan Greenblatt (jon-50e)]

I figured it out by attaching to the process with gdb and doing a backtrace. Apparently my wtmp and btmp files got full from devices doing remote logins on a regular basis. I became suspicious that there was something particular to what I was doing based on how few people were affected by this. Here is the size of my files:

-rw-rw---- 1 root utmp 7160832 Nov 24 21:23 /var/log/btmp
-rw-rw-r-- 1 root utmp 116148480 Nov 25 22:39 /var/log/wtmp

I need to have a cleanup job for these files anyway illregardless of what the accounts-daemon does but it may help if the accounts-deamon does not process the while file if possible. I have not looked into what accounts-deamon does with the files so I am not sure if it is feasable just to look at the end of the files instead of the whole file.

Even with truncating the files accounts-deamon seems to be too much of a busy body but for now at least the %CPU is below 1% on average but I do see it peak at times to 10%. The polkitd now shows up at the top more than accounts-deamon, from the threads on this I beleive that deamon is part of the same processing.

[Jonathan Greenblatt (jon-50e)]

My solotion for now based on my comment above is to modify /etc/logrotate.conf to back these files up daily with three backups and compress them. This should hold off the problem for anyone having this due to the wtmp and btmp files being large unless they have even more logins daily than I do. Be careful how you modify the cleanup of these files, the information is needed to triage intruders.

[Sebastien Bacher (seb128)]

netikras, could you check if your b|wtmp files are also the issue?

[Launchpad Janitor (janitor)]

[Expired for accountsservice (Ubuntu) because there has been no activity for 60 days.]

[netikras (netikras)]

Sebastien,

After nearly one and a half year? I don't think my b|wtmp files have that any more even if they did back then :)

Anyway, it got fixed automatically somehow. Maybe some update came in - don't really know. But I know I haven't seen that bug since then [three knocks on wood & three spits over left shoulder].

And Thanks to you Jon, I'll keep an eye on those dump files if the problem reoccurs.

[Slash (slash-u)]

I am seeing the same issue on Ubuntu Xenial with proftpd.

I am hosting 2 fairly busy proftpd servers that handle up to 10 login/secs. My /var/log/btmp are at size 0 but the /var/log/wtmp file are up to 83M, maybe more because I restarted the servers before finding this ticket to take a look at those.

The result is that the process /usr/lib/accountsservice/accounts-daemon eats one full CPU core at all time.

[Derek Murawsky (dmurawsky)]

I am also having this issue. CPU pegged, large /var/log/wtmp. Also running ProFTPd.

[Christian Et (christian.et)]

Same here, this time on Debian 3.16.43-2+deb8u1 with kernel 3.16.0-4-amd64. I know this is an Ubuntu forum, but this bug ticket is the closest one for this issue that I could find. Package accountsservice has version 0.6.37-3+b1.

This machine is running ProFTPd 1.3.5 with a high number of FTP sessions.

As you can see the wtmp file gets rather large:
-rw------- 1 root utmp 3072 Jul 14 09:53 btmp
-rw------- 1 root utmp 44259072 Jul 17 16:07 wtmp
-rw------- 1 root utmp 119944320 Jul 17 07:49 wtmp.1
-rw------- 1 root utmp 120503424 Jul 16 07:35 wtmp.2
-rw------- 1 root utmp 120549888 Jul 15 07:35 wtmp.3

This is probably the reason why accounts-daemon is eating away most of by CPU:

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
  587 root 20 0 481984 98616 2520 R 73.5 9.6 1204:19 accounts-daemon

[Christian Et (christian.et)]

Apparently, wtmp just keeps a log of _all_ logins. Removing that file disables this logging (I have it in proftpd.log anyways) and solved my problem!

rm /var/log/wtmp

From man wtmp:
The wtmp file records all logins and logouts. (...) wtmp is maintained by login(1), init(8), and some versions of getty(8) (e.g., mingetty(8) or agetty(8)). None of these programs creates the file, so if it is removed, record-keeping is turned off.

[Christian Et (christian.et)]

Sorry for spamming, but I found that above solution is not 100% perfect: It causes warnings in proftpd.log. Instead of removing /var/log/wtmp better write the following directive in /etc/proftpd/proftpd.conf:

WtmpLog off