systemd pid 1 memory leak

[Sergey Borodavkin]

[impact]

pid1 leaks memory when rebuilding unit path cache

[test case]

see original description.

also, the patch contains an example of how to reproduce:

$ for i in {1..300}; do cp ~/.config/systemd/user/test0001.service ~/.config/systemd/user/test$(printf %04d $i).service; systemctl --user start test$(printf %04d $i).service;done

[regression]

any problems would occur when rebuilding the path cache, possibly resulting in memory leaks or pid1 crashes.

[scope]

this is needed only in f

fixed upstream by 3fb2326f3ed87aa0b26078d307ebfb299e36286d which is included in v246, so fixed in h and later

the code in b is very different and doesn't appear to have the leak, per original report

[original description]

Hi everybody.

We've meet a memory leak of pid1 process on the focal release.
When we launch chef-client, several systemd .service and .timers are checked for state.
Every time of this run pid1 increase VSZ/RSS on ~ 232 Kb, this don't happen on xenial and bionic releases.
I straced pid1 when that leak happen and found brk call.
On pmap view of pid 1 it's anon memory grow on the same address and all marked as dirty.
All that leak memory can be freed by calling systemctl daemon-reexec.

Searching in systemd github repo i found this commit https://github.com/systemd/systemd/commit/3fb2326f3ed87aa0b26078d307ebfb299e36286d - it may be related to this leak.

------------------------------------------------------------------------

Environment:
  Distributor ID: Ubuntu
  Description: Ubuntu 20.04.2 LTS
  Release: 20.04
  Codename: focal
  Uname: 5.4.0-77-generic #83-Ubuntu SMP Sat May 8 02:35:39 UTC 2021 x86_64

Package:
  systemd:
    Installed: 245.4-4ubuntu3.7

[Sergey Borodavkin (bocmanpy)]

Just checked again, bug still exist for latest systemd package on focal.

# apt-cache policy systemd
systemd:
  Installed: 245.4-4ubuntu3.11

[Dan Streetman (ddstreet)]

Can you check the systemd build in this ppa to see if it fixes the leak for you?
https://launchpad.net/~ddstreet/+archive/ubuntu/systemd

[Sergey Borodavkin (bocmanpy)]

Thanks for answering!
Yep, that build fixed leak for me.

# apt-cache policy systemd
systemd:
  Installed: 245.4-4ubuntu3.12~202108090622~ubuntu20.04.1

This build will be used for official systemd-245.4-4ubuntu3.12 package ?

[Dan Streetman (ddstreet)]

> This build will be used for official systemd-245.4-4ubuntu3.12 package ?

not that specific build, no, but I do queue up patches there before uploading so this should be fixed in the next upload for focal.

[Łukasz Zemczak (sil2100)]

Hello Sergey, or anyone else affected,

Accepted systemd into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/245.4-4ubuntu3.12 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted systemd (245.4-4ubuntu3.12) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

flatpak/1.6.5-0ubuntu0.3 (amd64)
gvfs/1.44.1-1ubuntu1 (amd64, arm64)
munin/2.0.56-1ubuntu1 (s390x)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Dan Streetman (ddstreet)]

@bocmanpy, can you check with the version of systemd from -proposed (instructions for how to install it are in comment 7) and verify it still fixes the issue for you?

[Sergey Borodavkin (bocmanpy)]

Yep, i checked on friday and no memory leak since then.

Version of package and source repository:

# apt-cache policy systemd
systemd:
  Installed: 245.4-4ubuntu3.12
  Candidate: 245.4-4ubuntu3.12
  Version table:
 *** 245.4-4ubuntu3.12 500
        500 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 Packages

Test was performed like i was describe at original post, i run chef-client and it start checking for all .service and .timers units, and it cause memory leak, after upgrade no mem leak.

[Brian Murray (brian-murray)]

Hello Sergey, or anyone else affected,

Accepted systemd into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/245.4-4ubuntu3.13 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted systemd (245.4-4ubuntu3.13) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

gvfs/1.44.1-1ubuntu1 (amd64, ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Sergey Borodavkin (bocmanpy)]

@brian-murray Hi. Tested that build, all seems fine, still no memory leak.

# apt-cache policy systemd
systemd:
  Installed: 245.4-4ubuntu3.13
  Candidate: 245.4-4ubuntu3.13
  Version table:
 *** 245.4-4ubuntu3.13 500
        500 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 Packages
        100 /var/lib/dpkg/status

Test was performed like i was describe at original post, i run chef-client and it start checking for all .service and .timers units, and it cause memory leak, after upgrade no mem leak.

[Chris Halse Rogers (raof)]

The verification of the Stable Release Update for systemd has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 245.4-4ubuntu3.13

---------------
systemd (245.4-4ubuntu3.13) focal; urgency=medium

  * d/p/dell-clamshell-accel-location-base-with-sku.patch:
    Revert incorrect patch (LP: #1942899)

systemd (245.4-4ubuntu3.12) focal; urgency=medium

  [ Yao Wei ]
  * d/p/dell-clamshell-accel-location-base.patch:
    Add ACCEL_LOCATION=base property for Dell clamshell models (LP: #1938259)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=5c1be33900edee94da0dc9a4ade8edcd079b4c85

  [ Lukas Märdian ]
  * Add d/p/lp1934221-resolved-disable-event-sources-before-unreffing-them.patch
    - Fix segfault in systemd-resolve (LP: #1934221)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=6c401900c70962052f56c7108fdc02fe7f84c9bf

  [ Simon Chopin ]
  * d/p/lp1914740-network-enable-DHCP-broadcast-flag-if-required-by-in.patch:
    - Apply upstream patch to fix Hipersocket DHCP mode (LP: #1914740)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=326ae43b7966d9e7c5f7124027185a79a07fa276

  [ Dan Streetman ]
  * d/p/lp1934981-correct-suspend-then-sleep-string.patch:
    Fix sleep verb used by logind during suspend-then-hibernate
    (LP: #1934981)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=997f3a7da3d5db22e3c63626c3f7dc3dff0830b0
  * d/p/lp1937238-util-return-the-correct-correct-wd-from-inotify-help.patch:
    Fix watch for time sync (LP: #1937238)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=dbabff8a03eb232c19174eff1335cd7cb7d7860c
  * d/extra/dhclient-enter-resolved-hook:
    Reset start limit counter for systemd-resolved in dhclient hook
    (LP: #1939255)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=9d3a91a0b70a4b2bcc166f366cd0a880fd494812
  * d/p/lp1935051-shared-unit-file-make-sure-the-old-hashmaps-and-sets.patch:
    Fix memory leak in path cache (LP: #1935051)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=12d6bdeb35f309158fe8d4242c6dd9be4d067604
  * d/p/lp1934147/0001-cgroup-do-catchup-for-unit-cgroup-inotify-watch-file.patch,
    d/p/lp1934147/0002-core-Make-sure-cgroup_oom_queue-is-flushed-on-manage.patch:
    Catchup cgroup inotify watch after reexec/reload (LP: #1934147)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=63eabc88b8e0005eb40b15b543538ce35377bdbd

 -- Dan Streetman <email address hidden> Tue, 07 Sep 2021 14:37:22 -0400