[SRU] Poor performance of libnss-db on large db files

[Ponnuvel Palaniyappan]

libdnss-db opens and closes the DB file each time. This is normally not an issue as the DB files are small.

However, in a user environment, this takes considerably longer because the DB files have entries to the tune of 20k which severely impacts the performance of lookups.

----

[ Impact ]

The libnss-db library, which processes Berkeley DB files, closes
and re-opens the DB file for each entry in the file that it currently scans.
This results in very poor performance and noticiable botteneck on the
application side. In a user environment that has 20k entries in the DB,
this results in about 40 times worse performance (compared to keeping
the file open).

[ Test Plan ]

The test plan sets up some DB files and queries them to see if the open(2) and close(2) calls are repeatedly called.

1. update /etc/default/libnss-db
DBS = passwd group shadow
VAR_DB = /var/lib/misc
ETC = ${VAR_DB}/etc
AWK = awk
MAKEDB = makedb --quiet

2. update /etc/nsswitch.conf
passwd: files systemd db
group: files systemd db
shadow: files systemd db

3. create required files
mkdir -p /var/lib/misc/etc
touch /var/lib/misc/etc/{passwd,group,shadow}
cp /etc/login.defs /var/lib/misc/etc

4. update /var/lib/misc/etc/login.defs so that UIDs are less likely to conflict
UID_MIN 5000

5. create test users in berkelydb file
cd /var/lib/misc
for i in $(seq 0001 1000); do useradd -P /var/lib/misc test$i; done
make

6. verify users in berkeleydb is recognized
id test1000

7. verify that 'getent passwd' and 'getenv group' calls correctly retrieve the entr fom
verify that there's not as many open/close calls as there are entries in the DB files

[ Where problems could occur ]

To my understanding the historical reason for doing close-open for each line is to
minimise possible resource usage and reduce the possibility of DB files getting stale
if DB files get updated externally. Potential risks revolve around those two.

In the former case, this could result in higher load on the system (modern hardware should be capable of handling that).
In the latter case, lookups could fail and likely result in re-scans from the application.

[ Other Info ]

Debian bug and patch(not merged): https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1101371

[Ponnuvel Palaniyappan (pponnuvel)]

Attaching the debdiff for Questing.

[Lukas Märdian (slyon)]

Thank you for preparing a patch!

We're currently in Beta freeze for Questing, so this can only be uploaded starting next week. Please re-ping the corresponding patch-pilot next week.

For now, I can provide you with some review comments for improvement of the patch:

- debian/changelog: Please use a proper reference to the Launchpad bug so that it can be detected by the automation, e.g. (LP: #2121543)
- debian/patches/bug2121543.patch: (nitpick) we usually refer to bugs as "lp-2121543-SHORT-DESC.patch", might (or might not) consider renaming the patch file, e.g. "lp-2121543-do-not-close-and-reopen-DB.patch"
- kudos: Thanks for adding DEP-3 headers
  => But you're using Author and Origin at the same time. Was the patch created by you, or does it originate from upstream? You should delete one or the other header.
  => Also, you should add a full URL to the actual upstream patch/commit (if merged already), "Origin: upstream, https://..."
  => If it's not yet merged upstream, you should consider adding a "Forwarded: ..." header, pointing to a full URL where the change has been proposed/discussed with the upstream developers.
  => The "Bug" header should probably read "Bug-Debian", as it's not pointing towards the upstream bug tracker.

[Ponnuvel Palaniyappan (pponnuvel)]

Thanks Lukas for the review!

I've modified and re-did the patch and uploaded. Please take a look. Thanks.

[Lukas Märdian (slyon)]

Hi Ponnuvel, thank you for addressing my remarks from comment #2!

Unfortunately, Ubuntu 25.10 is in final freeze by now, so we'll need to make this an SRU.

Could you please modify the bug description according to the corresponding template? https://documentation.ubuntu.com/sru/en/latest/reference/bug-template/ – we would need this anyway to get the change landed in 24.04 LTS eventually.

Also I have one open question left:
In the patch you mention "Origin: upstream" without giving a specific reference. Was this patch discussed and/or discussed with the upstream community? Can you please point us to the original upstream changes?

I see you set "Forwarded: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1101371" – Should Debian be considered as the upstream of libnss-db? It seems to be an orphaned package in Debian, owned by the QA team and there has been no reaction so far on this patch.

All of this unclarity about the upstream situation makes me a bit hesitant to upload this patch into the Ubuntu packages. Would you be able to point us to the upstream reference of this patch, or start a discussion with upstream (not Debian, but the actual upstream libnss-db project) about it, before introducing it in downstream distributions? It seems to be a generally useful change that others might want to have, too!

[Andreas Hasenack (ahasenack)]

Removing sponsors, please resubscribe when the questions from the comment above were addressed/answered, thanks.

[Ponnuvel Palaniyappan (pponnuvel)]

libdnss-db opens and closes the DB file each time. This is normally not an issue as the DB files are small. This results in considerably poor performance when the DB files large (e.g. in an environment with 20k lines of DB file, the performance was 40times compared to keeping it open for processing all lines).

Rather than re-opening for scanning each line, the file can be kept open for each call into libnss as proposed in [1].

Reported elsewhere:

[0] https://bugs.launchpad.net/ubuntu/+source/libnss-db/+bug/2121543
[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1101371

[Ponnuvel Palaniyappan (pponnuvel)]

Created attachment 16429
Proposed patch

Attaching the proposed patch here. Also sent to glibc-alpha.

[Ponnuvel Palaniyappan (pponnuvel)]

Hi Lukas,

Thanks for looking into again. You're right that it's not "upstream upstream" (Debian's); rather a proposal to Debian. Since the original code is a part of glibc, I've opened a bug against glibc:

https://sourceware.org/bugzilla/show_bug.cgi?id=33558

And emailed glibc-alpha to get the opinion of the nss maintainers in glibc:

https://marc.info/?l=glibc-alpha&m=176102749124614&w=3

I can followup on that and update here based on the discussion there. So for now, I think it's ok to keep this on hold. I'll come back once I've concrete info from glibc. Thanks!