You’re right: glibc seems to treat the absence of libnss-resolve itself as UNAVAIL, which is the same code returned on DNSSEC validation failures when libnss-resolve is working. I don’t see a way around this other than patching libnss-resolve to return NOTFOUND (or TRYAGAIN?) on validation failure.
It looks like there may be other ways for an active attacker to force a TRYAGAIN code (with a response that doesn’t fit in the caller-provided buffer), which suggests that the right configuration is [!UNAVAIL=return], not merely [NOTFOUND=return].
You’re right: glibc seems to treat the absence of libnss-resolve itself as UNAVAIL, which is the same code returned on DNSSEC validation failures when libnss-resolve is working. I don’t see a way around this other than patching libnss-resolve to return NOTFOUND (or TRYAGAIN?) on validation failure.
It looks like there may be other ways for an active attacker to force a TRYAGAIN code (with a response that doesn’t fit in the caller-provided buffer), which suggests that the right configuration is [!UNAVAIL=return], not merely [NOTFOUND=return].