EDNS0: res_nopt truncates buffer size incorrectly

[Hauke Lampe]

Resolver functions allow buffer sizes > 65535 bytes. If RES_USE_EDNS0 is set, res_nopt() truncates this value to 16 bit, resulting in an incorrect buffer size advertised in EDNS query headers.

portable OpenSSH triggers this behaviour, as described here:
http://marc.info/?l=openssh-unix-dev&m=124625332427704&w=2

openbsd-compat/getrrsetbyname() sets a buffer size of 65536 bytes. In the glibc stub-resolver, it is eventually passed on as "anslen" to __res_nopt() in resolv/res_mkquery.c:
[...]
        NS_PUT16(anslen & 0xffff, cp); /* CLASS = UDP payload size */

and sent out to the recursor (UDPsize: 0xf0000 & 0xffff == 0)

| IP 127.0.0.1.44138 > 127.0.0.1.53: 31454+ [1au] SSHFP?
orbit.attraktor.org. ar: . OPT UDPsize=0 (48)
| IP 127.0.0.1.53 > 127.0.0.1.44138: 31454 ServFail-| [0q] 0/0/0 (12)

[Bugzilla-webform (bugzilla-webform)]

Resolver functions allow buffer sizes > 65535 bytes. If RES_USE_EDNS0 is set,
res_nopt() truncates this value to 16 bit, resulting in an incorrect buffer size
advertised in EDNS query headers.

portable OpenSSH triggers this behaviour, as described here:
http://marc.info/?l=openssh-unix-dev&m=124625332427704&w=2

openbsd-compat/getrrsetbyname() sets a buffer size of 65536 bytes. In the glibc
stub-resolver, it is eventually passed on as "anslen" to __res_nopt() in
resolv/res_mkquery.c:
[...]
        NS_PUT16(anslen & 0xffff, cp); /* CLASS = UDP payload size */

and sent out to the recursor (UDPsize: 0xf0000 & 0xffff == 0)

| IP 127.0.0.1.44138 > 127.0.0.1.53: 31454+ [1au] SSHFP?
orbit.attraktor.org. ar: . OPT UDPsize=0 (48)
| IP 127.0.0.1.53 > 127.0.0.1.44138: 31454 ServFail-| [0q] 0/0/0 (12)

[Bugzilla-webform (bugzilla-webform)]

Created attachment 4035
cap anslen in res_nopt() at 0xffff. assert() inserted because I'm not sure if negative values could be passed to res_nopt()

cap anslen in res_nopt() at 0xffff

assert() inserted because I'm not sure if negative values could be passed to
res_nopt()

[Hauke Lampe (hauke)]

reported upstream: http://sources.redhat.com/bugzilla/show_bug.cgi?id=10360

[Drepper-fsp (drepper-fsp)]

The code comes from bind upstream. I've checked in a similar patch.

[Hauke Lampe (hauke)]

fixed upstream:
http://sources.redhat.com/git/gitweb.cgi?p=glibc.git;a=commitdiff;h=1c0ab5bd34159d2ae53390571113844ebadc161b

[Matthias Klose (doko)]

fixed in lucid and maverick