Comment 11 for bug 315591

So the setgid bit which causes this effect is actually on purpose; from README.Debian:

Setgid ssh-agent and environment variables
------------------------------------------

As of version 1:3.5p1-1, ssh-agent is installed setgid to prevent ptrace()
attacks retrieving private key material. This has the side-effect of causing
glibc to remove certain environment variables which might have security
implications for set-id programs, including LD_PRELOAD, LD_LIBRARY_PATH, and
TMPDIR.

If you need to set any of these environment variables, you will need to do
so in the program exec()ed by ssh-agent. This may involve creating a small
wrapper script.