It looks like the fundamental problem is that gpg-agent is started with X from Xsessions.d, but doesn't get registered (presumably RegisterClient()), so when X exists it doesn't know to stop gpg-agent.
It looks like the fundamental problem is that gpg-agent is started with X from Xsessions.d, but doesn't get registered (presumably RegisterClient()), so when X exists it doesn't know to stop gpg-agent.