I have tracked down the issue, it is because that inside the chroot the environment (specifically LC_ALL is set to "C") and so when it tries to process the manuals for the other languages, like de, it bails when it sees invalid "US ASCII characters". Then later on it really fails when it can't find the localized manuals to complete the processing.
What the fix is is that you need to install the us locale in your chroot and setting LC_ALL before the build. I think the key here is that the environment is <somthing.UTF-8, but there's no reason not to use en_US, the other manuals are not broken by this process from what I see.
This failure also explains to me why it builds fine outside a chroot on my dev box or in a regular VM.
I will push a branch for this shortly.