[Not sure If I've filed this aginst the right package. It should be linux-3.5.0, but that doesn't exist according to launchpad. ]
Attached is a first-cut patch adding arm64 support to the linux-3.5.0 package so that it builds linux-source packages which can be used to build cross tools against. The arm64 support is just the core stuff so isn't much use for making binaries. It's also an unholy backport of 3.6/3.7 vintage stuff to 3.5 so it probably bust in important ways. However that's not relevant for the primiary purpose here of building cross-tools, which just needs a correct set of headers in linux-libc-dev, and arch/package machinery in linux-source.
I'm not expecting this to be uploaded but wanted to put it somewhere public for anyone else to build with.