Update to this. I sent the quick patch for upstream review and it seems this needs a more complex approach. Regarding the warning the response was that it is bogus for any platform beside ARM. But is there because ARM might sleep in there.
I will try to get a better approach together but that needs a bit of thinking.