I had the same problem and found moving to Unity-2D stops this happening, but then you lose several area of functionality.
I had the same problem and found moving to Unity-2D stops this happening, but then you lose several area of functionality.