Comment 18 for bug 1624013

Embarrassingly, we may have just discovered the root cause. More thorough testing is required, but it looks like there is something *very* strange going on with networking in our environment. We’ve been using physical VLANs for management/storage/neutron and it seems that somehow these VLANs are not behaving correctly.
Basic testing shows everything working, however when used in the MOS deployment environment this very slow mysql SST behaviour is seen.

We have tested creating GRE tunnels to host the management/storage/neutron networks and are unable to reproduce this issue. Our current belief is that there is some unusual behaviour in the NICs or virtual connect switch between the hosts which is somehow causing this effect.

We’re just at the start of understanding this, and don’t have any real answers yet, but I wanted to let you know that we’re now thinking this is hardware related but do not yet understand it, or why the reproduction also occurs on multiple chassis from different vendors.