There is no some single migration step which takes long time or hangs. It is just very slow all time in case of failed test. E.g. first step, called "Running upgrade -> kilo" took 6 minutes (!) in failed test and 2.5 second in good run.
Comparing dstat from those 2 runs I can see that in this failed test there was less than 100 IOPS for most of the time. In case of this passed test it was around 2k - 3k IOPS.
So I think that this issue is just because of some slow VMs which maybe are on some overloaded nodes.
Today I checked one failed test, from http:// logs.openstack. org/30/ 626830/ 9/check/ neutron- functional/ e66470c/ logs/dsvm- functional- logs/neutron. tests.functiona l.db.test_ migrations. TestModelsMigra tionsMysql. test_check_ mysql_engine. txt.gz and compared it with same test which passed: http:// logs.openstack. org/85/ 624685/ 2/check/ neutron- functional/ fede00f/ logs/dsvm- functional- logs/neutron. tests.functiona l.db.test_ migrations. TestModelsMigra tionsMysql. test_check_ mysql_engine. txt.gz
There is no some single migration step which takes long time or hangs. It is just very slow all time in case of failed test. E.g. first step, called "Running upgrade -> kilo" took 6 minutes (!) in failed test and 2.5 second in good run.
Comparing dstat from those 2 runs I can see that in this failed test there was less than 100 IOPS for most of the time. In case of this passed test it was around 2k - 3k IOPS.
So I think that this issue is just because of some slow VMs which maybe are on some overloaded nodes.