2016-09-22 15:14:01 INFO [11824] Starting OS provisioning for nodes: 6,21,23,59,74,76,78,93,98,100,106,110,114,123,149,152,157,160,165,169,175,177,180,186,195,198,201,21 4,218,220,230,235,236,244,250,253,265,267,273,279,288,291,299,303,304,312,330,336,342,344,354,358,367,368,370,373,379,384,393,411,425,427,441,450,452,461,466,470,479,486 ,488,490,494,495,515,516,532,554,555,560,561,564,565,571,573,574,577,578,580,591,592,601,605,608,611,615,616,623,629,638,641,654,658,662,675,677,681,689,695,700,702,712, 720,724,728,729,732,737,738,742,744,746,747,749,751,753,754,758,763,764,773,775,776,786,793,795,801,802,807,810,811,812,816,819,820,824,827,830,831,836,837,838,846,851,8 54,855,860,864,876,888,891,894,899,900,905,910,911,912,918,921,922,923,924,927,928,930,934,937,945,950,952,958,960,968,970,973,974,977,982,988,990,991,1001,1005,1011,101 6,1027,1028,1030,1034 2016-09-22 15:14:11 DEBUG [11824] Starting provision Uploading provision data, sync task. 2016-09-22 15:15:25 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 6 took 00:00:01 2016-09-22 15:15:26 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 21 took 00:00:01 2016-09-22 15:15:27 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 23 took 00:00:01 2016-09-22 15:15:28 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 59 took 00:00:01 2016-09-22 15:15:29 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 74 took 00:00:01 2016-09-22 15:15:30 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 76 took 00:00:01 2016-09-22 15:15:31 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 78 took 00:00:01 2016-09-22 15:15:32 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 93 took 00:00:01 2016-09-22 15:15:33 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 98 took 00:00:01 2016-09-22 15:15:34 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 100 took 00:00:01 2016-09-22 15:15:35 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 106 took 00:00:01 2016-09-22 15:15:36 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 110 took 00:00:01 2016-09-22 15:15:37 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 114 took 00:00:01 2016-09-22 15:15:38 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 123 took 00:00:01 2016-09-22 15:15:39 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 149 took 00:00:01 2016-09-22 15:15:40 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 152 took 00:00:01 2016-09-22 15:15:41 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 157 took 00:00:01 2016-09-22 15:15:42 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 160 took 00:00:01 2016-09-22 15:15:43 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 165 took 00:00:01 2016-09-22 15:15:44 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 169 took 00:00:01 2016-09-22 15:15:45 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 175 took 00:00:01 2016-09-22 15:15:46 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 177 took 00:00:01 2016-09-22 15:15:48 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 180 took 00:00:01 2016-09-22 15:15:49 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 186 took 00:00:01 2016-09-22 15:15:50 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 195 took 00:00:01 2016-09-22 15:15:51 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 198 took 00:00:01 2016-09-22 15:15:52 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 201 took 00:00:01 2016-09-22 15:15:53 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 214 took 00:00:01 2016-09-22 15:15:54 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 218 took 00:00:01 2016-09-22 15:15:55 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 220 took 00:00:01 2016-09-22 15:15:56 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 230 took 00:00:01 2016-09-22 15:15:57 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 235 took 00:00:01 2016-09-22 15:15:58 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 236 took 00:00:01 2016-09-22 15:15:59 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 244 took 00:00:01 2016-09-22 15:16:00 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 250 took 00:00:01 2016-09-22 15:16:01 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 253 took 00:00:01 2016-09-22 15:16:02 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 265 took 00:00:01 2016-09-22 15:16:03 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 267 took 00:00:01 2016-09-22 15:16:04 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 273 took 00:00:01 2016-09-22 15:16:05 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 279 took 00:00:01 2016-09-22 15:16:06 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 288 took 00:00:01 2016-09-22 15:16:07 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 291 took 00:00:01 2016-09-22 15:16:08 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 299 took 00:00:01 2016-09-22 15:16:09 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 303 took 00:00:01 2016-09-22 15:16:10 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 304 took 00:00:01 2016-09-22 15:16:11 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 312 took 00:00:01 2016-09-22 15:16:12 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 330 took 00:00:01 2016-09-22 15:16:13 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 336 took 00:00:01 2016-09-22 15:16:14 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 342 took 00:00:01 2016-09-22 15:16:15 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 344 took 00:00:01 2016-09-22 15:16:15 DEBUG [11824] a74c915b-dde3-4284-83a6-fe0437c48a59: running provision script: 6, 21, 23, 59, 74, 76, 78, 93, 98, 100, 106, 110, 114, 123, 149, 152, 157, 160, 165, 169, 175, 177, 180, 186, 195, 198, 201, 214, 218, 220, 230, 235, 236, 244, 250, 253, 265, 267, 273, 279, 288, 291, 299, 303, 304, 312, 330, 336, 342, 344 Provisioning took less than 5mins, no delays. Good. Next bunch of nodes. 2016-09-22 15:20:45 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 354 took 00:00:01 2016-09-22 15:20:46 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 358 took 00:00:01 2016-09-22 15:20:47 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 367 took 00:00:01 2016-09-22 15:20:48 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 368 took 00:00:01 2016-09-22 15:20:49 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 370 took 00:00:01 2016-09-22 15:20:50 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 373 took 00:00:01 2016-09-22 15:20:51 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 379 took 00:00:01 2016-09-22 15:20:52 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 384 took 00:00:01 2016-09-22 15:20:53 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 393 took 00:00:01 2016-09-22 15:20:54 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 411 took 00:00:01 2016-09-22 15:20:55 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 425 took 00:00:01 2016-09-22 15:20:56 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 427 took 00:00:01 2016-09-22 15:20:57 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 441 took 00:00:01 2016-09-22 15:20:58 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 450 took 00:00:01 2016-09-22 15:20:59 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 452 took 00:00:01 2016-09-22 15:21:00 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 461 took 00:00:01 2016-09-22 15:21:01 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 466 took 00:00:01 2016-09-22 15:21:02 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 470 took 00:00:01 2016-09-22 15:21:03 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 479 took 00:00:01 2016-09-22 15:21:04 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 486 took 00:00:01 2016-09-22 15:21:05 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 488 took 00:00:01 2016-09-22 15:21:06 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 490 took 00:00:01 2016-09-22 15:21:07 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 494 took 00:00:01 2016-09-22 15:21:09 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 495 took 00:00:02 2016-09-22 15:21:10 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 515 took 00:00:01 2016-09-22 15:21:11 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 516 took 00:00:01 2016-09-22 15:21:12 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 532 took 00:00:01 2016-09-22 15:21:13 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 554 took 00:00:01 2016-09-22 15:21:14 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 555 took 00:00:01 2016-09-22 15:21:15 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 560 took 00:00:01 2016-09-22 15:21:16 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 561 took 00:00:01 2016-09-22 15:21:17 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 564 took 00:00:01 2016-09-22 15:21:18 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 565 took 00:00:01 2016-09-22 15:21:19 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 571 took 00:00:01 2016-09-22 15:21:20 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 573 took 00:00:01 2016-09-22 15:21:21 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 574 took 00:00:01 2016-09-22 15:21:22 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 577 took 00:00:01 2016-09-22 15:21:23 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 578 took 00:00:01 2016-09-22 15:21:24 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 580 took 00:00:01 2016-09-22 15:21:25 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 591 took 00:00:01 2016-09-22 15:21:26 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 592 took 00:00:01 2016-09-22 15:21:27 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 601 took 00:00:01 2016-09-22 15:21:28 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 605 took 00:00:01 2016-09-22 15:21:29 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 608 took 00:00:01 2016-09-22 15:21:30 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 611 took 00:00:01 2016-09-22 15:21:31 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 615 took 00:00:01 2016-09-22 15:21:32 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 616 took 00:00:01 2016-09-22 15:21:33 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 623 took 00:00:01 2016-09-22 15:21:34 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 629 took 00:00:01 2016-09-22 15:21:35 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 638 took 00:00:01 2016-09-22 15:21:35 DEBUG [11824] a74c915b-dde3-4284-83a6-fe0437c48a59: running provision script: 354, 358, 367, 368, 370, 373, 379, 384, 393, 411, 425, 427, 441, 450, 452, 461, 466, 470, 479, 486, 488, 490, 494, 495, 515, 516, 532, 554, 555, 560, 561, 564, 565, 571, 573, 574, 577, 578, 580, 591, 592, 601, 605, 608, 611, 615, 616, 623, 629, 638 Provisioning took less than 5mins, no delays. Good. Next bunch of nodes. 2016-09-22 15:26:14 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 641 took 00:00:01 2016-09-22 15:26:16 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 654 took 00:00:01 2016-09-22 15:26:17 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 658 took 00:00:01 2016-09-22 15:26:18 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 662 took 00:00:01 2016-09-22 15:26:19 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 675 took 00:00:01 2016-09-22 15:26:20 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 677 took 00:00:01 2016-09-22 15:26:21 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 681 took 00:00:01 2016-09-22 15:26:22 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 689 took 00:00:01 2016-09-22 15:26:23 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 695 took 00:00:01 2016-09-22 15:26:24 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 700 took 00:00:01 2016-09-22 15:26:25 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 702 took 00:00:01 2016-09-22 15:26:26 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 712 took 00:00:01 2016-09-22 15:26:27 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 720 took 00:00:01 2016-09-22 15:37:56 DEBUG [11824] Task time summary: upload_provision_data with status failed on node 724 took 00:11:28 2016-09-22 15:37:57 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 728 took 00:00:01 2016-09-22 15:37:58 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 729 took 00:00:01 2016-09-22 15:37:59 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 732 took 00:00:01 2016-09-22 15:49:27 DEBUG [11824] Task time summary: upload_provision_data with status failed on node 737 took 00:11:28 2016-09-22 16:00:54 DEBUG [11824] Task time summary: upload_provision_data with status failed on node 738 took 00:11:27 2016-09-22 16:12:22 DEBUG [11824] Task time summary: upload_provision_data with status failed on node 742 took 00:11:28 2016-09-22 16:12:23 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 744 took 00:00:01 2016-09-22 16:12:24 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 746 took 00:00:01 2016-09-22 16:12:25 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 747 took 00:00:01 2016-09-22 16:12:26 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 749 took 00:00:01 2016-09-22 16:12:27 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 751 took 00:00:01 2016-09-22 16:12:28 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 753 took 00:00:01 2016-09-22 16:12:29 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 754 took 00:00:01 2016-09-22 16:12:30 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 758 took 00:00:01 2016-09-22 16:12:31 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 763 took 00:00:01 2016-09-22 16:12:32 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 764 took 00:00:01 2016-09-22 16:12:33 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 773 took 00:00:01 2016-09-22 16:12:34 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 775 took 00:00:01 2016-09-22 16:12:35 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 776 took 00:00:01 2016-09-22 16:24:04 DEBUG [11824] Task time summary: upload_provision_data with status failed on node 786 took 00:11:29 2016-09-22 16:24:05 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 793 took 00:00:01 2016-09-22 16:24:06 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 795 took 00:00:01 2016-09-22 16:24:07 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 801 took 00:00:01 2016-09-22 16:24:08 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 802 took 00:00:01 2016-09-22 16:24:09 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 807 took 00:00:01 2016-09-22 16:24:10 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 810 took 00:00:01 2016-09-22 16:24:11 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 811 took 00:00:01 2016-09-22 16:24:13 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 812 took 00:00:02 2016-09-22 16:35:42 DEBUG [11824] Task time summary: upload_provision_data with status failed on node 816 took 00:11:29 2016-09-22 16:35:43 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 819 took 00:00:01 2016-09-22 16:35:44 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 820 took 00:00:01 2016-09-22 16:35:45 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 824 took 00:00:01 2016-09-22 16:35:46 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 827 took 00:00:01 2016-09-22 16:35:47 DEBUG [11824] Task time summary: upload_provision_data with status successful on node 830 took 00:00:01 2016-09-22 16:47:17 DEBUG [11824] Task time summary: upload_provision_data with status failed on node 831 took 00:11:30 2016-09-22 16:58:45 DEBUG [11824] Task time summary: upload_provision_data with status failed on node 836 took 00:11:28 2016-09-22 16:58:45 DEBUG [11824] a74c915b-dde3-4284-83a6-fe0437c48a59: running provision script: 641, 654, 658, 662, 675, 677, 681, 689, 695, 700, 702, 712, 720, 728, 729, 732, 744, 746, 747, 749, 751, 753, 754, 758, 763, 764, 773, 775, 776, 793, 795, 801, 802, 807, 810, 811, 812, 819, 820, 824, 827, 830 Delayed by 8 * 11.5 mins = 5520 secs = 1.5H way too much! HORRIBLE!