TimeoutError: Waiting task u'deploy' timeout

[Valentyn Yakovlev]

Steps to reproduce:
  1. Create new environment
  2. Choose Neutron, VLAN
  3. Add 4 controller
  4. Add 2 compute
  5. Add 3 cinder
  6. Change disk configuration for all Cinder nodes.
     Change 'Cinder' volume for vdc
  7. Verify networks
  8. Deploy the environment << Fail
  9. Verify networks
  10. Check disk configuration
  11. Run OSTF tests

Expected result:
  Cluster is ready after deployment

Actual result:
  TimeoutError: Waiting task u'deploy' timeout 7800 sec was exceeded

In UI:
Deployment has failed. All nodes are finished. Failed tasks: Task[openstack-network-server- config/1], Task[openstack-network-server-config/2], Task[openstack-network-server-config/5], Task[openstack-network-server-config/4] Stopping the deployment process!

shotgun2 short-report:

cat /etc/fuel_build_id:
 495
cat /etc/fuel_build_number:
 495
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 mitaka-9.0
rpm -qa | egrep 'fuel|astute|network-checker|nailgun|packetary|shotgun':
 fuel-release-9.0.0-1.mos6359.noarch
 fuel-provisioning-scripts-9.0.0-1.mos8914.noarch
 fuel-bootstrap-cli-9.0.0-1.mos291.noarch
 fuel-setup-9.0.0-1.mos6359.noarch
 fuel-nailgun-9.0.0-1.mos8914.noarch
 network-checker-9.0.0-1.mos77.x86_64
 fuel-library9.0-9.0.0-1.mos8653.noarch
 fuel-9.0.0-1.mos6359.noarch
 shotgun-9.0.0-1.mos90.noarch
 python-fuelclient-9.0.0-1.mos363.noarch
 fuel-mirror-9.0.0-1.mos158.noarch
 fuel-openstack-metadata-9.0.0-1.mos8914.noarch
 fuel-utils-9.0.0-1.mos8653.noarch
 fuelmenu-9.0.0-1.mos276.noarch
 fuel-ui-9.0.0-1.mos2843.noarch
 rubygem-astute-9.0.0-1.mos782.noarch
 fuel-misc-9.0.0-1.mos8653.noarch
 fuel-ostf-9.0.0-1.mos947.noarch
 fuel-notify-9.0.0-1.mos8653.noarch
 fuel-agent-9.0.0-1.mos291.noarch
 python-packetary-9.0.0-1.mos158.noarch
 nailgun-mcagents-9.0.0-1.mos782.noarch
 fuel-migrate-9.0.0-1.mos8653.noarch

[Valentyn Yakovlev (vyakovlev)]

https://custom-ci.infra.mirantis.net/view/9.x/job/9.x.custom.ubuntu.system_test/680/console

[Valentyn Yakovlev (vyakovlev)]

Another test, with the same fail:

        Scenario:
            1. Create new environment
            2. Choose Neutron, VLAN
            3. Choose cinder for volumes and ceph for images, ceph for
               ephemeral and Rados GW for objects
            4. Add 3 controller
            5. Add 2 compute
            6. Add 3 ceph nodes
            7. Add 1 cinder node
            8. Change default public net mask from /24 to /25
            9. Change default partitioning for ceph and cinder nodes for vdc
            10. Change default dns server to any 2 public dns servers to the
                'Host OS DNS Servers' on Settings tab
            11. Change default ntp servers to any 2 public ntp servers to the
                'Host OS NTP Servers' on Settings tab
            12. Verify networks
            13. Deploy cluster << FAIL
            14. Verify networks
            15. Checking size of partitions
            16. Run OSTF

https://custom-ci.infra.mirantis.net/view/9.x/job/9.x.custom.ubuntu.system_test/679/console

[Oleksiy Molchanov (omolchanov)]

I will pass this to Mikhail Zhnichkov, he is working on neutron decomposition.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/408618

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/408619

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/408618
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=1828eade796d74419b65a3845033a29eafe3fa19
Submitter: Jenkins
Branch: master

commit 1828eade796d74419b65a3845033a29eafe3fa19
Author: Mikhail <email address hidden>
Date: Wed Dec 7 19:24:59 2016 +0400

    Add primary task for network-server-config

    Sometimes task network-server-config on non-primary-controller
    executes before primary-controller. We should specify the start order

    Change-Id: I0288d84b64c061a079c471fe47b2bc312a3f5659
    Closes-Bug: #1646799
    Closes-Bug: #1648028

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/408619
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=2f89958be73733a63f0833b8ccfdad36f8742cf7
Submitter: Jenkins
Branch: stable/newton

commit 2f89958be73733a63f0833b8ccfdad36f8742cf7
Author: Mikhail <email address hidden>
Date: Wed Dec 7 19:24:59 2016 +0400

    Add primary task for network-server-config

    Sometimes task network-server-config on non-primary-controller
    executes before primary-controller. We should specify the start order

    Change-Id: I0288d84b64c061a079c471fe47b2bc312a3f5659
    Closes-Bug: #1646799
    Closes-Bug: #1648028

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/408180
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=6eb0ef859edd651e638e0394f77aa083ecb6e3c8
Submitter: Jenkins
Branch: stable/mitaka

commit 6eb0ef859edd651e638e0394f77aa083ecb6e3c8
Author: Mikhail <email address hidden>
Date: Wed Dec 7 19:24:59 2016 +0400

    Add primary task for network-server-config

    Sometimes task network-server-config on non-primary-controller
    executes before primary-controller. We should specify the start order

    Change-Id: I0288d84b64c061a079c471fe47b2bc312a3f5659
    Closes-Bug: #1646799
    Closes-Bug: #1648028

[Valentyn Yakovlev (vyakovlev)]

Did not help

[root@nailgun ~]# shotgun2 short-report
cat /etc/fuel_build_id:
 495
cat /etc/fuel_build_number:
 495
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 mitaka-9.0
rpm -qa | egrep 'fuel|astute|network-checker|nailgun|packetary|shotgun':
 fuel-nailgun-9.0.0-1.mos8915.noarch
 network-checker-9.0.0-1.mos77.x86_64
 fuel-provisioning-scripts-9.0.0-1.mos8915.noarch
 fuel-ostf-9.0.0-1.mos947.noarch
 fuel-migrate-9.0.0-1.mos8666.noarch
 fuel-agent-9.0.0-1.mos291.noarch
 python-packetary-9.0.0-1.mos158.noarch
 nailgun-mcagents-9.0.0-1.mos783.noarch
 fuel-setup-9.0.0-1.mos6359.noarch
 shotgun-9.0.0-1.mos90.noarch
 fuel-utils-9.0.0-1.mos8666.noarch
 fuelmenu-9.0.0-1.mos276.noarch
 fuel-mirror-9.0.0-1.mos158.noarch
 rubygem-astute-9.0.0-1.mos783.noarch
 fuel-release-9.0.0-1.mos6359.noarch
 fuel-misc-9.0.0-1.mos8666.noarch
 fuel-notify-9.0.0-1.mos8666.noarch
 fuel-library9.0-9.0.0-1.mos8666.noarch
 fuel-bootstrap-cli-9.0.0-1.mos291.noarch
 fuel-openstack-metadata-9.0.0-1.mos8915.noarch
 python-fuelclient-9.0.0-1.mos363.noarch
 fuel-9.0.0-1.mos6359.noarch
 fuel-ui-9.0.0-1.mos2848.noarch

[Valentyn Yakovlev (vyakovlev)]

bug is not observed

https://custom-ci.infra.mirantis.net/view/9.x/job/9.x.custom.ubuntu.system_test/882/console

https://product-ci.infra.mirantis.net/view/baremetal/job/9.x.baremetal.ubuntu.four_controllers/31/console

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/fuel-library 11.0.0.0rc1 release candidate.