Node deploy should fail when network on node doesn't work

[Andrew Woodward]

After puppet completes network stage, no testing is performed to validate node's network. Network should be tested and provision should fail if not successful.

I think we could program the network stage to require successful ping of first controller, while this wont work to validate the first controller, it will help all subsequent nodes, and if all nodes fail then that would indicate that there is a problem on the first controller.

[Andrew Woodward (xarses)]

This is important as it would prevent issues like https://bugs.launchpad.net/fuel/+bug/1261711

[Vladimir Kuklin (vkuklin)]

This is going to be a part of our pluggable architecture allowing pre- and post-stage checks to implemented.

[Matthew Mosesohn (raytrac3r)]

This is a feature change because it changes behavior of puppet run dramatically. Still high priority, but not for 6.0.

[Bogdan Dobrelya (bogdando)]

Do we have a plans to address this issue as a blueprint or fix it as a bug in 6.1?

[Bogdan Dobrelya (bogdando)]

Superseded by https://blueprints.launchpad.net/fuel/+spec/granular-deployment-based-on-tasks

[Andrew Woodward (xarses)]

from https://bugs.launchpad.net/fuel/+bug/1451908

Now in 6.1, It's frequently found to have problems because of vrouter or package fetching. It's now even more important to verify network function after configuring it to ensure that we can meet our needs. If it doesn't work we need to fail in an obvious way that makes it easy to identify and troubleshoot further.

To solve this we should build a task that runs after l23network to verify that we can access every repo listed, this will verify 1) DNS, 2) network, 3) repo's should any fail the deployment should fail in an obvious way.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/180345

[Alex Schultz (alex-schultz)]

Related fix proposed to branch: master
Review: https://review.openstack.org/180538

[OpenStack Infra (hudson-openstack)]

Change abandoned by Andrew Woodward (<email address hidden>) on branch: master
Review: https://review.openstack.org/180345
Reason: Drop in favor of https://review.openstack.org/#/c/180538

[Bogdan Dobrelya (bogdando)]

This bug looks related or duplicate/superseded by https://bugs.launchpad.net/fuel/+bug/1439686

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/180538
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=57752a27fe045b8edc46d04173864fa902c5d814
Submitter: Jenkins
Branch: master

commit 57752a27fe045b8edc46d04173864fa902c5d814
Author: Alex Schultz <email address hidden>
Date: Tue May 5 20:55:20 2015 -0500

    Connectivity tests for external services

    In order to provide better error handling around configured external
    services, we are adding additional checks to some deployment tasks
    where we know that there are either external services or services
    that can be user configurable.

    *) Adding a repository connectivity test task to be run after
    netconfig to ensure that software repository access is OK before
    proceeding with the rest of the deployment.
    *) Adding in post connectivity tests before NTP server to ensure that
    those services will be able to reach the configured settings.

    With this change, we are adding two custom puppet functions
    (url_available and ntp_available) to the osnailyfacter module. These
    functions will throw a Puppet::Error if unable to properly
    communicate with the services.

    Change-Id: I6b0302ce403871384d377aceb7e94b09126b885e
    Closes-Bug: 1261940

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/183036

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/183036
Committed: https://git.openstack.org/cgit/stackforge/fuel-qa/commit/?id=ee8f493555c5557bde66354f16db514900dc331d
Submitter: Jenkins
Branch: master

commit ee8f493555c5557bde66354f16db514900dc331d
Author: Tatyana Leontovich <email address hidden>
Date: Wed May 13 18:44:36 2015 +0300

    Add task cluster-vrouter

    Add next tasks:
    * cluster-vrouter
    * connectivity_tests
    * swift-rebalance-cron

    Related-Bug: #1261940
    Related-Bug: #1436868
    Change-Id: Ic009a0cba9ab1d68e5b2e5d29d5badaca57f4714

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/183757

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/184334

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/184334
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=e2845376be1ec0f34747d8f0d725b1000b460132
Submitter: Jenkins
Branch: master

commit e2845376be1ec0f34747d8f0d725b1000b460132
Author: Alex Schultz <email address hidden>
Date: Tue May 19 16:23:46 2015 -0500

    Increase repo url availability timeout values

    In order to improve the experiance when deploying nodes on slow or
    congested network links, we need to increase the values of the
    timeouts for our url availability check. We are increasing them from
    5 seconds for the open and read timeouts to 60 seconds. Additionally
    we are increasing the overall timeout on the check to 180 seconds. We
    are also increasing the task timeout from 600 seconds to 3600 seconds.

    Change-Id: Ibd4e87fbf3a27d1fca1bcad39ee9f534c9a92f60
    Closes-Bug: 1456805
    Related-Bug: 1261940

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/183757
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=f1eaa1f99cfef53bae1f4d766c9ee4ecd6fbc7ee
Submitter: Jenkins
Branch: master

commit f1eaa1f99cfef53bae1f4d766c9ee4ecd6fbc7ee
Author: Alex Schultz <email address hidden>
Date: Fri May 15 20:54:38 2015 -0500

    Mock url_available and ntp_available functions

    With the addition of the url_available and ntp_available puppet
    functions as part of the connectivity tests, we need to mock them out
    for our noop tests so we don't actually fail when an test url is used.

    Change-Id: I578e266e931f12d2e620a80994257871181e9f9b
    Partial-Bug: 1261940
    Closes-Bug: 1455702

[Oleksiy Molchanov (omolchanov)]

Verified

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.1"
  openstack_version: "2014.2.2-6.1"
  api: "1.0"
  build_number: "496"
  build_id: "2015-06-02_06-16-08"
  nailgun_sha: "3830bdcb28ec050eed399fe782cc3dd5fbf31bde"
  python-fuelclient_sha: "4fc55db0265bbf39c369df398b9dc7d6469ba13b"
  astute_sha: "5d570ae5e03909182db8e284fbe6e4468c0a4e3e"
  fuel-library_sha: "5a2146c80186cd74021cf1a55449286c3c59a60b"
  fuel-ostf_sha: "7413186490e8d651b8837b9eee75efa53f5e230b"
  fuelmain_sha: "bcc909ffc5dd5156ba54cae348b6a07c1b607b24"