Astute repors "ready" even if DHCP checks are failed

[Ihor Kalnytskyi]

(the issue was found within bug #1569325)

We should report any sort of errors to Nailgun, they must not be hidden.

Steps to reproduce:

0. Deploy master node with few slaves.
1. Create any cluster.
2. Add few nodes to cluster.
3. Run network checker and make DHCP checks to fail (by hacking dhcpcheck to return error)

Expected results:

Astute reports to Nailgun, that "dhcp_check" task is failed.

Actual results:

Astute reports to Nailgun, that "dhcp_check" task is ready. However, we can see in Astute.log something like:

    "Spawning listener for enp0s6 failed.\nenp0s6: That device is not up\n2016-04-12 11:55:06 ERROR (api) enp0s6: That device is not up\nSpawning listener for enp0s7 failed.\nenp0s7: That device is not up\n\n",

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/316861

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/317319

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/316861
Committed: https://git.openstack.org/cgit/openstack/fuel-astute/commit/?id=1c29ced74b1556732f717e7882f043e1647a4254
Submitter: Jenkins
Branch: master

commit 1c29ced74b1556732f717e7882f043e1647a4254
Author: Artem Roma <email address hidden>
Date: Mon May 16 16:56:56 2016 +0300

    Return 'error' status for check_dhcp task if failed nodes

    * Fix processing of net_probe client response in 'format_dhcp_response'
    hepler function;
    * Make check_dhcp routine return 'error' as task status in case DHCP
      check failed for any node from given set;
    * Update tests for check_dhcp.

    Change-Id: I2e89f4a95289328128d33a8d079b09f6fbfa8d5e
    Closes-Bug: #1569339

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/319924

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/322201

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/319924
Committed: https://git.openstack.org/cgit/openstack/fuel-astute/commit/?id=a3872ac4f438d8691907175f3c4d856f39e3711c
Submitter: Jenkins
Branch: stable/mitaka

commit a3872ac4f438d8691907175f3c4d856f39e3711c
Author: Artem Roma <email address hidden>
Date: Mon May 16 16:56:56 2016 +0300

    Return 'error' status for check_dhcp task if failed nodes

    * Fix processing of net_probe client response in 'format_dhcp_response'
    hepler function;
    * Make check_dhcp routine return 'error' as task status in case DHCP
      check failed for any node from given set;
    * Update tests for check_dhcp.

    Change-Id: I2e89f4a95289328128d33a8d079b09f6fbfa8d5e
    Closes-Bug: #1569339
    (cherry picked from commit 1c29ced74b1556732f717e7882f043e1647a4254)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/322201
Committed: https://git.openstack.org/cgit/openstack/fuel-astute/commit/?id=230c096c45d1133f9f651bbfadd9c050a9e3dc42
Submitter: Jenkins
Branch: master

commit 230c096c45d1133f9f651bbfadd9c050a9e3dc42
Author: Artem Roma <email address hidden>
Date: Fri May 27 17:27:44 2016 +0300

    Save error status if err data is present in dhcp checker response

    Change-Id: I7cb0d9b575edf08b5051ea01cee97c3d574b8439
    Related-Bug: #1569339

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/325249

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/326043

[Grigory Mikhailov (gmikhailov)]

Verified on #fuel-9.0-mos-492 iso.
Network verification is successfully failed after new Exception adddition.

[root@nailgun ~]# shotgun2 short-report
cat /etc/fuel_build_id: 492
cat /etc/fuel_build_number: 492
cat /etc/fuel_release: 9.0
cat /etc/fuel_openstack_version: mitaka-9.0
rpm -qa | egrep 'fuel|astute|network-checker|nailgun|packetary|shotgun':
 fuel-release-9.0.0-1.mos6349.noarch
 fuel-misc-9.0.0-1.mos8460.noarch
 python-packetary-9.0.0-1.mos140.noarch
 fuel-bootstrap-cli-9.0.0-1.mos285.noarch
 fuel-migrate-9.0.0-1.mos8460.noarch
 rubygem-astute-9.0.0-1.mos750.noarch
 fuel-mirror-9.0.0-1.mos140.noarch
 shotgun-9.0.0-1.mos90.noarch
 fuel-openstack-metadata-9.0.0-1.mos8743.noarch
 fuel-notify-9.0.0-1.mos8460.noarch
 nailgun-mcagents-9.0.0-1.mos750.noarch
 python-fuelclient-9.0.0-1.mos325.noarch
 fuel-9.0.0-1.mos6349.noarch
 fuel-utils-9.0.0-1.mos8460.noarch
 fuel-setup-9.0.0-1.mos6349.noarch
 fuel-provisioning-scripts-9.0.0-1.mos8743.noarch
 fuel-library9.0-9.0.0-1.mos8460.noarch
 network-checker-9.0.0-1.mos74.x86_64
 fuel-agent-9.0.0-1.mos285.noarch
 fuel-ui-9.0.0-1.mos2717.noarch
 fuel-ostf-9.0.0-1.mos936.noarch
 fuelmenu-9.0.0-1.mos274.noarch
 fuel-nailgun-9.0.0-1.mos8743.noarch

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/317319
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=5e5e7284cfc497afe41cd4b96c504017b8556351
Submitter: Jenkins
Branch: master

commit 5e5e7284cfc497afe41cd4b96c504017b8556351
Author: Artem Roma <email address hidden>
Date: Mon May 16 14:25:40 2016 +0300

    Consider status of check_dhcp for each node

    DHCP check task may fail on particular node from given set. In this case
    data for that node will contain information about status of the
    operation and particular causes of the error (if supplied by dhcp
    checker itself). In such situation the task must be set to error status
    despite results from other nodes.

    Change-Id: I7dddbf727c31264596e5de6e169b09b2c0cfe20e
    Closes-Bug: #1569339

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/326043
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=51690232dec6a6653235ecfb5b2352b1c09abd09
Submitter: Jenkins
Branch: master

commit 51690232dec6a6653235ecfb5b2352b1c09abd09
Author: Artem Roma <email address hidden>
Date: Mon Jun 6 19:55:09 2016 +0300

    Handle exception when comparing of MACs fails in check_dhcp_resp

    In case MAC's value is empty string comparing utils.is_same_mac fails
    with ValueError exception which may leads to responses from other nodes not
    being processed. We need to properly handle such situation.

    Change-Id: I8843f9eabd139222ac326fdc26a43f5702ba751b
    Related-Bug: #1569339
    Related-Bug: #1569325

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/366496

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/366496
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=615d924b9cf91ee0e49000097d4654ad7686bb71
Submitter: Jenkins
Branch: stable/mitaka

commit 615d924b9cf91ee0e49000097d4654ad7686bb71
Author: Artem Roma <email address hidden>
Date: Mon Jun 6 19:55:09 2016 +0300

    Handle exception when comparing of MACs fails in check_dhcp_resp

    In case MAC's value is empty string comparing utils.is_same_mac fails
    with ValueError exception which may leads to responses from other nodes not
    being processed. We need to properly handle such situation.

    Change-Id: I8843f9eabd139222ac326fdc26a43f5702ba751b
    Related-Bug: #1569339
    Related-Bug: #1569325
    (cherry picked from commit 51690232dec6a6653235ecfb5b2352b1c09abd09)