Cannot delete 'controller+ceph' node and strange work of notifications for this error

[Anastasia Palkina]

I reproduced this bug twice

1. Create new environment (Ubuntu)
2. Choose tunneling segmentation
3. Choose Ceph for volumes
4. Add 3 controller+ceph, 2 ceph
5. Start deployment. It was successful
6. Start OSTF tests. It was successful
7. Add 2 computes
8. Start deployment. It was successful
9. Start OSTF tests. It was successful
10. Check one of 'controller+ceph' node and click "Delete" button
11. Add new controller node
12. Click "Deploy Changes" button
13. Deployment has failed immediately with Error:
Ceph data still exists on: node-10. You must manually remove the OSDs from the cluster and allow Ceph to rebalance before deleting these nodes.

After it many notifications appeared (see first screen).
A few minutes later error message change to (see second screen):

Error
Deployment has failed. Check these nodes:
Unknown error
Ceph data still exists on: node-10. You must manually remove the OSDs from the cluster and allow Ceph to rebalance before deleting these nodes.

Notifications continue to appear.

Logs are here: https://drive.google.com/a/mirantis.com/file/d/0B6SjzarTGFxaNE5SUkVUYTVtWE0/view?usp=sharing

{"fuel-createmirror_sha": "df6a93f7e2819d3dfa600052b0f901d9594eb0db", "api": "1.0", "fuelmain_sha": "3303f41f99cf9167da01d503dd5d2c8dab141447", "astute_sha": "a090546d43c770ac27ca81c6f8c78ff0ba4a93e0", "fuel-ostf_sha": "983d0e6fe64397d6ff3bd72311c26c44b02de3e8", "openstack_version": "2015.1.0-8.0", "build_id": "141", "build_number": "141", "auth_required": true, "fuel-library_sha": "cd1b4b67d2b00fb10264d6626327688b170f0bf8", "fuel-nailgun-agent_sha": "08e0a11cf1f29b705e4b910d9b9db5e9b708b6e3", "fuel-nailgun_sha": "1479c0b03ad928f2ea2a819fbf8218cff32e51b9", "fuel-agent_sha": "cf699820fb0a4d20bef001861e006dc9797b5733", "production": "docker", "python-fuelclient_sha": "769df968e19d95a4ab4f12b1d2c76d385cf3168c", "feature_groups": ["mirantis"], "release": "8.0", "release_versions": {"2015.1.0-8.0": {"VERSION": {"fuel-createmirror_sha": "df6a93f7e2819d3dfa600052b0f901d9594eb0db", "api": "1.0", "fuelmain_sha": "3303f41f99cf9167da01d503dd5d2c8dab141447", "astute_sha": "a090546d43c770ac27ca81c6f8c78ff0ba4a93e0", "fuel-ostf_sha": "983d0e6fe64397d6ff3bd72311c26c44b02de3e8", "openstack_version": "2015.1.0-8.0", "build_id": "141", "build_number": "141", "fuel-library_sha": "cd1b4b67d2b00fb10264d6626327688b170f0bf8", "fuel-nailgun-agent_sha": "08e0a11cf1f29b705e4b910d9b9db5e9b708b6e3", "fuel-nailgun_sha": "1479c0b03ad928f2ea2a819fbf8218cff32e51b9", "fuel-agent_sha": "cf699820fb0a4d20bef001861e006dc9797b5733", "production": "docker", "python-fuelclient_sha": "769df968e19d95a4ab4f12b1d2c76d385cf3168c", "feature_groups": ["mirantis"], "release": "8.0"}}}

[Dmitry Pyzhov (dpyzhov)]

This is intended behaviour with bad UX.

[Dmitry Pyzhov (dpyzhov)]

It was introduced here: https://review.openstack.org/#/c/161045/

[Mike Scherbakov (mihgen)]

I believe that we could decrease importance to Medium once we fix Notifications (provide better explanation, list nodes, and remove duplicated messages), and may be change wording on "Deployment Failure" banner to be more explicit.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/249848

[Maciej Kwiek (maciej-iai)]

I cannot reproduce it, on current master, asked Anastasia Palkina to verify that it's still a case.

[Anastasia Palkina (apalkina)]

I reproduced this issue on ISO #247 on VBox

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  openstack_version: "2015.1.0-8.0"
  api: "1.0"
  build_number: "247"
  build_id: "247"
  fuel-nailgun_sha: "86cebc1d92c7cc9ca25b00f5590954a7c4f880a0"
  python-fuelclient_sha: "91474bd8c526f4f536ab13368feb4a5c1b84d185"
  fuel-agent_sha: "660c6514caa8f5fcd482f1cc4008a6028243e009"
  fuel-nailgun-agent_sha: "a33a58d378c117c0f509b0e7badc6f0910364154"
  astute_sha: "b60624ee2c5f1d6d805619b6c27965a973508da1"
  fuel-library_sha: "032c707ec800f11044b32733dd4d395e06c209d0"
  fuel-ostf_sha: "65de07b5dce50349e7bc414f364505483c34e2b1"
  fuel-mirror_sha: "bfe7af26b7e6fdd46a16480481cc757f67958177"
  fuelmenu_sha: "fcb15df4fd1a790b17dd78cf675c11c279040941"
  shotgun_sha: "a0bd06508067935f2ae9be2523ed0d1717b995ce"
  network-checker_sha: "a3534f8885246afb15609c54f91d3b23d599a5b1"
  fuel-upgrade_sha: "1e894e26d4e1423a9b0d66abd6a79505f4175ff6"
  fuelmain_sha: "fda7c87dea9fb54c08bd3844d277b2e4778924e4"

[Anastasia Palkina (apalkina)]

Logs are here: https://drive.google.com/a/mirantis.com/file/d/0B6SjzarTGFxabHFqTm5MeTdhLWM/view?usp=sharing

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/249848
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=588b040d35a14d57fe37c796b1fe954e6c102341
Submitter: Jenkins
Branch: master

commit 588b040d35a14d57fe37c796b1fe954e6c102341
Author: Maciej Kwiek <email address hidden>
Date: Fri Nov 27 12:30:37 2015 +0100

    Change error notification for unknown nodes fail

    Before, notification was "Deployment has failed. Check these nodes:
    Unknown error".

    Now it is more comprehensible. Information about failed nodes is coming
    in one of next tasks, so we let the user know that.

    Change-Id: I0f397b37747ba654823a819cc957ff6fad450ba1
    Partial-Bug: #1506008

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/255327

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/255327
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=e56d02062062f6b6735e64bf1c1f9781c6702df9
Submitter: Jenkins
Branch: master

commit e56d02062062f6b6735e64bf1c1f9781c6702df9
Author: Maciej Kwiek <email address hidden>
Date: Wed Dec 9 16:43:30 2015 +0100

    Get rid of irrelevant notification

    Some notifications didn't give end user any information, especially when
    there were no nodes in error state. This patch makes them not appear
    anymore.

    Change-Id: Ic5f42972287f475c1a3158a57a8a0ab9e870e18e
    Closes-bug: #1506008

[Dmitriy Kruglov (dkruglov)]

Verified on MOS 8.0, build 328. The issue is fixed.

ISO details:
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "328"
  build_id: "328"
  fuel-nailgun_sha: "ec25ae8df28e1f1d87da653c5aab2711dff729f6"
  python-fuelclient_sha: "7c7a756fb6a3d091851c060003a2965c67aa353d"
  fuel-agent_sha: "d354cbe61b055db848a15ce66fb7ae92178d2c0a"
  fuel-nailgun-agent_sha: "a33a58d378c117c0f509b0e7badc6f0910364154"
  astute_sha: "c56dfde2da034151a7e707b381c4cf9d213b4ba2"
  fuel-library_sha: "bcc3d042a47547f6ad826360a85ef21dcaf25882"
  fuel-ostf_sha: "b2ebf15a3530b5c7b57707acf8642c1c3ac71bd8"
  fuel-mirror_sha: "8100acb3a566358d0d4ecc66de32d39626263028"
  fuelmenu_sha: "2942a85796d37f09ba8c8c6d762d8813292cf0d4"
  shotgun_sha: "cacb93cbc28910ff0dc38f30a855efa9af50d8ce"
  network-checker_sha: "d443ef47abeda58d319bc8d33d5005dd09440a02"
  fuel-upgrade_sha: "718aa3d7021fee2970f0fa6791cf5188578cc516"
  fuelmain_sha: "3faa824728ce60734abe602ff3778976f8a16eed"