Fuel does not see the missing SATA controller (all disks) in a slave node

Bug #1372466 reported by okosse on 2014-09-22
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Medium
Fuel Sustaining
6.1.x
Medium
Aleksey Kasatkin
7.0.x
Medium
Fuel Python (Deprecated)
Mitaka
Medium
Fuel Python (Deprecated)
Newton
Medium
Fuel Sustaining

Bug Description

If the node prior to the installation role (loaded bootstrap) will disappear ATA controller or all of the disks, the fuel does not display it.
=====================================================
Configuration:
===================================================
steps to reproduce:
1)set up lab on virtualbox from fuel-master-514-2014-09-18_01-09-23.iso
2)create env.
3)Turn off any slave node
4)Delete SATA controller or all disks
5)Start node

Expected result: After loading bootstrap in the Fuel appears nodе without disks.
Actual result: After loading bootstrap Fuel shows the status of the node has not changed

--------------------------Logs------------------------------------
---------------------fuel-version---------------------------------
api: '1.0'
astute_sha: a3e5da62af91b99f958ab958161d3dcec09c657b
auth_required: true
build_id: 2014-09-18_01-09-23
build_number: '514'
feature_groups:
- mirantis
fuellib_sha: ac4e608e259cb651793561918e4bd955a5d3643a
fuelmain_sha: 9a52633718c1937ef4d0cb244b78e3a14f4d0bd5
nailgun_sha: 72d0294e92b044dee8e45f050df61a4b864ea96a
ostf_sha: dd546af672de75fb23873fa9ed31d29b70215f55
production: docker
release: '6.0'
release_versions:
  2014.1.1-5.1:
    VERSION:
      api: '1.0'
      astute_sha: a3e5da62af91b99f958ab958161d3dcec09c657b
      build_id: 2014-09-18_01-09-23
      build_number: '514'
      feature_groups:
      - mirantis
      fuellib_sha: ac4e608e259cb651793561918e4bd955a5d3643a
      fuelmain_sha: 9a52633718c1937ef4d0cb244b78e3a14f4d0bd5
      nailgun_sha: 72d0294e92b044dee8e45f050df61a4b864ea96a
      ostf_sha: dd546af672de75fb23873fa9ed31d29b70215f55
      production: docker
      release: '6.0'

---------------------------- Fuel Master Web Backend -------------------------

[7f3595d94740] (node) Node Untitled (d9:26) has received an empty disks array - volume information will not be updated

okosse (okosse) wrote :
Changed in fuel:
milestone: none → 6.0
assignee: nobody → Fuel Python Team (fuel-python)
Changed in fuel:
importance: Undecided → Medium
status: New → Confirmed
Ihor Kalnytskyi (ikalnytskyi) wrote :

Well, the issue occurs because we have a hack:

    https://github.com/stackforge/fuel-web/blob/9fe750646b3429a2af6619482e159c37721dbdde/nailgun/nailgun/objects/node.py#L480

The hack is needed since by some reason nailgun agent sometimes sends an empty disks array, even it's not empty at all, and that leads to bad consequences.

Still, as far as I remember, the issue leads to bad consequences in case we have deployed node, not a bootstrap one, so we can add check for bootstrap around this hack.

Changed in fuel:
status: Confirmed → Triaged
Changed in fuel:
milestone: 6.0 → 6.1
Dmitry Pyzhov (dpyzhov) on 2015-03-27
tags: added: feature-hardware-change
removed: nailgun
Changed in fuel:
assignee: Fuel Python Team (fuel-python) → Aleksey Kasatkin (alekseyk-ru)

Fix proposed to branch: master
Review: https://review.openstack.org/170550

Changed in fuel:
status: Triaged → In Progress
Dmitry Pyzhov (dpyzhov) on 2015-04-07
Changed in fuel:
status: In Progress → Confirmed
milestone: 6.1 → 7.0
Changed in fuel:
assignee: Aleksey Kasatkin (alekseyk-ru) → Fuel Python Team (fuel-python)
Aleksey Kasatkin (alekseyk-ru) wrote :

There was the bug with volume info update https://bugs.launchpad.net/fuel/+bug/1278774 . It was solved by adding a hack that block volumes update with empty data. If we allow to update with empty data that bug will arrive again. So, this bug cannot be solved easily, it is moved to 7.0.

Dmitry Pyzhov (dpyzhov) on 2015-04-23
tags: added: module-volumes
tags: added: qa-agree-7.0
Aleksey Kasatkin (alekseyk-ru) wrote :

Proper design is required to solve all mentioned bugs. Feature. Does not break deployment. Moving to 8.0.

tags: added: release-notes
Changed in fuel:
milestone: 7.0 → 8.0
no longer affects: fuel/8.0.x
Dmitry Pyzhov (dpyzhov) on 2015-09-28
tags: added: feature

Oleksandr,

Can you please describe the goal of of use case that you outlined in bug description? I mean, bootstrap a node, shutdown a node, remove storage options from node, bring it back up. Do you need it for some partner integration scenario? If not - what for?

I'm asking because, while the hack that blocks volumes update with empty data can indeed lead to inconsistent pre-deployment state (e.g. Fuel will expose disks to user that are not present in the node anymore, user will partition/allocate them, the whole thing will fail while provisioning Host OS), I need to understand how often this can be exposed in real world. So far my assumption is that in case of such changes user will rather repeat PXE boot => discovery flow for a particular node instead when the HW config was changed (instead of just booting it back up)

okosse (okosse) wrote :

Dmitriy,

I assumed that the Fuel being used as an inventory tool too. In this case when I modify nodes configuration such as storage (this happens regularly in any DC) , I expect that fuel will show it me.

On the other hand it was synthetic test and I don't know any partner integration scenario that would be used it.

AlexeyZ,

I need your help with review of this bug. Judging from description, there's an issue when a node is "discovered" state is shut down, it's storage controller is disabled, the node is brought back. Due to some hack restricting empty updates of node's disk configuration the "discovered" node coming back online will not reflect the change of storage controller (it's removal).

Can you please validate whether the following flow will work in 8.0:

1. User boots a node into PXE boot, it's discovered by Fuel, provisioned with bootstrap image
2. User shuts down the node and removes disks/storage controllers from it
3. User boots the node again - into PXE boot
4. The node is re-discovered by Fuel as the one not having storage options in it

If it will work - I think we can solve this issue by documentation.

Dmitry Pyzhov (dpyzhov) on 2015-10-22
tags: added: area-python
Changed in fuel:
milestone: 8.0 → 9.0

We passed SCF in 8.0. Moving the bug to 9.0.

Not clear so far what we need to relnote or what instructions to provide in the docs. Once there's info, please file a bug with the "area-docs" tag.

tags: removed: release-notes
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers