node['meta']['interfaces'] field is empty after scaling cluster after upgrade

Bug #1562736 reported by Vladimir Khlyunev
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Invalid
High
Unassigned
7.0.x
Fix Released
Critical
Vladimir Sharshov
Newton
Invalid
High
Unassigned

Bug Description

Steps to reproduce:
1. Deploy any cluster using 7.0 Fuel (1 ctrl is preferable for speed up deployment)
2. Upgrade Fuel master as described at https://docs.mirantis.com/openstack/fuel/fuel-8.0/fuel-install-guide.html#upgrade-patch-top-ug (ask bug reporter for actual octane.rpm location)
3. Add 1 compute node to existing cluster (at this moment meta/interfaces contains correct data)
4. Deploy cluster
5. Get node data via api (/api/nodes)

Expected result:
On new node 'meta/interfaces' contains correct data

Actual result:
After deployment 'meta/interfaces' on new node is empty - http://puu.sh/nWHAn/1d3f52e762.png see bottom of the screenshot

Env info:
Fuel 7.0 Release + MU2, Fuel 8.0 Release
Diag.snapshot is not available but the env were snapshot'ed - feel free to request it.
Note: this bug can be reproduced on 9.0 without upgrade but I'm not sure that this is the same issue

Reproducibility:
5/5

Workaround:
Unknown

Impact:
This issue affects UI side of the product - http://puu.sh/nWHK2/5390726026.png

tags: added: team-upgrades
Changed in fuel:
assignee: nobody → Fuel Octane (fuel-octane-team)
Changed in fuel:
importance: Undecided → High
status: New → Confirmed
Revision history for this message
Maksym Strukov (unbelll) wrote :

I have env (Fuel 9.0-130) with node (one of many) which has wrong count (enp0s6 absent) of interfaces in meta->interfaces after cluster deployment process completed. Please ping me if you need env for finding out.

This lead to false negative crashes in fuel-qa system test, because it tries to search devops nodes by set of mac of all ifaces of slave. So bump bug to critical.

Changed in fuel:
importance: High → Critical
Revision history for this message
Maksym Strukov (unbelll) wrote :

Result of GET http://{ip}:8000/api/nodes : http://paste.openstack.org/show/492490/
Issue with meta/interfaces on node-2

Revision history for this message
Vladimir Khlyunev (vkhlyunev) wrote :

Fuel-python, we got something similar on master, please contact @mstrukov

Changed in fuel:
assignee: Fuel Octane (fuel-octane-team) → Fuel Python Team (fuel-python)
Dmitry Pyzhov (dpyzhov)
tags: added: area-python
Revision history for this message
Maksym Strukov (unbelll) wrote :

Ok, please ignore comments #1 and #2 please, because that was another issue and I have another bug for it - https://bugs.launchpad.net/fuel/+bug/1565706

Changed in fuel:
importance: Critical → High
Changed in fuel:
milestone: 9.0 → 10.0
Dmitry Pyzhov (dpyzhov)
Changed in fuel:
assignee: Fuel Python Team (fuel-python) → Fuel Octane (fuel-octane-team)
Changed in fuel:
importance: High → Medium
Revision history for this message
Vladimir Khlyunev (vkhlyunev) wrote :

This issue blocks autotests because we are relying on ['interfaces'] field

tags: added: blocker-for-qa
Revision history for this message
Oleg S. Gelbukh (gelbuhos) wrote :

The root cause of this issue is the lack of support in 7.0.0 version of fuel-nailgun-agent for persistent naming schema.

1. The new node runs bootstrap image, which has fuel-nailgun-agent-8.0.0. It supports the persistent naming schema using this code:

https://github.com/openstack/fuel-nailgun-agent/compare/stable/7.0...stable/8.0#diff-b33aed8f3134996703dc39f9a7c95783R286

2. At this point, fuel-railgun-agent properly reports configuration of the node with full interfaces metadata.

3. When node is being provisioned in an environment of version 7.0, the image installed onto it contains fuel-nailgun-agent-7.0.0, which doesn't support the persistent interfaces naming:

https://github.com/openstack/fuel-nailgun-agent/compare/stable/7.0...stable/8.0#diff-b33aed8f3134996703dc39f9a7c95783L271

4. After the provisioning finishes, the node reboots into an image from local disk and the fuel-railgun-agent-7.0.0 is started. It incorrectly reads configuration of interfaces in the system and updates the node metadata in Nailgun with empty 'interfaces' field.

A workaround to this problem is to apply patch to the fuel-nailgun-agent at the node from the code linked above. With the next update it will set proper metadata.

To solve the issue, we need to back port the current version of code from 'master' of fuel-railgun-agent to stable/7.0 branch.

no longer affects: fuel/mitaka
Revision history for this message
Alexey Shtokolov (ashtokolov) wrote :
tags: added: on-verification
Revision history for this message
Dmitry (dtsapikov) wrote :

Verified on 7.0+mu4

tags: removed: on-verification
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers