[nailgun] [octane] Nailgun returns an error 500 during node-upgrade via octane

Bug #1584044 reported by Andrey Volochay
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
High
Nikita Zubkov
7.0.x
Won't Fix
High
Registry Administrators
8.0.x
Fix Released
High
Anton Chevychalov
Mitaka
Fix Committed
High
Yuriy Taraday

Bug Description

Detailed bug description:
 During upgrade MOS6.0 to MOS8.0 via Octane(stable/8.0), I have faced with following bug.

 By doing second controller upgrade via Octane (octane upgrade-node <target_node_id> <seed_env_id>), Nailgun returns an error with code 500

2016-05-18 13:01:00.188 ERROR [7f39799c4880] (handlers) MOVE NODE FAILED
Traceback (most recent call last):
 File "/usr/lib/python2.7/site-packages/nailgun/extensions/cluster_upgrade/handlers.py", line 93, in POST
   upgrade.UpgradeHelper.assign_node_to_cluster(node, cluster)
 File "/usr/lib/python2.7/site-packages/nailgun/extensions/cluster_upgrade/upgrade.py", line 191, in assign_node_to_cluster
   node, netgroups_id_mapping)
 File "/usr/lib/python2.7/site-packages/nailgun/extensions/cluster_upgrade/objects/adapters.py", line 140, in set_bond_assignment_netgroups_ids
   node.node, mapping)
 File "/usr/lib/python2.7/site-packages/nailgun/network/manager.py", line 203, in set_bond_assignment_netgroups_ids
   netgroups_id_mapping[bond_assignment.network_id]
KeyError: 7

Steps to reproduce:
 1) Have MOS6.0 environment
 2) Upgrade master node (fuel 6.0) to 6.1 and then to 7.0
 3) Backup environment settings, following documented process
 4) Prepare new master node (fuel 8.0)
 5) Restore the backup from third step
 5) Upgrade restored environment via Octane
 6) Upgrade primary controller via Octane
 7) Upgrade DB via Octane
 8) Upgrade Control Plane via Octane
 9) Upgrade second and third controllers

Expected results:
 Provisioning of second and third controllers is started

Actual result:
 Nailgun returns an error 500 which in turn breaks node upgrade.

Reproducibility:
 Use network templates ans network bonding.

Workaround:
 Comment out following code in nailgin/cluster_upgrade/upgrade.py, when first controller is moved and then restart Nailgun:

   orig_manager.set_bond_assignment_netgroups_ids(
      node, netgroups_id_mapping)

Description of the environment:
 Versions of components: MOS8.0, Octane(stable/8.0)
 Network model: Environment uses network bonding
 Related projects installed: Octane

Additional information:
 It seems like Nailgun falls at "set_bond_assignment_netgroups_ids" method because the method performs an action, which has been done before.

 Presumably it happens at assign_node_to_cluster method.
1) assign_node_to_cluster
2) node.update_cluster_assignment(seed_cluster)
3) update_cluster_assignment
4) set_network_template
5) apply_network_template #there is a condition IF, in case when network_template is not absent
6) assign_networks_by_template #there are mentioned actions

Dmitry Klenov (dklenov)
tags: added: area-python
tags: added: non-release
no longer affects: fuel/newton
Revision history for this message
Ilya Kharin (akscram) wrote :

Could you please provide a sample of the network template that was used in this case?

Revision history for this message
Andrey Volochay (wallavv) wrote :

Here you go

description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-web (master)

Fix proposed to branch: master
Review: https://review.openstack.org/339387

Changed in fuel:
assignee: Fuel Octane (fuel-octane-team) → Anastasiya (atolochkova)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-web (master)

Reviewed: https://review.openstack.org/339387
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=bbb68b239e64770e2718264780a25e4db3248f34
Submitter: Jenkins
Branch: master

commit bbb68b239e64770e2718264780a25e4db3248f34
Author: Anastasiya <email address hidden>
Date: Fri Jul 8 10:34:07 2016 +0300

    Assign node to cluster with network template

    Don't need to assign nics and bonds to netgroups
    if network template has been applied to new cluster.

    Change-Id: Ibd06de87964bf7a2038899d32e3d8a0189b9cbbd
    Partial-bug: 1584044

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-nailgun-extension-cluster-upgrade (master)

Fix proposed to branch: master
Review: https://review.openstack.org/348423

Changed in fuel:
assignee: Anastasiya (atolochkova) → Nikita Zubkov (zubchick)
Nikita Zubkov (zubchick)
Changed in fuel:
assignee: Nikita Zubkov (zubchick) → Anastasiya (atolochkova)
Changed in fuel:
assignee: Anastasiya (atolochkova) → Nikita Zubkov (zubchick)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-nailgun-extension-cluster-upgrade (master)

Reviewed: https://review.openstack.org/348423
Committed: https://git.openstack.org/cgit/openstack/fuel-nailgun-extension-cluster-upgrade/commit/?id=56dc7c1f8fc95eea2cbc5410cbcf75ed99f6909d
Submitter: Jenkins
Branch: master

commit 56dc7c1f8fc95eea2cbc5410cbcf75ed99f6909d
Author: Anastasiya <email address hidden>
Date: Fri Jul 8 10:34:07 2016 +0300

    Assign node to cluster with network template

    Don't need to assign nics and bonds to netgroups
    if network template has been applied to new cluster.

    Change-Id: Ibd06de87964bf7a2038899d32e3d8a0189b9cbbd
    Partial-bug: 1584044

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-web (stable/8.0)

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/353876

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-web (stable/8.0)

Reviewed: https://review.openstack.org/353876
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=6a54dda6a3cc8d968e17438a85151780f15f92b6
Submitter: Jenkins
Branch: stable/8.0

commit 6a54dda6a3cc8d968e17438a85151780f15f92b6
Author: Anastasiya <email address hidden>
Date: Fri Jul 8 10:34:07 2016 +0300

    Assign node to cluster with network template

    Don't need to assign nics and bonds to netgroups
    if network template has been applied to new cluster.

    Change-Id: Ibd06de87964bf7a2038899d32e3d8a0189b9cbbd
    Partial-bug: 1584044
    (cherry picked from commit bbb68b239e64770e2718264780a25e4db3248f34)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-nailgun-extension-cluster-upgrade (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/360086

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-nailgun-extension-cluster-upgrade (stable/mitaka)
Download full text (21.0 KiB)

Reviewed: https://review.openstack.org/360086
Committed: https://git.openstack.org/cgit/openstack/fuel-nailgun-extension-cluster-upgrade/commit/?id=40dd411fe461441e943de806f5de632eeb64dc4a
Submitter: Jenkins
Branch: stable/mitaka

commit 17ab5a3aaa7d2c4ae4652d1a4dc0410b31e7c47d
Author: Sergey Abramov <email address hidden>
Date: Wed Aug 24 18:32:39 2016 +0300

    Add create upgrade release handler

    Required for create new release just for upgrade, that have overwrited
    params. This params are valid for orig cluster release.

    Change-Id: Ib2387b9c2b74902c7289ee8f69a5f5d323ec82ca

commit 8de47e0dbf5955b3cbe59a3956e1ca1f4d8679ef
Author: Nikita Zubkov <email address hidden>
Date: Wed Aug 24 15:42:32 2016 +0300

    Add VIPs transformer

    Move upgrade vips code to the transformer

    Change-Id: Ia6935521b8b90b73fb74ba92859e7febf74c4ced

commit b54f9d4c29808c4561b20af508b85b9e40fa72b6
Author: Nikita Zubkov <email address hidden>
Date: Wed Aug 24 17:51:39 2016 +0300

    Move partition info transformation to extension

    octane have some dirty hacks to change volumes attributes using nailgun
    as a library, such modifications should be done in a extension

    Change-Id: I422bb368916f3a319e286edcc6103a2834097a87

commit b3ce0d348cc20ed988b5adf66474836f631eb12d
Author: Anastasiya <email address hidden>
Date: Mon Aug 1 15:47:13 2016 +0300

    Add tests for assign_node_to_cluster

    Change-Id: Iafa1baa6a1ca4d701ec89e49dd9d6f969804c82e

commit a764ba306dcd0cfb65a82c89f07ec0f3d129c66b
Author: Nikita Zubkov <email address hidden>
Date: Tue Aug 23 14:05:16 2016 +0300

    Update README

    Add instalation section

    Change-Id: Ib16132ab9c18d757e96e98304fdd8339ddf5497a

commit 95ff3a3598397bb6e3c68793051298aaa006d163
Author: Yuriy Taraday <email address hidden>
Date: Tue Aug 23 23:01:16 2016 +0300

    Add cluster transformations

    Implement transformations that are applied to cluster attributes during
    environment cloning.

    Conversion from text to text_list type has been limited to dns_list and
    ntp_list keys only to keep predictable behavior.

    Change-Id: I1ff596f850bd42243697cad1c1c35f0cf1386376

commit 163ce243fbade3dac05eb535ad2987687a57f87d
Author: Yuriy Taraday <email address hidden>
Date: Tue Aug 23 22:51:19 2016 +0300

    Add pluggable transformations for data migration

    This change introduces new transformation mechanism:
    - all available transformations are listed in setuptools entry points
      under namespace like this (for cluster transformations):

        nailgun.cluster_upgrade.transformations.cluster.9.0 =
            dns_list = ...
            ntp_list = ...
        nailgun.cluster_upgrade.transformations.cluster.8.0 =
            ...
        <etc>

    - config file will include section that specifies enabled
      transformations like this:

        CLUSTER_UPGRADE_TRANSFORMATIONS:
          cluster:
            9.0: dns_list ntp_list ...
            8.0: ...
            7.0: ...

      (only default values are implemented here, actual config support will
      follow)

    - when transformations...

tags: added: in-stable-mitaka
Nikita Zubkov (zubchick)
Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
Vadim Rovachev (vrovachev) wrote :

Verifying of this bug blocked by https://bugs.launchpad.net/fuel/+bug/1616817 . Status was changed for 8.0 release to fix released after discussion with fuel-upgrade-qa team.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.