reset environment also reset addresses on admin\pxe interaface

Bug #1455610 reported by Pavel Vaylov
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
High
Vladimir Sharshov

Bug Description

=== SUMMARY ===

During reset of a cluster Fuel gives new IP address for nodes in Admin\PXE segment.
It is normal when we have "standard" environment.
But when an environment configured from CLI using ymal files it causes an issue.

IP addresses for Admin\PXE interfaces already stored in database since we uploaded customized yaml files.
After cluster was reset new IP addresses appear on Admin\PXE interfaces and new deployment starts with these new addresses.
Then addresses that were in yaml files will be assigned to the interfaces and deployment will fail due to misconfiguration of IP addresses on Admin\PXE interface.

=== STEPS TO REPRODUCE ===

1. Create an environment of few nodes.
2. Customize deployment information for all nodes in the environment.
3. Deploy the environment.
4. Reset the environment.
4. Deploy it one more time.

=== EXPECTED BEHAVIOR: ===

Environment will be deployed using customized settings.

=== ACTUAL BEHAVIOR ===

Most probably deployment will fail due to wrong configuration of IP addresses on Admin\PXE interfaces.

=== WORKAROUND ===

Each time after reset and before redeploy edit and upload existing yaml files. Actual IP addresses for Admin\PXE interfaces should be in yaml files before upload them.

=== IMPACT ===

Any time when customize environment need to be redeployed manual changing of IP addresses in yaml files needed.

Revision history for this message
Dmitry Pyzhov (dpyzhov) wrote :

It is really risky to fix in 6.1 even with quick workarounds. Should we change our workflow and assign IPs as soon as we add node to the cluster? Looks like it will cover several user requests. For example, it will allow to configure IP addresses before deployment.

Changed in fuel:
assignee: nobody → Fuel Python Team (fuel-python)
importance: Undecided → High
milestone: none → 7.0
status: New → Confirmed
Changed in fuel:
assignee: Fuel Python Team (fuel-python) → Alexander Kislitsky (akislitsky)
tags: added: feautre
tags: added: feature
removed: feautre
Changed in fuel:
assignee: Alexander Kislitsky (akislitsky) → Fuel Python Team (fuel-python)
Changed in fuel:
assignee: Fuel Python Team (fuel-python) → Przemyslaw Kaminski (pkaminski)
status: Confirmed → In Progress
Revision history for this message
Przemyslaw Kaminski (pkaminski) wrote :

Most probably API didn't accept your request, but it returns task with 200 status and error in dict. CLI cannot understand this and thinks all is fine and doesn't report error. Fix is complex: best would be to fix API and get rid of the task (https://bugs.launchpad.net/fuel/+bug/1423916). Since that requires some work, it's easier now to fix the client and parse task status and report error if any is found.

Revision history for this message
Przemyslaw Kaminski (pkaminski) wrote :

Since I don't yet know if fixing CLI will fix this bug, I report a separate bug for CLI: https://bugs.launchpad.net/fuel/+bug/1483211

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to python-fuelclient (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/211126

Changed in fuel:
assignee: Przemyslaw Kaminski (pkaminski) → Fuel Python Team (fuel-python)
Revision history for this message
Dima Shulyak (dshulyak) wrote :

The root problem here is that on reset - mapping in dnsmasq for mac <-> ip will be removed.
And if for some reason we wont remove it in nailgun, we will easily get a situation where ips assigned by dhcp will be different from stored in database.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to python-fuelclient (master)

Reviewed: https://review.openstack.org/211126
Committed: https://git.openstack.org/cgit/stackforge/python-fuelclient/commit/?id=cb810128d8d79612ef42e4b0da1909ad89c88bdb
Submitter: Jenkins
Branch: master

commit cb810128d8d79612ef42e4b0da1909ad89c88bdb
Author: Przemyslaw Kaminski <email address hidden>
Date: Mon Aug 10 14:34:35 2015 +0200

    Properly parse returned task from network configuration

    Change-Id: Ic994c359c689ab4192481f393d40c65939a39285
    Closes-Bug: #1483211
    Related-Bug: 1455610

Changed in fuel:
assignee: Fuel Python Team (fuel-python) → Vladimir Sharshov (vsharshov)
tags: added: feature-image-based feature-native-provisioning module-astute
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-astute (master)

Fix proposed to branch: master
Review: https://review.openstack.org/213244

Revision history for this message
Vladimir Sharshov (vsharshov) wrote :

I marking it as tricky because it required many tests not only for 7.0, but for 6.1 which upgraded to 7.0 also.

tags: added: tricky
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-astute (master)

Reviewed: https://review.openstack.org/213244
Committed: https://git.openstack.org/cgit/stackforge/fuel-astute/commit/?id=494cb4c305b107d68c0d595f2a3069043e7740ef
Submitter: Jenkins
Branch: master

commit 494cb4c305b107d68c0d595f2a3069043e7740ef
Author: Vladimir Sharshov (warpc) <email address hidden>
Date: Fri Aug 14 19:35:16 2015 +0300

    Save addresses on admin/pxe interfaces

    In case of reset instead of deleting Cobbler profile
    now Astute is going to perform such actions:

      1) chage it type to bootstrap type;
      2) enable pxe boot;
      3) reboot nodes.

    After starting provision after reset Astute is going to perform:

      1) change type of all nodes to bootstrap;
      2) enable pxe boot for all nodes;
      3) erase all nodes;
      4) reboot already existed nodes;
      5) delete all cluster nodes profiles;
      6) add all right after deleting (save ip);
      7) peform usual provision action.

    In case of cluster deletion we will destroy all cobbler profiles,
    erase nodes and reboot it as usual.

    Astute will use same mechanism as for node reinstallation in case of
    reseting operation to save addresses on admin/pxe interfaces.
    Closes-Bug: #1455610
    Closes-bug: #1486157
    Change-Id: I93b9eed571c2624028b84d801ea618a3137d6701

Changed in fuel:
status: In Progress → Fix Committed
tags: added: release-notes
tags: added: on-verification
Revision history for this message
Oleksiy Molchanov (omolchanov) wrote :

Verified

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "7.0"
  openstack_version: "2015.1.0-7.0"
  api: "1.0"
  build_number: "284"
  build_id: "284"
  nailgun_sha: "5c33995a2e6d9b1b8cdddfa2630689da5084506f"
  python-fuelclient_sha: "1ce8ecd8beb640f2f62f73435f4e18d1469979ac"
  fuel-agent_sha: "082a47bf014002e515001be05f99040437281a2d"
  fuel-nailgun-agent_sha: "d7027952870a35db8dc52f185bb1158cdd3d1ebd"
  astute_sha: "8283dc2932c24caab852ae9de15f94605cc350c6"
  fuel-library_sha: "f81fdabe6c05be7a3d11d88a7c3a8f3931921c73"
  fuel-ostf_sha: "1f08e6e71021179b9881a824d9c999957fcc7045"
  fuelmain_sha: "9ab01caf960013dc882825dc9b0e11ccf0b81cb0"

Changed in fuel:
status: Fix Committed → Fix Released
tags: removed: on-verification
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-docs (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/223475

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-docs (master)

Reviewed: https://review.openstack.org/223475
Committed: https://git.openstack.org/cgit/stackforge/fuel-docs/commit/?id=d78a8343733833b58c535b6107bb0e363e8550a9
Submitter: Jenkins
Branch: master

commit d78a8343733833b58c535b6107bb0e363e8550a9
Author: evkonstantinov <email address hidden>
Date: Tue Sep 15 11:20:11 2015 +0300

    Add redeployment with custom settings resolved issue to relnotes

    Change-Id: I0a12b5fe8e06a37410e647e5cee9d1c0034f0ecf
    Related-Bug:#1455610

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.