Bug #1506398 “octane upgrade-control breaks mysql and rabbit clu...” : Bugs : Fuel for OpenStack

Alexander Kislitsky (akislitsky) on 2015-10-15

Changed in fuel:
milestone:	none → 8.0

Dmitry Pyzhov (dpyzhov) on 2015-10-15

no longer affects:

fuel/8.0.x

Oleg S. Gelbukh (gelbuhos) on 2015-10-16

Changed in fuel:
status:	New → Confirmed

Oleg S. Gelbukh (gelbuhos) on 2015-10-16

tags:

added: module-octane

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-10-20: Fix proposed to fuel-octane (stable/7.0)

#1

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/237525

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-10-21: Fix merged to fuel-octane (stable/7.0)

#2

Reviewed: https://review.openstack.org/237525
Committed: https://git.openstack.org/cgit/openstack/fuel-octane/commit/?id=6b0a7649cfcf546ffe410e62aaaaf72bec9e5c2d
Submitter: Jenkins
Branch: stable/7.0

commit 6b0a7649cfcf546ffe410e62aaaaf72bec9e5c2d
Author: Oleg Gelbukh <email address hidden>
Date: Tue Oct 20 11:22:12 2015 +0000

Stop corosync in upgrade seed during control plane switch

    Stop corosync cluster before switching to upgraded control plane.
    Start corosync cluster once switch is complete.
    Don't stop cluster if there is only one controller in seed environment.

Closes-bug: 1506398
Change-Id: Ie5684155899675ce1c35636d9e892d399e41b790

Dmitry Pyzhov (dpyzhov) on 2015-10-22

tags:

added: area-octane

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-10-23: Related fix proposed to fuel-octane (stable/7.0)

#3

Related fix proposed to branch: stable/7.0
Review: https://review.openstack.org/238925

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-10-26: Related fix merged to fuel-octane (stable/7.0)

#4

Reviewed: https://review.openstack.org/238925
Committed: https://git.openstack.org/cgit/openstack/fuel-octane/commit/?id=a3226740bfacc15da032746d3093f77ecc0323ad
Submitter: Jenkins
Branch: stable/7.0

commit a3226740bfacc15da032746d3093f77ecc0323ad
Author: Oleg Gelbukh <email address hidden>
Date: Fri Oct 23 12:48:27 2015 +0000

Start corosync services after control plane switch for multiple controllers

    Before cluster is stopped, some corosync services are in 'Stop' status.
    Starting cluster doesn't bring them up, just restores state of the cluster.
    Explicitly start corosync services after the cluster is back.

Related-bug: 1506398
Change-Id: I38e90180f71e2786e18ffb67ec43a50aa2c5bee6

Dmitry Pyzhov (dpyzhov) on 2015-10-29

tags:

added: area-python
removed: area-octane

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-11-17: Related fix proposed to fuel-octane (master)

#5

Related fix proposed to branch: master
Review: https://review.openstack.org/246676

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-11-18: Related fix merged to fuel-octane (master)

#6

Download full text (11.6 KiB)

Reviewed: https://review.openstack.org/246676
Committed: https://git.openstack.org/cgit/openstack/fuel-octane/commit/?id=cae1d8af5d3ed7a6901fdb363f40e3bf7c37f26d
Submitter: Jenkins
Branch: master

commit e27cea930477dcaf5be29bfa4febd270dbe99e97
Author: Oleg Gelbukh <email address hidden>
Date: Thu Nov 12 15:15:57 2015 +0000

Version bump to 1.0.0

Change-Id: I51735d21337d51f52eead9625e2d9f3378c88799

commit 4d3c949e1f6c8319658ac713461e9f8f4a345eb4
Author: Oleg Gelbukh <email address hidden>
Date: Thu Nov 12 14:30:09 2015 +0000

Revert "Set RPM version 0.2.0"

This reverts commit 1c722afa0f78cfa3601f3a2b71085c6788052885.
Version downgrade must be avoided.

Change-Id: I6b3817ffa48ab321fe99bed7cb2df6a0941211ce

commit 1c722afa0f78cfa3601f3a2b71085c6788052885
Author: Oleg Gelbukh <email address hidden>
Date: Sun Nov 8 14:41:40 2015 +0000

Set RPM version 0.2.0

Change-Id: Ie0c75b979375e716c33082b9a4e83ffaffc7f42c

commit 3d06b9d8ddd3714166bdedeb91da5df6b8c9f7a6
Author: Valyavskiy Viacheslav <email address hidden>
Date: Sun Oct 4 03:14:37 2015 +0300

Use more correct approach to detect resources status

Closes-bug: 1499696
Change-Id: Ied72bc68ba41e47334796f09b407ebd8a3fb7f40

commit c315735e681928648fd4549956c3621c4caba4f9
Author: Sergey Novikov <email address hidden>
Date: Thu Oct 29 18:57:21 2015 +0300

Add handling of case when node may haven't full name in services data

Change-Id: Icf8dc176fc74fc1c5ee888ddb920b1acc58c127d

commit cf83b87b846e6613dc996d8a8dc70723bcbe84d1
Author: Oleg Gelbukh <email address hidden>
Date: Tue Oct 27 13:57:04 2015 +0000

Try to revert patches to Puppet modules every time

    If octane command was interrupted or failed before patches
    to puppet were reverted, the patch will be applied and repeated
    command will fail.

Try to revert the patch every time before apply it and skip
failures. Only actually apply the patch if revert is not assumed.

Change-Id: I5719dba3c621f307fe5d6ae2356f15d38ee28ff9

commit a3226740bfacc15da032746d3093f77ecc0323ad
Author: Oleg Gelbukh <email address hidden>
Date: Fri Oct 23 12:48:27 2015 +0000

Start corosync services after control plane switch for multiple controllers

    Before cluster is stopped, some corosync services are in 'Stop' status.
    Starting cluster doesn't bring them up, just restores state of the cluster.
    Explicitly start corosync services after the cluster is back.

Related-bug: 1506398
Change-Id: I38e90180f71e2786e18ffb67ec43a50aa2c5bee6

commit b3e496b43ecdd92a2a696147e2a487fcfe0b2c0d
Author: Oleg Gelbukh <email address hidden>
Date: Fri Oct 23 11:58:35 2015 +0000

Delete shell scripts and unused shell libs

As migration to Python is complete, remove unused shell scripts and libs

Change-Id: I5cc13bebfd0423f73c46722d276807154f2b79e8

commit 6b0a7649cfcf546ffe410e62aaaaf72bec9e5c2d
Author: Oleg Gelbukh <email address hidden>
Date: Tue Oct 20 11:22:12 2015 +0000

Stop corosync in upgrade seed during control plane switch

Stop corosync cluster before switching to upgrade...

Reviewed:  https://review.openstack.org/246676
Committed: https://git.openstack.org/cgit/openstack/fuel-octane/commit/?id=cae1d8af5d3ed7a6901fdb363f40e3bf7c37f26d
Submitter: Jenkins
Branch:    master

commit e27cea930477dcaf5be29bfa4febd270dbe99e97
Author: Oleg Gelbukh <ogelbukh@mirantis.com>
Date:   Thu Nov 12 15:15:57 2015 +0000

Version bump to 1.0.0
    
    Change-Id: I51735d21337d51f52eead9625e2d9f3378c88799

commit 4d3c949e1f6c8319658ac713461e9f8f4a345eb4
Author: Oleg Gelbukh <ogelbukh@mirantis.com>
Date:   Thu Nov 12 14:30:09 2015 +0000

Revert "Set RPM version 0.2.0"
    
    This reverts commit 1c722afa0f78cfa3601f3a2b71085c6788052885.
    Version downgrade must be avoided.
    
    Change-Id: I6b3817ffa48ab321fe99bed7cb2df6a0941211ce

commit 1c722afa0f78cfa3601f3a2b71085c6788052885
Author: Oleg Gelbukh <ogelbukh@mirantis.com>
Date:   Sun Nov 8 14:41:40 2015 +0000

Set RPM version 0.2.0
    
    Change-Id: Ie0c75b979375e716c33082b9a4e83ffaffc7f42c

commit 3d06b9d8ddd3714166bdedeb91da5df6b8c9f7a6
Author: Valyavskiy Viacheslav <vvalyavskiy@mirantis.com>
Date:   Sun Oct 4 03:14:37 2015 +0300

Use more correct approach to detect resources status
    
    Closes-bug: 1499696
    Change-Id: Ied72bc68ba41e47334796f09b407ebd8a3fb7f40

commit c315735e681928648fd4549956c3621c4caba4f9
Author: Sergey Novikov <snovikov@mirantis.com>
Date:   Thu Oct 29 18:57:21 2015 +0300

Add handling of case when node may haven't full name in services data
    
    Change-Id: Icf8dc176fc74fc1c5ee888ddb920b1acc58c127d

commit cf83b87b846e6613dc996d8a8dc70723bcbe84d1
Author: Oleg Gelbukh <ogelbukh@mirantis.com>
Date:   Tue Oct 27 13:57:04 2015 +0000

Try to revert patches to Puppet modules every time
    
    If octane command was interrupted or failed before patches
    to puppet were reverted, the patch will be applied and repeated
    command will fail.
    
    Try to revert the patch every time before apply it and skip
    failures. Only actually apply the patch if revert is not assumed.
    
    Change-Id: I5719dba3c621f307fe5d6ae2356f15d38ee28ff9

commit a3226740bfacc15da032746d3093f77ecc0323ad
Author: Oleg Gelbukh <ogelbukh@mirantis.com>
Date:   Fri Oct 23 12:48:27 2015 +0000

Start corosync services after control plane switch for multiple controllers
    
    Before cluster is stopped, some corosync services are in 'Stop' status.
    Starting cluster doesn't bring them up, just restores state of the cluster.
    Explicitly start corosync services after the cluster is back.
    
    Related-bug: 1506398
    Change-Id: I38e90180f71e2786e18ffb67ec43a50aa2c5bee6

commit b3e496b43ecdd92a2a696147e2a487fcfe0b2c0d
Author: Oleg Gelbukh <ogelbukh@mirantis.com>
Date:   Fri Oct 23 11:58:35 2015 +0000

Delete shell scripts and unused shell libs
    
    As migration to Python is complete, remove unused shell scripts and libs
    
    Change-Id: I5cc13bebfd0423f73c46722d276807154f2b79e8

commit 6b0a7649cfcf546ffe410e62aaaaf72bec9e5c2d
Author: Oleg Gelbukh <ogelbukh@mirantis.com>
Date:   Tue Oct 20 11:22:12 2015 +0000

Stop corosync in upgrade seed during control plane switch
    
    Stop corosync cluster before switching to upgraded control plane.
    Start corosync cluster once switch is complete.
    Don't stop cluster if there is only one controller in seed environment.
    
    Closes-bug: 1506398
    Change-Id: Ie5684155899675ce1c35636d9e892d399e41b790

commit 7cfa6da2b23cc1ac52ca2c7aec6a39d5608fef19
Author: Valyavskiy Viacheslav <vvalyavskiy@mirantis.com>
Date:   Mon Oct 5 13:47:48 2015 +0300

Add possibility to switch controlplane back
    
    Change-Id: Ia8fe76b59fa113e883044c04c5d14389e0f59586

commit b8a43ecd4b2f3cf34c2a4ab0fef7a5d012b81b92
Author: Valyavskiy Viacheslav <vvalyavskiy@mirantis.com>
Date:   Mon Oct 5 13:39:27 2015 +0300

Refactor upgrade_controlplane parts
    
    * move network related methods from `upgrade_controlplane` to `env`
      module
    * collect yaml files from nodes if bup info is not present
    * extract `delete_overlay_networks` action to separate function
    
    Change-Id: I476a4f7fe884500e7d0c4c401cdeeb8c9af46e63

commit c434b6a85a9bb1bfbb36a2c114ce358ed24a97dd
Author: Oleg Gelbukh <ogelbukh@mirantis.com>
Date:   Mon Oct 19 11:21:03 2015 +0000

Use /tmp as cache dir for temporary files
    
    Temporary files stored during the upgrade procedure don't need special
    directory. They can be kept in /tmp directly.
    
    Closes-bug: 1507544
    Change-Id: Ib4dc68768892f70096596fb5fc10ec2f8cea262b

commit edce3ec92c9e8b6947463c11fca3a8248f422ce9
Author: Jeremy Stanley <fungi@yuggoth.org>
Date:   Sat Oct 17 22:20:47 2015 +0000

Update .gitreview for new namespace
    
    Change-Id: I69511f6202c422ac6742282caf1971a6db45cb21

commit 91eec566094d6ad36cb28ed0985958c62866cfd3
Author: Ilya Kharin <akscram@gmail.com>
Date:   Wed Oct 14 14:05:21 2015 +0300

Set pyzabbix as an optional dependency
    
    This patch adds the zabbix extra feature and the
    'update-plugin-settings' command depends on this feature.
    
    DocImpact
    Closes-Bug: #1501667
    Change-Id: I7a638ca5cb7502ac9ef9f4fb6ec69529c16ce1c9

commit f28bcb1e307b52a6e935b72a568186700048688a
Author: Maksim Malchuk <maksim.malchuk@gmail.com>
Date:   Thu Oct 15 16:21:04 2015 +0300

Increase default timeout for provision and deploy nodes.
    Also add some more verbose.
    
    Change-Id: Icff2c152b5af523a4ad9f56c117b60bc05c558a4
    Closes-Bug: #1506407

commit 7aeccac69042e78cd793dbe8c6fee79b8a860c0b
Author: Oleg Gelbukh <ogelbukh@mirantis.com>
Date:   Wed Oct 7 13:09:24 2015 +0000

Add MAINTAINERS file in fuel-octane
    
    Change-Id: Ie981589779edc9075568de3a9a0e1d2901e292c2

commit dc9b58a708f648167808f30deca9135c14b28125
Author: Oleg Gelbukh <ogelbukh@mirantis.com>
Date:   Tue Oct 6 14:25:35 2015 +0000

Preserve data on Ceph OSD devices
    
    Patch puppet-ceph module with backported patch from
    8.0 to ensure that it doesn't run 'ceph-deploy prepare'
    on existing Ceph OSD devices.
    
    Closes-bug: 1503424
    Change-Id: I4ddd7b6ba3077ad32961079eebbaf719521bf47a

commit 02e2394f1b08d3ba5ddc6a830e31026324ef4246
Author: Valyavskiy Viacheslav <vvalyavskiy@mirantis.com>
Date:   Thu Oct 1 13:49:24 2015 +0300

Fix network functions
    
    * Add support of 5.1 releases for network methods
    * Make `connect_to_networks` and `disconnect_networks` methods idempotent
    
    Change-Id: I569ed5a3bf493dc10050267a89963ab474e6bb8a

commit 318714011aa1dd09d8b66f1da19b3bb70c08146c
Author: Valyavskiy Viacheslav <vvalyavskiy@mirantis.com>
Date:   Mon Oct 12 17:20:40 2015 +0300

Re-upload whole deployment info in one chunk
    
    Change-Id: Ifd10e655f8fa5612e7b28055b8068dec511a0ba3

commit 50841eb85d692adef92563930d6eb1b2b42c0271
Author: Oleg Gelbukh <ogelbukh@mirantis.com>
Date:   Sun Oct 11 20:13:54 2015 +0000

Upgrade one node without deploy changes
    
    Deployment of single node allows to install 1 ceph-osd node in env with
    replication factor of 2 without failure. It also allows to avoid unnecessary
    deployment run on controllers.
    
    Change-Id: Ia1f7193f1e2bae57f457b88ca3e6ade3d89c7993
    Closes-bug: 1501776

commit 0570c0d8e2c0b11db030390b506b571a1b50c4cc
Author: Ilya Kharin <akscram@gmail.com>
Date:   Wed Sep 16 12:08:20 2015 +0300

Implement upgrade-ceph in Python
    
    Change-Id: Id46d1fc6d016f5bab780c891bc343dbdcdd67a65

commit 537f38f70d79bc798f0cec3845a296734a62fb35
Author: smurashov <smurashov@mirantis.com>
Date:   Wed Oct 7 15:48:57 2015 +0300

Set upgrade_levels in post_deploy handler
    
    Set upgrade_levels in post_deploy handlers
    for controllers and computes
    
    Change-Id: I139ce54440b69f9699e50a9d15083f16af1aa259
    Closes-Bug: #1501692

commit f793e64833c36b1165340a914d0f6eafb5a77c9b
Author: smurashov <smurashov@mirantis.com>
Date:   Wed Oct 7 16:19:51 2015 +0300

Replace node-{id} to fqdn
    
    Change-Id: Id4faf74815009a455fa56d5f1ba380cf8c06d491
    Closes-Bug: #1501748

commit 2ca66b00053c0ba3d76ada6e5cf54bf670ce07f3
Author: Oleg Gelbukh <ogelbukh@mirantis.com>
Date:   Wed Oct 7 04:57:27 2015 +0000

Cap six version according to fuelclient requires
    
    New 1.10.0 version of six is incompatible with fuelclient requirements.
    
    Change-Id: I95d9899912ca088f425f138da7fcf4b524adca19

commit d58ab97613f1d1fe989386656a3584328ce2ba95
Author: Oleg Gelbukh <ogelbukh@mirantis.com>
Date:   Tue Sep 29 12:30:34 2015 +0000

Properly use deployment info in 7.0
    
    In 7.0, default deployment information is returned
    for 'ready' nodes and must be used instead of outdated
    replaced deployment information.
    
    Closes-bug: 1494507
    Closes-bug: 1501728
    Change-Id: I545bfabb6c548667464de03fd5aa005d3336002d

commit b9bffce397bb365222ae3e77cb1582bf84d879c0
Author: Yuriy Taraday <yorik.sar@gmail.com>
Date:   Mon Oct 5 22:04:12 2015 +0300

Cap versions of stevedore and oslo.config
    
    We have to follow python-fuelclient requirements
    
    Change-Id: Id76d2de585577788896d849ce466d8939003f511

commit ad0ebc4f8b73b6a5ee2e19694cd2bc06b4b88871
Author: Maksim Malchuk <mmalchuk@mirantis.com>
Date:   Sat Oct 3 01:45:07 2015 +0300

Exclude configure_route task from deployment graph
    
    Currently, configure_route task reset node's default gw to
    vroute's ip. It leads to missing connectivity and deployment
    failure for nodes where task is applied.
    
    Change-Id: If496d63091bf75ea541c39d016f968737918cc67

commit 3bbc8b9c5fcc3f8c523042a30c1cbf137e71c081
Author: Maksim Malchuk <mmalchuk@mirantis.com>
Date:   Fri Oct 2 18:16:06 2015 +0300

Be more informative when provision and deploy.
    
    Change-Id: Icdcdb69f16503cd57eef9f31f9319dd69f8d1537

commit d63028bec7b1f05559de9a906dfed4202bd32d5b
Author: Maksim Malchuk <mmalchuk@mirantis.com>
Date:   Fri Oct 2 18:13:23 2015 +0300

Be more informative when provision and deploy.
    
    Change-Id: I02fb70c6ddc00c44219e32fdda7c20f48f534068

commit 90b6ba5d07617e9b9b41736fcc6a2daaf82ba5b2
Author: Yuriy Taraday <yorik.sar@gmail.com>
Date:   Wed Sep 30 11:18:30 2015 +0300

Point to stable/7.0 in .gitreview
    
    Change-Id: I98006ef8a89666fabbbab7b6441a745e615ecc1c

commit 4ff1bb1fb50014a74262e1a7feb2fb5ad9164b3f
Author: Oleg Gelbukh <ogelbukh@mirantis.com>
Date:   Fri Sep 18 13:54:47 2015 +0000

Properly cache nova service tenant id
    
    The bug causes update_neutron_config function in upgrade_controlplane module
    look for ID of service tenant in a file with an id of seed env in its name.
    However, the file is created with id of original env.
    This bug won't affect scenario when sync-images command is run, because it will
    create the file with id of seed env.
    Fixed by splitting function to get tenant id from env and cache tenant id to
    file. Use cache function to write/get tenant id from source env.
    
    Change-Id: I5ae4aa39099084763a6f1d702e7d4131d69710ed

commit b5e00be549b4c7b6639201780953a9f54c2c87dc
Author: Oleg Gelbukh <ogelbukh@mirantis.com>
Date:   Fri Sep 25 12:55:21 2015 +0000

Use default gw in admin network for deployment
    
    Set default gw in admin network for isolated deployment of controller, if
    defined, otherwise set default gw to master node's ip address.
    
    Change-Id: I0adb9e3b78c29d5fb32ccd60e078fd0fff352647

commit 789412b962bd7e950146c23ec136b12f2eb0b641
Author: Oleg Gelbukh <ogelbukh@mirantis.com>
Date:   Tue Sep 29 19:13:23 2015 +0000

Cap version of pytest to avoid InvocationError in python 2.6
    
    Since 2.8.0, pytest exit code is 5 if it can't find tests for some files.
    Work around that by capping version of pytest<2.8.0.
    
    Change-Id: Ia078c4ecb246b4cc302fd5a17734dc9f59a1b9ea
    (cherry picked from commit 09f4ba7df56e8c75a24127e93a1cf4b0add7c96b)

Oleg S. Gelbukh (gelbuhos) on 2015-11-19

Changed in fuel:
status:	Confirmed → Fix Committed

Nastya Urlapova (aurlapova) on 2016-02-09

tags:

added: dev-to-verify

Oleg S. Gelbukh (gelbuhos) on 2016-05-18

Changed in fuel:
assignee:	Registry Administrators (registry) → Fuel Octane (fuel-octane-team)

Nastya Urlapova (aurlapova) on 2016-06-17

tags:

added: team-upgrades
removed: dev-to-verify

Curtis Hovey (sinzui) on 2017-04-13

Changed in fuel:
assignee:	Registry Administrators (registry) → nobody

Affects		Status	Importance	Assigned to	Milestone
	Fuel for OpenStack	Fix Committed	High	Unassigned	Fuel for OpenStack 8.0
	7.0.x	Fix Committed	High	Oleg S. Gelbukh	Fuel for OpenStack 7.0-updates

Fuel for OpenStack

octane upgrade-control breaks mysql and rabbit clusters

Bug Description

Other bug subscribers

Remote bug watches