[octane] Upgrade of compute node fails due to mismatch in partition table

Bug #1575054 reported by Pavel Chechetin
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
High
Sergey Abramov
8.0.x
Fix Committed
High
Sergey Abramov
Newton
Fix Committed
High
Sergey Abramov

Bug Description

Detailed bug description:
  Upgrade of compute node fails due to mismatch in partition table.
  During upgrade octane complained that machine which undergoes upgrade fell in error state.
  Astute log revealed the line below:
  "Unexpected error\nUnexpected error while running command.\nCommand: mkswap /dev/vda5\nExit code: 1\nStdout: ''\nStderr:'/dev/vda5: Device or resource busy\\n'\n"

  /dev/vda4 is a swap partition.

Steps to reproduce:
  octane upgrade-node 8 8
Expected results:
  Compute node is upgraded
Actual result:
  Compute node isn't upgraded
Reproducibility:
  Always if partition preservation is used

Description of the environment:
  Source Fuel version: 6.0
  Destination Fuel version: 8.0
  Operation system: Ubuntu

Changed in fuel:
status: New → Confirmed
importance: Undecided → High
assignee: nobody → Fuel Octane (fuel-octane-team)
description: updated
Revision history for this message
Oleg S. Gelbukh (gelbuhos) wrote :

To fix this bug, we need to patch fuel-agent installed in the bootstrap image so it properly handles the partitions schema created in the initial provisioning (i.e. in version 6.x). In that versions, the following partitions were created:

1. boot
2. root
3. swap

Since version 7.0, new provisioning schema (image-based) was made defualt. This schema creates the following partitions:

1. boot
2. UEFI boot
3. root
4. swap

This leads to the situation when swap partition physically is 3rd (due to partition preservation), but fuel-agent expects it to be 4th.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-octane (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/321474

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Related fix proposed to branch: master
Review: https://review.openstack.org/321495

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-octane (master)

Fix proposed to branch: master
Review: https://review.openstack.org/321553

Changed in fuel:
assignee: Sergey Abramov (sabramov) → Oleg S. Gelbukh (gelbuhos)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-octane (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/322197

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-octane (master)

Reviewed: https://review.openstack.org/321474
Committed: https://git.openstack.org/cgit/openstack/fuel-octane/commit/?id=258d16b0f3dd9ebb8c4466108709911192507b69
Submitter: Jenkins
Branch: master

commit 258d16b0f3dd9ebb8c4466108709911192507b69
Author: Oleg Gelbukh <email address hidden>
Date: Thu May 26 11:05:04 2016 +0000

    Move disk.py module to util module

    Current location for disk.py modules is inconsistent with the
    usage of utils for basic operations on disks, not only their
    metadata. Non-metadata operations must be implemented by util
    module.

    Change-Id: Ia53b831f4b4d5c31da8801a8561b4721fd35d511
    Related-bug: 1585938
    Related-bug: 1575054

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-octane (stable/8.0)

Related fix proposed to branch: stable/8.0
Review: https://review.openstack.org/322839

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-octane (stable/8.0)

Reviewed: https://review.openstack.org/322839
Committed: https://git.openstack.org/cgit/openstack/fuel-octane/commit/?id=75aa4161fe1e6edd64e4f46f2aed8a7ebf1bbced
Submitter: Jenkins
Branch: stable/8.0

commit 75aa4161fe1e6edd64e4f46f2aed8a7ebf1bbced
Author: Oleg Gelbukh <email address hidden>
Date: Thu May 26 11:05:04 2016 +0000

    Move disk.py module to util module

    Current location for disk.py modules is inconsistent with the
    usage of utils for basic operations on disks, not only their
    metadata. Non-metadata operations must be implemented by util
    module.

    Change-Id: Ia53b831f4b4d5c31da8801a8561b4721fd35d511
    Related-bug: 1585938
    Related-bug: 1575054
    (cherry picked from commit 258d16b0f3dd9ebb8c4466108709911192507b69)

Changed in fuel:
assignee: Oleg S. Gelbukh (gelbuhos) → Pavel Chechetin (paulche)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-octane (master)

Reviewed: https://review.openstack.org/321478
Committed: https://git.openstack.org/cgit/openstack/fuel-octane/commit/?id=c45e4be0a88addf06821f7ff246f941799e6b6b1
Submitter: Jenkins
Branch: master

commit c45e4be0a88addf06821f7ff246f941799e6b6b1
Author: Pavel Chechetin <email address hidden>
Date: Mon Jun 6 16:50:11 2016 +0300

    Update fuel_agent patch for Fuel 8.0

    Partial-Bug: 1575054
    Change-Id: I48ba3a79c30ca8b21b880dab7cbb196ae3cb0712

Changed in fuel:
assignee: Pavel Chechetin (paulche) → Sergey Abramov (sabramov)
Changed in fuel:
assignee: Sergey Abramov (sabramov) → Oleg S. Gelbukh (gelbuhos)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-octane (master)

Reviewed: https://review.openstack.org/322197
Committed: https://git.openstack.org/cgit/openstack/fuel-octane/commit/?id=7f9d271db41f43292503239291accffe87459312
Submitter: Jenkins
Branch: master

commit 7f9d271db41f43292503239291accffe87459312
Author: Oleg Gelbukh <email address hidden>
Date: Fri May 27 14:23:55 2016 +0000

    Replace ssh.call with ssh.call_output for listing partitions

    Going from ssh.call to ssh.call_output simplifies code and testing.

    Change-Id: I5d115ccd1ae69f6b11434ca8f3ba84ef3b3aa049
    Related-bug: 1575054

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-octane (stable/8.0)

Related fix proposed to branch: stable/8.0
Review: https://review.openstack.org/330551

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-octane (stable/8.0)

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/330614

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-octane (stable/8.0)

Reviewed: https://review.openstack.org/330551
Committed: https://git.openstack.org/cgit/openstack/fuel-octane/commit/?id=0bb9c7578d8e623ded0a627c939109804285fab3
Submitter: Jenkins
Branch: stable/8.0

commit 0bb9c7578d8e623ded0a627c939109804285fab3
Author: Oleg Gelbukh <email address hidden>
Date: Fri May 27 14:23:55 2016 +0000

    Replace ssh.call with ssh.call_output for listing partitions

    Going from ssh.call to ssh.call_output simplifies code and testing.

    Change-Id: I5d115ccd1ae69f6b11434ca8f3ba84ef3b3aa049
    Related-bug: 1575054
    (cherry picked from commit 7f9d271db41f43292503239291accffe87459312)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-octane (stable/8.0)

Reviewed: https://review.openstack.org/330614
Committed: https://git.openstack.org/cgit/openstack/fuel-octane/commit/?id=b04c8c3fd55949cb84967a582361a1c60ad09102
Submitter: Jenkins
Branch: stable/8.0

commit b04c8c3fd55949cb84967a582361a1c60ad09102
Author: Pavel Chechetin <email address hidden>
Date: Mon Jun 6 16:50:11 2016 +0300

    Update fuel_agent patch for Fuel 8.0

    Partial-Bug: 1575054
    Change-Id: I48ba3a79c30ca8b21b880dab7cbb196ae3cb0712
    (cherry picked from commit c45e4be0a88addf06821f7ff246f941799e6b6b1)

Changed in fuel:
assignee: Oleg S. Gelbukh (gelbuhos) → Sergey Abramov (sabramov)
Changed in fuel:
assignee: Sergey Abramov (sabramov) → Oleg S. Gelbukh (gelbuhos)
Changed in fuel:
assignee: Oleg S. Gelbukh (gelbuhos) → Sergey Abramov (sabramov)
Changed in fuel:
assignee: Sergey Abramov (sabramov) → Oleg S. Gelbukh (gelbuhos)
Changed in fuel:
assignee: Oleg S. Gelbukh (gelbuhos) → Sergey Abramov (sabramov)
assignee: Sergey Abramov (sabramov) → Oleg S. Gelbukh (gelbuhos)
assignee: Oleg S. Gelbukh (gelbuhos) → Sergey Abramov (sabramov)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-octane (master)

Reviewed: https://review.openstack.org/321495
Committed: https://git.openstack.org/cgit/openstack/fuel-octane/commit/?id=b5d884296c35c5bf41ec3928c7a96d2f55e3c2b5
Submitter: Jenkins
Branch: master

commit b5d884296c35c5bf41ec3928c7a96d2f55e3c2b5
Author: Oleg Gelbukh <email address hidden>
Date: Thu May 26 11:54:57 2016 +0000

    Function to create config-drive partition

    Add function to create partition for config-drive to util/disk.py
    module. This is required to switch from 'cobbler' to 'image'
    provision method.

    Change-Id: Ia52195a2e25e4e459c8947eb2dfaea97195dbe7f
    Related-bug: 1575054

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-octane (master)

Reviewed: https://review.openstack.org/321553
Committed: https://git.openstack.org/cgit/openstack/fuel-octane/commit/?id=46586a62df962b2ce00b5d8a63f6fd34c920a1f4
Submitter: Jenkins
Branch: master

commit 46586a62df962b2ce00b5d8a63f6fd34c920a1f4
Author: Oleg Gelbukh <email address hidden>
Date: Thu May 26 13:07:52 2016 +0000

    Improve partitions and disk metadata handling

    Partitions schema and disk metadata at nodes must be updated only
    if they are scheduled for reprovisioning, and the provisioning
    method has changed to incompatible (i.e. from 'cobbler' to 'image')
    due to the shift of release version.

    Move logic that manages partitions from upgrade handlers to function
    that actually initiates the re-provisioning of node. This ensures
    that for every node this is applied only once and only if necessary.

    Change-Id: I28718564543e3ccd92d985bd79c76e509e8eae1f
    Closes-bug: 1585938
    Closes-bug: 1575054

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-octane (stable/8.0)

Related fix proposed to branch: stable/8.0
Review: https://review.openstack.org/332831

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-octane (stable/8.0)

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/332832

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/332877

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-octane (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/332931

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-octane (stable/mitaka)
Download full text (40.5 KiB)

Reviewed: https://review.openstack.org/332931
Committed: https://git.openstack.org/cgit/openstack/fuel-octane/commit/?id=24f3c396612eb0c69fbf06bab3cebbb9ca829426
Submitter: Jenkins
Branch: stable/mitaka

commit b34d749f1c281dafbdbf155bd86830dc8f2a9aa2
Author: Ilya Kharin <email address hidden>
Date: Wed Jun 22 21:10:56 2016 +0300

    Support mock==1.8.0 in unit tests

    Use assert_called_once_with instead of assert_called_once that was
    introduced in 2.0.0.

    Change-Id: Ifb7699d4b552d148984961727355f0e23b487c7c

commit d60f1905143bb7576ffd670595de1c4aeafd7b34
Author: Ilya Kharin <email address hidden>
Date: Wed Jun 22 02:01:52 2016 +0300

    Allow to authorize by the predefined admin_token

    The admin_token_auth middleware is added to keystone pipelines to
    allow authorization by admin_token.

    Change-Id: Ic03150305a669fad1446436a68051fb9aa25b892

commit cc7fab59f44ffef60285f8732d798f52469b2530
Author: Ilya Kharin <email address hidden>
Date: Wed Jun 22 01:47:01 2016 +0300

    Reset default_domain_id before the keystone task

    The default_domain_id should be removed from keystone.conf after restore
    of DB and before to apply the keystone puppet task to avoid of using the
    configured domain as a default.

    Change-Id: I05a6c48532e8042496b3d8ccef53d65bf8c44653

commit e3f82399d567dbcfc1ae9a1ecbddba7bf5028fc8
Author: Ilya Kharin <email address hidden>
Date: Wed Jun 22 01:32:47 2016 +0300

    Add helper function to iterate over parameters

    The helpers.iterate_parameters function allows to iterate over lines of
    INI-like files along with a context of information, such section,
    parameter and value.

    Change-Id: I55b179118116fd5dacf100754057ea6589782dc2

commit 5ed370a4fbb3369d0f75873d205ac7f0c655f93a
Author: Ilya Kharin <email address hidden>
Date: Wed Jun 22 01:18:05 2016 +0300

    Add update_file context manager for local files

    The subprocess.update_file function provides an ability to update
    content of a local file by iterating over lines of an original file and
    forming a result content in a temporary file to replace the original
    file in the end. This function is very useful to change configuration
    files.

    Change-Id: I433a5da67887b231400dd63131799019f45c277c

commit 58f31e6c5f408630c42565ee53de6b59457bc84c
Author: Oleg Gelbukh <email address hidden>
Date: Wed Jun 22 14:34:03 2016 +0000

    Escape passwords passed to openstack client

    If special symbols are used in password for 'admin' user in
    OpenStack, octane passes them to command line client as is
    and it breaks shell.

    Properly escape the password before passing it to subprocess.

    Change-Id: Iad635aec6d5b5cc32975937e00205b7e89dc99d9
    Closes-bug: 1585960

commit 896aba1191eeb59cf85cc8be6a2ae67e08b76070
Author: Yuriy Taraday <email address hidden>
Date: Wed Jun 22 15:28:45 2016 +0300

    Add absolute_import to util/docker.py to avoid local tempfile module

    Closes-Bug: 1595156
    Change-Id: I9484efce6fa7aec1b41cf592f9e9768d85931fa7

commit 46586a62df962b2ce00b5d8a63f6fd34c920a1f4
Author: Oleg Gelbukh <email address hidden>
Date: Thu M...

tags: added: in-stable-mitaka
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-octane (master)

Reviewed: https://review.openstack.org/327566
Committed: https://git.openstack.org/cgit/openstack/fuel-octane/commit/?id=3a432b63c9ab21b97810896696d212328d738bc4
Submitter: Jenkins
Branch: master

commit 3a432b63c9ab21b97810896696d212328d738bc4
Author: Sergey Abramov <email address hidden>
Date: Thu Jun 9 13:08:46 2016 +0300

    Add patch-active-img command

    Required for creating new patched bootstrap image
    based on active bootstrap image.

    It patches active bootstrap image to skip the step
    that add UEFI.

    Note:

    * You should active your patched image using command:

        fuel-bootstrap activate <img_uuid>

    * squashfs-tools should be installed on master node

        yum install squashfs-tools

    Usage:

        octane patch-active-img

    Closes-bug: 1575054

    Change-Id: I7b12db209b5d2db158d4c26bc162bab504aa1590

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-octane (stable/8.0)

Reviewed: https://review.openstack.org/332831
Committed: https://git.openstack.org/cgit/openstack/fuel-octane/commit/?id=f0ed3cb7b1259c653d3dd62a76b6d0e12514aeb0
Submitter: Jenkins
Branch: stable/8.0

commit f0ed3cb7b1259c653d3dd62a76b6d0e12514aeb0
Author: Oleg Gelbukh <email address hidden>
Date: Thu May 26 11:54:57 2016 +0000

    Function to create config-drive partition

    Add function to create partition for config-drive to util/disk.py
    module. This is required to switch from 'cobbler' to 'image'
    provision method.

    Change-Id: Ia52195a2e25e4e459c8947eb2dfaea97195dbe7f
    Related-bug: 1575054
    (cherry picked from commit b5d884296c35c5bf41ec3928c7a96d2f55e3c2b5)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-octane (stable/8.0)

Reviewed: https://review.openstack.org/332832
Committed: https://git.openstack.org/cgit/openstack/fuel-octane/commit/?id=a35313d561c2b66b9c7576ec0d9ff8e3425f94ed
Submitter: Jenkins
Branch: stable/8.0

commit a35313d561c2b66b9c7576ec0d9ff8e3425f94ed
Author: Oleg Gelbukh <email address hidden>
Date: Thu May 26 13:07:52 2016 +0000

    Improve partitions and disk metadata handling

    Partitions schema and disk metadata at nodes must be updated only
    if they are scheduled for reprovisioning, and the provisioning
    method has changed to incompatible (i.e. from 'cobbler' to 'image')
    due to the shift of release version.

    Move logic that manages partitions from upgrade handlers to function
    that actually initiates the re-provisioning of node. This ensures
    that for every node this is applied only once and only if necessary.

    Change-Id: I28718564543e3ccd92d985bd79c76e509e8eae1f
    Closes-bug: 1585938
    Closes-bug: 1575054
    (cherry picked from commit 46586a62df962b2ce00b5d8a63f6fd34c920a1f4)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/332877
Committed: https://git.openstack.org/cgit/openstack/fuel-octane/commit/?id=d0f3e3e373395c452054542abe5fdbadea8d43ae
Submitter: Jenkins
Branch: stable/8.0

commit d0f3e3e373395c452054542abe5fdbadea8d43ae
Author: Sergey Abramov <email address hidden>
Date: Thu Jun 9 13:08:46 2016 +0300

    Add patch-active-img command

    Required for creating new patched bootstrap image
    based on active bootstrap image.

    It patches active bootstrap image to skip the step
    that add UEFI.

    Note:

    * You should active your patched image using command:

        fuel-bootstrap activate <img_uuid>

    * squashfs-tools should be installed on master node

        yum install squashfs-tools

    Usage:

        octane patch-active-img

    Closes-bug: 1575054

    Change-Id: I7b12db209b5d2db158d4c26bc162bab504aa1590
    (cherry picked from commit 3a432b63c9ab21b97810896696d212328d738bc4)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-octane (master)

Fix proposed to branch: master
Review: https://review.openstack.org/351721

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-octane (master)

Change abandoned by Ilya Kharin (<email address hidden>) on branch: master
Review: https://review.openstack.org/351721
Reason: This patch have to go in stable/mitaka instead of master.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-octane (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/351728

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-octane (stable/mitaka)
Download full text (17.8 KiB)

Reviewed: https://review.openstack.org/351728
Committed: https://git.openstack.org/cgit/openstack/fuel-octane/commit/?id=3101bf302ea0f2d501fd42a6ae6046c3587ac093
Submitter: Jenkins
Branch: stable/mitaka

commit a05d2ecf13e15c12f10e1517fae425e583e0d714
Author: Nikita Zubkov <email address hidden>
Date: Thu Aug 4 17:51:56 2016 +0300

    fix ssh connection auth

    According to spec [1] service user must use key authentication​ for ssh

    [1] https://github.com/openstack/fuel-specs/blame/ba4504/specs/9.0/fuel-nonroot-openstack-nodes.rst#L61-L64

    Change-Id: I461f659d4f0985283a43ec6a702bdd78af91a072

commit 7140398e501879a4e1f5605f3d1cc99195464850
Author: Sergey Abramov <email address hidden>
Date: Tue Aug 2 12:12:57 2016 +0300

    Setup upgrade levels for each fuel version

    Setup upgrade levels block in nova.conf for each fuel version,
    this is required for backward compatibility on RPC.

    Change-Id: I6ccbc34457d11616b015cdb1e46e733b0b49ce58

commit 9e2a419ac79ad591faee9afe16419bf4497eacad
Author: Vladimir Khlyunev <email address hidden>
Date: Mon Aug 1 21:31:56 2016 +0300

    Fix several typos

    Copypasting is evil, lets remove typos

    Change-Id: Icf4bec1a3de954911a469fa98380af2b6175da08

commit 013a7ded2275245a38ee58289dac2bc8b90fa37e
Author: Sergey Abramov <email address hidden>
Date: Wed Jul 20 16:05:10 2016 +0300

    Host evacuation using nova

    Evacuate instance from compute node using
    nova host-evacuate-live except host_evacuation.sh script

    Change-Id: I01a04cfa1ed2aafce7987e523e6b589dea21fbb0

commit 00a1d0dce1f7ae5db6706a59353403516f7913dc
Author: Sergey Abramov <email address hidden>
Date: Tue Jul 19 17:10:44 2016 +0300

    Change shutoff vm command

    * run nova commands using nova util function
    * host filter using get nova node handle method(
        hostname for fuel < 6.1 and fqdn for newer version)

    Change-Id: Ib5c373714e359982b13910a225787167db7b35e9

commit 49357c98313c5f551b13d0877581c5fb5e108db1
Author: Nikita Zubkov <email address hidden>
Date: Mon Jun 27 18:40:13 2016 +0300

    Add `stderr_log_level` parameter to popen

    Now posible to set logging level in popen for logging stderr

    fuel-bootstrap build comand stderr now logged with INFO level

    Change-Id: Ibaa2be54c68ce7a428e07052c917f54e99b95a5e

commit fb582b271dab4bdd10d27149a77c0e1f6a580b9e
Author: Sergey Abramov <email address hidden>
Date: Thu Jul 21 17:56:41 2016 +0300

    Add run_nova_cmd function

    This is required for running nova command on contoller without
    manipulating manipulating with command.

    It makes code easier and more readable

    Usage:

        from fuelclient.objects import environment as environment_obj

        from octane.util import env
        from octane.util import nova

        env_id = ...
        my_env = environment_obj.Environment(env_id)
        controller = env.get_one_controller(my_env)
        nova_cmd_just_run = [...]
        nova_cmd_with_output = [...]
        nova.run_nova_cmd(nova_cmd_just_run, controller, False)
        result = nova.run_nova_cmd(nova_cmd_w...

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.