Fuel for OpenStack

fuel-octane fails during upgrade of compute node

Bug #1501748 reported by Sergey Murashov
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
High
Sergey Murashov
Fuel for OpenStack 7.0-updates

Bug Description

fuel-octane fails during upgrade of compute node, after start of deployment with the following trace, it happens due to using wrong hostname:

2015-10-01 12:37:18 ERROR octane.util.subprocess sh[at node-3] stderr: ERROR (NotFound): Could not find binary nova-compute on host node-2. (HTTP 404) (Request-ID: req-71c0b093-a
9ed-4202-a4b0-4f9c44044f55)
2015-10-01 12:37:18 DEBUG paramiko.transport [chan 2] EOF received (2)
2015-10-01 12:37:18 INFO octane.util.subprocess Process sh finished with return value 1
2015-10-01 12:37:18 ERROR octane Command 'sh' returned non-zero exit status 1
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/cliff/app.py", line 280, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.6/site-packages/cliff/command.py", line 50, in run
    self.take_action(parsed_args)
  File "/root/fuel-octane/octane/commands/upgrade_node.py", line 102, in take_action
    network_template=parsed_args.template)
  File "/root/fuel-octane/octane/commands/upgrade_node.py", line 62, in upgrade_node
    call_handlers('postdeploy')
  File "/root/fuel-octane/octane/handlers/__init__.py", line 57, in call_method_on_all
    getattr(handler, method)()
  File "/root/fuel-octane/octane/handlers/upgrade/compute.py", line 41, in postdeploy
    node=controller,
  File "/root/fuel-octane/octane/util/ssh.py", line 174, in call
    return subprocess.call(cmd, popen_class=SSHPopen, **kwargs)
  File "/root/fuel-octane/octane/util/subprocess.py", line 200, in call
    return proc.communicate()
  File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
    self.gen.next()
  File "/root/fuel-octane/octane/util/subprocess.py", line 195, in popen
    raise CalledProcessError(rv, name)
CalledProcessError: Command 'sh' returned non-zero exit status 1

Sergey Murashov (smurashov)
Changed in fuel:
assignee: nobody → Fuel Octane Dev Team (fuel-octane)
Oleg S. Gelbukh (gelbuhos)
Changed in fuel:
status: New → Confirmed
importance: Undecided → High
milestone: none → 7.0-updates
Sergey Murashov (smurashov)
Changed in fuel:
assignee: Fuel Octane Dev Team (fuel-octane) → Sergey Murashov (smurashov)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-octane (master)

Fix proposed to branch: master
Review: https://review.openstack.org/231987

Changed in fuel:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-octane (stable/7.0)

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/231988

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-octane (stable/7.0)

Reviewed: https://review.openstack.org/231988
Committed: https://git.openstack.org/cgit/stackforge/fuel-octane/commit/?id=f793e64833c36b1165340a914d0f6eafb5a77c9b
Submitter: Jenkins
Branch: stable/7.0

commit f793e64833c36b1165340a914d0f6eafb5a77c9b
Author: smurashov <email address hidden>
Date: Wed Oct 7 16:19:51 2015 +0300

    Replace node-{id} to fqdn

    Change-Id: Id4faf74815009a455fa56d5f1ba380cf8c06d491
    Closes-Bug: #1501748

Oleg S. Gelbukh (gelbuhos)
Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-octane (master)

Fix proposed to branch: master
Review: https://review.openstack.org/246676

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-octane (master)
Download full text (11.6 KiB)

Reviewed: https://review.openstack.org/246676
Committed: https://git.openstack.org/cgit/openstack/fuel-octane/commit/?id=cae1d8af5d3ed7a6901fdb363f40e3bf7c37f26d
Submitter: Jenkins
Branch: master

commit e27cea930477dcaf5be29bfa4febd270dbe99e97
Author: Oleg Gelbukh <email address hidden>
Date: Thu Nov 12 15:15:57 2015 +0000

    Version bump to 1.0.0

    Change-Id: I51735d21337d51f52eead9625e2d9f3378c88799

commit 4d3c949e1f6c8319658ac713461e9f8f4a345eb4
Author: Oleg Gelbukh <email address hidden>
Date: Thu Nov 12 14:30:09 2015 +0000

    Revert "Set RPM version 0.2.0"

    This reverts commit 1c722afa0f78cfa3601f3a2b71085c6788052885.
    Version downgrade must be avoided.

    Change-Id: I6b3817ffa48ab321fe99bed7cb2df6a0941211ce

commit 1c722afa0f78cfa3601f3a2b71085c6788052885
Author: Oleg Gelbukh <email address hidden>
Date: Sun Nov 8 14:41:40 2015 +0000

    Set RPM version 0.2.0

    Change-Id: Ie0c75b979375e716c33082b9a4e83ffaffc7f42c

commit 3d06b9d8ddd3714166bdedeb91da5df6b8c9f7a6
Author: Valyavskiy Viacheslav <email address hidden>
Date: Sun Oct 4 03:14:37 2015 +0300

    Use more correct approach to detect resources status

    Closes-bug: 1499696
    Change-Id: Ied72bc68ba41e47334796f09b407ebd8a3fb7f40

commit c315735e681928648fd4549956c3621c4caba4f9
Author: Sergey Novikov <email address hidden>
Date: Thu Oct 29 18:57:21 2015 +0300

    Add handling of case when node may haven't full name in services data

    Change-Id: Icf8dc176fc74fc1c5ee888ddb920b1acc58c127d

commit cf83b87b846e6613dc996d8a8dc70723bcbe84d1
Author: Oleg Gelbukh <email address hidden>
Date: Tue Oct 27 13:57:04 2015 +0000

    Try to revert patches to Puppet modules every time

    If octane command was interrupted or failed before patches
    to puppet were reverted, the patch will be applied and repeated
    command will fail.

    Try to revert the patch every time before apply it and skip
    failures. Only actually apply the patch if revert is not assumed.

    Change-Id: I5719dba3c621f307fe5d6ae2356f15d38ee28ff9

commit a3226740bfacc15da032746d3093f77ecc0323ad
Author: Oleg Gelbukh <email address hidden>
Date: Fri Oct 23 12:48:27 2015 +0000

    Start corosync services after control plane switch for multiple controllers

    Before cluster is stopped, some corosync services are in 'Stop' status.
    Starting cluster doesn't bring them up, just restores state of the cluster.
    Explicitly start corosync services after the cluster is back.

    Related-bug: 1506398
    Change-Id: I38e90180f71e2786e18ffb67ec43a50aa2c5bee6

commit b3e496b43ecdd92a2a696147e2a487fcfe0b2c0d
Author: Oleg Gelbukh <email address hidden>
Date: Fri Oct 23 11:58:35 2015 +0000

    Delete shell scripts and unused shell libs

    As migration to Python is complete, remove unused shell scripts and libs

    Change-Id: I5cc13bebfd0423f73c46722d276807154f2b79e8

commit 6b0a7649cfcf546ffe410e62aaaaf72bec9e5c2d
Author: Oleg Gelbukh <email address hidden>
Date: Tue Oct 20 11:22:12 2015 +0000

    Stop corosync in upgrade seed during control plane switch

    Stop corosync cluster before switching to upgrade...

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-octane (master)

Change abandoned by Oleg Gelbukh (<email address hidden>) on branch: master
Review: https://review.openstack.org/231987

Revision history for this message
Khadeer (kmdkhadeer78) wrote :
Download full text (8.1 KiB)

Hi All

I have upgraded Fuel form 7.0 to 8.0 as per this Doc "http://docs.openstack.org/developer/fuel-docs/userdocs/fuel-install-guide/upgrade/upgrade-fuel.html"
Then Imported the backup successfully in New 8.0 Fuel Master. Then where i am not able to add are remove node from fuel 8.0. Error While add a new node "Error
list index out of range"

Then i Tried to Upgrade Environment from Kilo to Liberty i followed this DOC "https://docs.mirantis.com/openstack/fuel/fuel-7.0/operations.html#upgrade-environment-experimental"
and downloaded scripts form git clone "https://git.openstack.org/openstack/fuel-octane -b stable/8.0"
But got struck here tried different CMDs but not able to Upgrade env

octane upgrade-env 44
2016-05-17 00:40:18 INFO requests.packages.urllib3.connectionpool Starting new HTTP connection (1): 10.18.203.2
2016-05-17 00:40:18 INFO requests.packages.urllib3.connectionpool Starting new HTTP connection (1): 10.18.203.2
2016-05-17 00:40:18 INFO requests.packages.urllib3.connectionpool Starting new HTTP connection (1): 10.18.203.2
2016-05-17 00:40:18 INFO requests.packages.urllib3.connectionpool Starting new HTTP connection (1): 10.18.203.2
2016-05-17 00:40:19 INFO octane.util.env Cloning env 44 for release Liberty on Ubuntu 14.04
2016-05-17 00:40:19 INFO octane.util.subprocess Started process fuel2[28418]: fuel2 env clone -f json 44 1c47c3eb7a67402e894734a780f4f8c9 4
2016-05-17 00:40:19 ERROR octane.util.subprocess fuel2[28418] stderr: 400 Client Error: Bad Request
2016-05-17 00:40:19 INFO octane.util.subprocess Process fuel2 finished with return value 1
2016-05-17 00:40:19 ERROR octane Command 'fuel2' returned non-zero exit status 1

*************************************

[root@fuel bin]# octane upgrade-node --isolated 47 69
2016-05-17 00:14:24 INFO requests.packages.urllib3.connectionpool Starting new HTTP connection (1): 10.18.203.2
2016-05-17 00:14:24 INFO requests.packages.urllib3.connectionpool Starting new HTTP connection (1): 10.18.203.2
2016-05-17 00:14:24 INFO requests.packages.urllib3.connectionpool Starting new HTTP connection (1): 10.18.203.2
2016-05-17 00:14:24 INFO requests.packages.urllib3.connectionpool Starting new HTTP connection (1): 10.18.203.2
2016-05-17 00:14:24 INFO octane.util.subprocess Started process tar[26670]: dockerctl shell nailgun tar -xv --overwrite -f - -C /tmp
2016-05-17 00:14:24 INFO octane.util.subprocess tar[26670] stdout: nova.conf
2016-05-17 00:14:24 INFO octane.util.subprocess tar[26670] stdout: neutron-upgrade.patch
2016-05-17 00:14:24 INFO octane.util.subprocess tar[26670] stdout: timeout.patch
2016-05-17 00:14:24 INFO octane.util.subprocess tar[26670] stdout: neutron-template.tar
2016-05-17 00:14:24 INFO octane.util.subprocess tar[26670] stdout: nailgun-clone-ips.patch
2016-05-17 00:14:24 INFO octane.util.subprocess tar[26670] stdout: update_node_partition_info.py
2016-05-17 00:14:24 INFO octane.util.subprocess tar[26670] stdout: fuel_agent/patch
2016-05-17 00:14:24 INFO octane.util.subprocess tar[26670] stdout: puppet/revert.sh
2016-05-17 00:14:24 INFO octane.util.subprocess tar[26670] st...

Read more...

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.