ceph-deploy problem on controller upgrade 7.0 -> 8.0

Bug #1584658 reported by Sergey Abramov
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
High
Sergey Abramov
8.0.x
Fix Committed
High
Sergey Abramov

Bug Description

Detailed bug description:
I am trying to upgrade env from release 7.0 up to release 8.0.
Env contains 3 computes + ceph and 3 controllers.
1 controller upgraded in isolation successfuly.
2 other containers set in error state. One of them has an error:
`(/Stage[main]/Ceph::Conf/Exec[ceph-deploy config pull]/returns) change from notrun to 0 failed: ceph-deploy --overwrite-conf config pull node-5 returned 1 instead of one of [0]`
Posible reason is ' (/Stage[main]/Ceph::Conf/Exec[ceph-deploy config pull]/returns) ssh: connect to host node-5 port 22: No route to host'

After that I ran script on controller node and got that result:
root@node-5:~# ceph-deploy --overwrite-conf config pull node-5
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.20): /usr/bin/ceph-deploy --overwrite-conf config pull node-5
[ceph_deploy.config][DEBUG ] Checking node-5 for /etc/ceph/ceph.conf
[node-5][DEBUG ] connected to host: node-5
[node-5][DEBUG ] detect platform information from remote host
[node-5][DEBUG ] detect machine type
[node-5][DEBUG ] fetch remote file
[ceph_deploy.config][DEBUG ] Got /etc/ceph/ceph.conf from node-5

Steps to reproduce:
install fuel-70
create env and deploy it 3 controllers and 3 computes + ceph
make backup
install fuel-80
restore backup
octane upgrade-node --isolated $SEED_ID <ID>
octane upgrade-db $ORIG_ID $SEED_ID
octane upgrade-ceph $ORIG_ID $SEED_ID
octane upgrade-control $ORIG_ID $SEED_ID
octane upgrade-node $SEED_ID <ID1> <ID2>
Actual result:
Controller in error state
Expected results:
Controller in ready state
Additional information:
2016-05-20 11:00:43 ERR (/Stage[main]/Ceph::Conf/Exec[ceph-deploy config pull]/returns) change from notrun to 0 failed: ceph-deploy --overwrite-conf config pull node-5 returned 1 instead of one of [0]
2016-05-20 11:00:43 ERR /usr/bin/puppet:4:in `<main>'
2016-05-20 11:00:43 ERR /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:91:in `execute'
2016-05-20 11:00:43 ERR /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:137:in `run'
2016-05-20 11:00:43 ERR /usr/lib/ruby/vendor_ruby/puppet/application.rb:364:in `run'
2016-05-20 11:00:43 ERR /usr/lib/ruby/vendor_ruby/puppet/util.rb:478:in `exit_on_fail'
2016-05-20 11:00:43 ERR /usr/lib/ruby/vendor_ruby/puppet/application.rb:364:in `block in run'
2016-05-20 11:00:43 ERR /usr/lib/ruby/vendor_ruby/puppet/application.rb:470:in `plugin_hook'
2016-05-20 11:00:43 ERR /usr/lib/ruby/vendor_ruby/puppet/application.rb:364:in `block (2 levels) in run'
2016-05-20 11:00:43 ERR /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:146:in `run_command'
2016-05-20 11:00:43 ERR /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:218:in `main'
2016-05-20 11:00:43 ERR /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:268:in `apply_catalog'
2016-05-20 11:00:43 ERR /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:192:in `run'
2016-05-20 11:00:43 ERR /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:124:in `apply_catalog'
2016-05-20 11:00:43 ERR /usr/lib/ruby/vendor_ruby/puppet/util.rb:160:in `benchmark'
2016-05-20 11:00:43 ERR /usr/lib/ruby/1.9.1/benchmark.rb:295:in `realtime'
2016-05-20 11:00:43 ERR /usr/lib/ruby/vendor_ruby/puppet/util.rb:161:in `block in benchmark'
2016-05-20 11:00:43 ERR /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:125:in `block in apply_catalog'
2016-05-20 11:00:43 ERR /usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:163:in `apply'
2016-05-20 11:00:43 ERR /usr/lib/ruby/vendor_ruby/puppet/transaction/report.rb:108:in `as_logging_destination'
2016-05-20 11:00:43 ERR /usr/lib/ruby/vendor_ruby/puppet/util/log.rb:149:in `with_destination'
2016-05-20 11:00:43 ERR /usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:164:in `block in apply'
2016-05-20 11:00:43 ERR /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:108:in `evaluate'
2016-05-20 11:00:43 ERR /usr/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:118:in `traverse'
2016-05-20 11:00:43 ERR /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:117:in `block in evaluate'
2016-05-20 11:00:43 ERR /usr/lib/ruby/vendor_ruby/puppet/util.rb:326:in `thinmark'
2016-05-20 11:00:43 ERR /usr/lib/ruby/1.9.1/benchmark.rb:295:in `realtime'
2016-05-20 11:00:43 ERR /usr/lib/ruby/vendor_ruby/puppet/util.rb:327:in `block in thinmark'
2016-05-20 11:00:43 ERR /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:117:in `block (2 levels) in evaluate'
2016-05-20 11:00:43 ERR /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:117:in `call'
2016-05-20 11:00:43 ERR /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:187:in `eval_resource'
2016-05-20 11:00:43 ERR /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:174:in `apply'
2016-05-20 11:00:43 ERR /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:20:in `evaluate'
2016-05-20 11:00:43 ERR /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:88:in `perform_changes'
2016-05-20 11:00:43 ERR /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:88:in `each'
2016-05-20 11:00:43 ERR /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:89:in `block in perform_changes'
2016-05-20 11:00:43 ERR /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:128:in `sync_if_needed'
2016-05-20 11:00:43 ERR /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:191:in `sync'
2016-05-20 11:00:43 ERR /usr/lib/ruby/vendor_ruby/puppet/type/exec.rb:142:in `sync'
2016-05-20 11:00:43 ERR /usr/lib/ruby/vendor_ruby/puppet/util/errors.rb:97:in `fail'
2016-05-20 11:00:43 ERR ceph-deploy --overwrite-conf config pull node-5 returned 1 instead of one of [0]
2016-05-20 11:00:43 NOTICE (/Stage[main]/Ceph::Conf/Exec[ceph-deploy config pull]/returns) [ceph_deploy][ERROR ] GenericError: Failed to fetch config from 1 hosts
2016-05-20 11:00:43 NOTICE (/Stage[main]/Ceph::Conf/Exec[ceph-deploy config pull]/returns) [ceph_deploy.config][ERROR ] Unable to pull /etc/ceph/ceph.conf from node-5
2016-05-20 11:00:43 NOTICE (/Stage[main]/Ceph::Conf/Exec[ceph-deploy config pull]/returns) ssh: connect to host node-5 port 22: No route to host
2016-05-20 11:00:43 NOTICE (/Stage[main]/Ceph::Conf/Exec[ceph-deploy config pull]/returns) [ceph_deploy.config][DEBUG ] Checking node-5 for /etc/ceph/ceph.conf
2016-05-20 11:00:43 NOTICE (/Stage[main]/Ceph::Conf/Exec[ceph-deploy config pull]/returns) [ceph_deploy.cli][INFO ] Invoked (1.5.20): /usr/bin/ceph-deploy --overwrite-conf config pull node-5
2016-05-20 11:00:43 NOTICE (/Stage[main]/Ceph::Conf/Exec[ceph-deploy config pull]/returns) [ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf

Changed in fuel:
assignee: nobody → Fuel Octane (fuel-octane-team)
Changed in fuel:
milestone: none → 8.0-updates
importance: Undecided → High
status: New → Confirmed
tags: added: team-upgrades
Revision history for this message
Oleg S. Gelbukh (gelbuhos) wrote :

Sergey, could you please include the reproducibility of the issue in the description above?

Changed in fuel:
assignee: Fuel Octane (fuel-octane-team) → Sergey Abramov (sabramov)
status: Incomplete → In Progress
Changed in fuel:
status: Incomplete → In Progress
no longer affects: fuel/newton
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-octane (master)

Reviewed: https://review.openstack.org/334490
Committed: https://git.openstack.org/cgit/openstack/fuel-octane/commit/?id=9c1615b88deb240d8264519bc3f69bf27dd5382c
Submitter: Jenkins
Branch: master

commit 9c1615b88deb240d8264519bc3f69bf27dd5382c
Author: Sergey Abramov <email address hidden>
Date: Mon Jun 27 16:31:02 2016 +0300

    Allocated file names from patches in a separate function

    It returns files from all sent patch files with prefix removed from
    their paths

    Change-Id: I564cf909f653be6703931dbe942170431325ee87
    Partial-bug: 1584658
    Partial-bug: 1584041

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/322855
Committed: https://git.openstack.org/cgit/openstack/fuel-octane/commit/?id=fdafa2666dd2948175c8d5297727d42472dc6537
Submitter: Jenkins
Branch: master

commit fdafa2666dd2948175c8d5297727d42472dc6537
Author: Sergey Abramov <email address hidden>
Date: Mon May 30 17:40:57 2016 +0300

    Patch nailgun serializer on cluster upgrade handlers

    CopyKeys task won't be yielded if get_uids returns empty list.

    Change-Id: Ifd4589b6a0e9a8e8636fea9a36afa091b230ab11
    Closes-bug: 1584658
    Closes-bug: 1584041
    Related-bug: 1541256

Changed in fuel:
status: In Progress → Fix Committed
Dmitry Klenov (dklenov)
tags: added: area-python
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-octane (stable/8.0)

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/335848

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/335860

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-octane (stable/8.0)

Reviewed: https://review.openstack.org/335848
Committed: https://git.openstack.org/cgit/openstack/fuel-octane/commit/?id=1b27f81d7acf6c41485a5f53764d1bc365419840
Submitter: Jenkins
Branch: stable/8.0

commit 1b27f81d7acf6c41485a5f53764d1bc365419840
Author: Sergey Abramov <email address hidden>
Date: Mon Jun 27 16:31:02 2016 +0300

    Allocated file names from patches in a separate function

    It returns files from all sent patch files with prefix removed from
    their paths

    Change-Id: I564cf909f653be6703931dbe942170431325ee87
    Partial-bug: 1584658
    Partial-bug: 1584041

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/335860
Committed: https://git.openstack.org/cgit/openstack/fuel-octane/commit/?id=b991d9996f64301d3308a03dfb1bb2f00aec580e
Submitter: Jenkins
Branch: stable/8.0

commit b991d9996f64301d3308a03dfb1bb2f00aec580e
Author: Sergey Abramov <email address hidden>
Date: Mon May 30 17:40:57 2016 +0300

    Patch nailgun serializer on cluster upgrade handlers

    CopyKeys task won't be yielded if get_uids returns empty list.

    Change-Id: Ifd4589b6a0e9a8e8636fea9a36afa091b230ab11
    Closes-bug: 1584658
    Closes-bug: 1584041
    Related-bug: 1541256
    (cherry picked from commit fdafa2666dd2948175c8d5297727d42472dc6537)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-octane (master)

Fix proposed to branch: master
Review: https://review.openstack.org/351721

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-octane (master)

Change abandoned by Ilya Kharin (<email address hidden>) on branch: master
Review: https://review.openstack.org/351721
Reason: This patch have to go in stable/mitaka instead of master.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-octane (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/351728

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-octane (stable/mitaka)
Download full text (17.8 KiB)

Reviewed: https://review.openstack.org/351728
Committed: https://git.openstack.org/cgit/openstack/fuel-octane/commit/?id=3101bf302ea0f2d501fd42a6ae6046c3587ac093
Submitter: Jenkins
Branch: stable/mitaka

commit a05d2ecf13e15c12f10e1517fae425e583e0d714
Author: Nikita Zubkov <email address hidden>
Date: Thu Aug 4 17:51:56 2016 +0300

    fix ssh connection auth

    According to spec [1] service user must use key authentication​ for ssh

    [1] https://github.com/openstack/fuel-specs/blame/ba4504/specs/9.0/fuel-nonroot-openstack-nodes.rst#L61-L64

    Change-Id: I461f659d4f0985283a43ec6a702bdd78af91a072

commit 7140398e501879a4e1f5605f3d1cc99195464850
Author: Sergey Abramov <email address hidden>
Date: Tue Aug 2 12:12:57 2016 +0300

    Setup upgrade levels for each fuel version

    Setup upgrade levels block in nova.conf for each fuel version,
    this is required for backward compatibility on RPC.

    Change-Id: I6ccbc34457d11616b015cdb1e46e733b0b49ce58

commit 9e2a419ac79ad591faee9afe16419bf4497eacad
Author: Vladimir Khlyunev <email address hidden>
Date: Mon Aug 1 21:31:56 2016 +0300

    Fix several typos

    Copypasting is evil, lets remove typos

    Change-Id: Icf4bec1a3de954911a469fa98380af2b6175da08

commit 013a7ded2275245a38ee58289dac2bc8b90fa37e
Author: Sergey Abramov <email address hidden>
Date: Wed Jul 20 16:05:10 2016 +0300

    Host evacuation using nova

    Evacuate instance from compute node using
    nova host-evacuate-live except host_evacuation.sh script

    Change-Id: I01a04cfa1ed2aafce7987e523e6b589dea21fbb0

commit 00a1d0dce1f7ae5db6706a59353403516f7913dc
Author: Sergey Abramov <email address hidden>
Date: Tue Jul 19 17:10:44 2016 +0300

    Change shutoff vm command

    * run nova commands using nova util function
    * host filter using get nova node handle method(
        hostname for fuel < 6.1 and fqdn for newer version)

    Change-Id: Ib5c373714e359982b13910a225787167db7b35e9

commit 49357c98313c5f551b13d0877581c5fb5e108db1
Author: Nikita Zubkov <email address hidden>
Date: Mon Jun 27 18:40:13 2016 +0300

    Add `stderr_log_level` parameter to popen

    Now posible to set logging level in popen for logging stderr

    fuel-bootstrap build comand stderr now logged with INFO level

    Change-Id: Ibaa2be54c68ce7a428e07052c917f54e99b95a5e

commit fb582b271dab4bdd10d27149a77c0e1f6a580b9e
Author: Sergey Abramov <email address hidden>
Date: Thu Jul 21 17:56:41 2016 +0300

    Add run_nova_cmd function

    This is required for running nova command on contoller without
    manipulating manipulating with command.

    It makes code easier and more readable

    Usage:

        from fuelclient.objects import environment as environment_obj

        from octane.util import env
        from octane.util import nova

        env_id = ...
        my_env = environment_obj.Environment(env_id)
        controller = env.get_one_controller(my_env)
        nova_cmd_just_run = [...]
        nova_cmd_with_output = [...]
        nova.run_nova_cmd(nova_cmd_just_run, controller, False)
        result = nova.run_nova_cmd(nova_cmd_w...

tags: added: in-stable-mitaka
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.