Fuel for OpenStack

  1. Bug #1503657
  2. Activity log

Activity log for bug #1503657

Date Who What changed Old value New value Message
2015-10-07 10:57:14 Alexander Kurenyshev bug added bug
2015-10-07 10:57:14 Alexander Kurenyshev attachment added fail_error_ceph_ha_one_controller_compact-fuel-snapshot-2015-10-06_22-30-50.tar.xz https://bugs.launchpad.net/bugs/1503657/+attachment/4487173/+files/fail_error_ceph_ha_one_controller_compact-fuel-snapshot-2015-10-06_22-30-50.tar.xz
2015-10-07 12:09:37 Alexander Kurenyshev attachment added fail_error_ceph_ha_one_controller_with_cinder-fuel-snapshot-2015-10-06_22-46-41.tar.xz https://bugs.launchpad.net/fuel/+bug/1503657/+attachment/4487253/+files/fail_error_ceph_ha_one_controller_with_cinder-fuel-snapshot-2015-10-06_22-46-41.tar.xz
2015-10-07 13:49:15 Alexander Kurenyshev attachment added fail_error_ceph_ha-fuel-snapshot-2015-10-06_23-31-36.tar.xz https://bugs.launchpad.net/fuel/+bug/1503657/+attachment/4487286/+files/fail_error_ceph_ha-fuel-snapshot-2015-10-06_23-31-36.tar.xz
2015-10-07 13:50:05 Alexander Kurenyshev fuel: importance Undecided Critical
2015-10-07 14:24:24 Nastya Urlapova fuel: status New Confirmed
2015-10-07 15:34:36 Ivan Berezovskiy fuel: assignee Fuel Library Team (fuel-library) MOS Puppet Team (mos-puppet)
2015-10-07 16:48:54 Alexander Kurenyshev description CI System Test job [1] was filed with "AssertionError: Task 'deploy' has incorrect status. error != ready, 'Deployment has failed. Method granular_deploy. Deployment failed on nodes 1. Inspect Astute logs for the details'" At the Astute log: 2015-10-06 22:30:46 ERR [642] Error running RPC method granular_deploy: Deployment failed on nodes 1, trace: ["/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/deployment_engine.rb:68:in `block (3 levels) in deploy'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/deployment_engine.rb:51:in `each_slice'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/deployment_engine.rb:51:in `block (2 levels) in deploy'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/deployment_engine.rb:49:in `each'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/deployment_engine.rb:49:in `block in deploy'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/deployment_engine.rb:45:in `each'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/deployment_engine.rb:45:in `deploy'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/orchestrator.rb:179:in `deploy_cluster'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/orchestrator.rb:45:in `granular_deploy'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/server/dispatcher.rb:91:in `granular_deploy'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/server/server.rb:146:in `dispatch_message'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/server/server.rb:107:in `block in dispatch'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/server/task_queue.rb:64:in `call'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/server/task_queue.rb:64:in `block in each'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/server/task_queue.rb:56:in `each'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/server/task_queue.rb:56:in `each'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/server/server.rb:105:in `each_with_index'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/server/server.rb:105:in `dispatch'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/server/server.rb:89:in `block in perform_main_job'"] At the puppet log from node 1: Execution of '/usr/bin/nova network-create novanetwork --fixed-range-v4 10.0.0.0/16 --dns1 8.8.4.4 --dns2 8.8.8.8' returned 1: ERROR (ClientException): The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-09ed79c2-395f-4e6e-9071-4caeab952d77) When I manually executed this command after reverting environment I got the same error 500 and timeout for waiting message at the nova-all.log: <179>Oct 7 10:39:17 node-1 nova-api 2015-10-07 10:39:17.380 12188 ERROR nova.api.openstack [req-2fc43ae5-b105-45b8-8109-fd2754e35143 ea5cb4d6f1bd40b8a51b575cfbf9e600 b81a978ecfea42568010d2cbc87a6a60 - - -] Caug ht error: Timed out waiting for a reply to message ID aa435b402b7247c3b682cbde2f509855 For this test we have next steps: 1. Create cluster 2. Add 1 node with controller and ceph OSD roles 3. Add 2 node with compute and ceph OSD roles 4. Deploy the cluster [1] https://product-ci.infra.mirantis.net/job/8.0.system_test.ubuntu.ceph_ha_one_controller/12/console Fuel used: api: '1.0' astute_sha: 0f753467a3f16e4d46e7e9f1979905fb178e4d5b auth_required: true build_id: '126' build_number: '126' feature_groups: - mirantis fuel-agent_sha: e881f0dabd09af4be4f3e22768b02fe76278e20e fuel-library_sha: 37734918592cef7d355f357d14cbf960540e09cd fuel-nailgun-agent_sha: d66f188a1832a9c23b04884a14ef00fc5605ec6d fuel-nailgun_sha: 70d8b7e80573728e04ac5478c112850afcfa9802 fuel-ostf_sha: 41aa5059243cbb25d7a80b97f8e1060a502b99dd fuelmain_sha: 51614465980e5f62a5796779d3f6c3305c1d5739 openstack_version: 2015.1.0-8.0 production: docker python-fuelclient_sha: 56fbd6bad7f60f0944b3845c2db14d0b8cabd4d3 release: '8.0' release_versions: 2015.1.0-8.0: VERSION: api: '1.0' astute_sha: 0f753467a3f16e4d46e7e9f1979905fb178e4d5b build_id: '126' build_number: '126' feature_groups: - mirantis fuel-agent_sha: e881f0dabd09af4be4f3e22768b02fe76278e20e fuel-library_sha: 37734918592cef7d355f357d14cbf960540e09cd fuel-nailgun-agent_sha: d66f188a1832a9c23b04884a14ef00fc5605ec6d fuel-nailgun_sha: 70d8b7e80573728e04ac5478c112850afcfa9802 fuel-ostf_sha: 41aa5059243cbb25d7a80b97f8e1060a502b99dd fuelmain_sha: 51614465980e5f62a5796779d3f6c3305c1d5739 openstack_version: 2015.1.0-8.0 production: docker python-fuelclient_sha: 56fbd6bad7f60f0944b3845c2db14d0b8cabd4d3 release: '8.0' Let's modify description and say in other words. The main point here is that fuel should correctly process situations when network provider is not defined. We have a variable that is used for enable/disable Neutron at our ceph tests. When variable is not set there is no word about net-provider at cluster config we try to upload. And fuel should use neutron. But at some reasons it calls nova. This situations we can see at logs below. CI System Test job [1] was filed with "AssertionError: Task 'deploy' has incorrect status. error != ready, 'Deployment has failed. Method granular_deploy. Deployment failed on nodes 1. Inspect Astute logs for the details'" At the Astute log: 2015-10-06 22:30:46 ERR [642] Error running RPC method granular_deploy: Deployment failed on nodes 1, trace: ["/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/deployment_engine.rb:68:in `block (3 levels) in deploy'",  "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/deployment_engine.rb:51:in `each_slice'",  "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/deployment_engine.rb:51:in `block (2 levels) in deploy'",  "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/deployment_engine.rb:49:in `each'",  "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/deployment_engine.rb:49:in `block in deploy'",  "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/deployment_engine.rb:45:in `each'",  "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/deployment_engine.rb:45:in `deploy'",  "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/orchestrator.rb:179:in `deploy_cluster'",  "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/orchestrator.rb:45:in `granular_deploy'",  "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/server/dispatcher.rb:91:in `granular_deploy'",  "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/server/server.rb:146:in `dispatch_message'",  "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/server/server.rb:107:in `block in dispatch'",  "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/server/task_queue.rb:64:in `call'",  "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/server/task_queue.rb:64:in `block in each'",  "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/server/task_queue.rb:56:in `each'",  "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/server/task_queue.rb:56:in `each'",  "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/server/server.rb:105:in `each_with_index'",  "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/server/server.rb:105:in `dispatch'",  "/usr/lib64/ruby/gems/2.1.0/gems/astute-8.0.0/lib/astute/server/server.rb:89:in `block in perform_main_job'"] At the puppet log from node 1: Execution of '/usr/bin/nova network-create novanetwork --fixed-range-v4 10.0.0.0/16 --dns1 8.8.4.4 --dns2 8.8.8.8' returned 1: ERROR (ClientException): The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-09ed79c2-395f-4e6e-9071-4caeab952d77) When I manually executed this command after reverting environment I got the same error 500 and timeout for waiting message at the nova-all.log: <179>Oct 7 10:39:17 node-1 nova-api 2015-10-07 10:39:17.380 12188 ERROR nova.api.openstack [req-2fc43ae5-b105-45b8-8109-fd2754e35143 ea5cb4d6f1bd40b8a51b575cfbf9e600 b81a978ecfea42568010d2cbc87a6a60 - - -] Caug ht error: Timed out waiting for a reply to message ID aa435b402b7247c3b682cbde2f509855 For this test we have next steps: 1. Create cluster 2. Add 1 node with controller and ceph OSD roles 3. Add 2 node with compute and ceph OSD roles 4. Deploy the cluster [1] https://product-ci.infra.mirantis.net/job/8.0.system_test.ubuntu.ceph_ha_one_controller/12/console Fuel used: api: '1.0' astute_sha: 0f753467a3f16e4d46e7e9f1979905fb178e4d5b auth_required: true build_id: '126' build_number: '126' feature_groups: - mirantis fuel-agent_sha: e881f0dabd09af4be4f3e22768b02fe76278e20e fuel-library_sha: 37734918592cef7d355f357d14cbf960540e09cd fuel-nailgun-agent_sha: d66f188a1832a9c23b04884a14ef00fc5605ec6d fuel-nailgun_sha: 70d8b7e80573728e04ac5478c112850afcfa9802 fuel-ostf_sha: 41aa5059243cbb25d7a80b97f8e1060a502b99dd fuelmain_sha: 51614465980e5f62a5796779d3f6c3305c1d5739 openstack_version: 2015.1.0-8.0 production: docker python-fuelclient_sha: 56fbd6bad7f60f0944b3845c2db14d0b8cabd4d3 release: '8.0' release_versions:   2015.1.0-8.0:     VERSION:       api: '1.0'       astute_sha: 0f753467a3f16e4d46e7e9f1979905fb178e4d5b       build_id: '126'       build_number: '126'       feature_groups:       - mirantis       fuel-agent_sha: e881f0dabd09af4be4f3e22768b02fe76278e20e       fuel-library_sha: 37734918592cef7d355f357d14cbf960540e09cd       fuel-nailgun-agent_sha: d66f188a1832a9c23b04884a14ef00fc5605ec6d       fuel-nailgun_sha: 70d8b7e80573728e04ac5478c112850afcfa9802       fuel-ostf_sha: 41aa5059243cbb25d7a80b97f8e1060a502b99dd       fuelmain_sha: 51614465980e5f62a5796779d3f6c3305c1d5739       openstack_version: 2015.1.0-8.0       production: docker       python-fuelclient_sha: 56fbd6bad7f60f0944b3845c2db14d0b8cabd4d3       release: '8.0'
2015-10-07 16:50:32 Alexander Kurenyshev summary Command `nova network-create novanetwork` failed on the controller node with timeout. Deploy failed Fuel should correctly process situations when net-provider is not defined at the config
2015-10-07 16:52:55 Alexander Kurenyshev fuel: assignee MOS Puppet Team (mos-puppet) Fuel Library Team (fuel-library)
2015-10-07 17:01:01 Matthew Mosesohn fuel: assignee Fuel Library Team (fuel-library) Fuel Python Team (fuel-python)
2015-10-07 17:01:09 Matthew Mosesohn fuel: importance Critical High
2015-10-12 12:27:15 Dmitry Pyzhov fuel: assignee Fuel Python Team (fuel-python) Igor Kalnitsky (ikalnitsky)
2015-10-13 10:22:26 OpenStack Infra fuel: status Confirmed In Progress
2015-10-13 14:06:05 OpenStack Infra fuel: status In Progress Fix Committed
2015-10-22 03:11:24 Dmitry Pyzhov tags swarm-blocker area-python swarm-blocker
2016-02-02 10:33:41 Dmitriy Kruglov tags area-python swarm-blocker area-python on-verification swarm-blocker
2016-02-02 18:42:52 Dmitriy Kruglov fuel: status Fix Committed Fix Released
2016-02-02 18:43:03 Dmitriy Kruglov tags area-python on-verification swarm-blocker area-python swarm-blocker