glance was not installed properly

[Aleksey Kasatkin]

Custom 5.1 ISO: http://jenkins-product.srt.mirantis.net:8080/view/custom_iso/job/custom_master_iso/55/

Public network is allocated on controller only.

Env: Ubuntu, non-HA, neutron-vlan, ceph, 1 ctrl + 1 comp + 2 ceph.

Deployment has failed:

"Deployment has failed. Error occurred while running method 'deploy'. Inspect Astute logs for the details "

From astute log:

2014-08-20 16:25:41 ERR

[439] Error running RPC method deploy: Disabling the upload of disk image because glance was not installed properly, trace: ["/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/post_deploy_actions/upload_cirros_image.rb:96:in `raise_cirros_error'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/post_deploy_actions/upload_cirros_image.rb:50:in `process'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/post_deploy_actions.rb:30:in `block in process'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/post_deploy_actions.rb:30:in `each'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/post_deploy_actions.rb:30:in `process'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/orchestrator.rb:45:in `deploy'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/dispatcher.rb:58:in `deploy'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/server.rb:136:in `dispatch_message'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/server.rb:99:in `block in dispatch'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/task_queue.rb:64:in `call'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/task_queue.rb:64:in `block in each'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/task_queue.rb:56:in `each'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/task_queue.rb:56:in `each'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/server.rb:97:in `each_with_index'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/server.rb:97:in `dispatch'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/server.rb:85:in `block in perform_main_job'"]

2014-08-20 16:25:41 ERR

[439] d04491ac-0c9c-404b-a7d8-7b5ee1c0738f: Disabling the upload of disk image because glance was not installed properly

2014-08-20 16:25:41 ERR

[439] d04491ac-0c9c-404b-a7d8-7b5ee1c0738f: cmd: . /root/openrc && /usr/bin/glance index
                                               mcollective error: d04491ac-0c9c-404b-a7d8-7b5ee1c0738f: MCollective agents '29' didn't respond within the allotted time.

2014-08-20 16:25:41 ERR

[439] MCollective agents '29' didn't respond within the allotted time.

[Dima Shulyak (dshulyak)]

Canyou ssh on controller and make:

source openrc && glance index

But actually it looks like same issue with MC_RETRIES

[Aleksey Kasatkin (alekseyk-ru)]

Isn't reproduced within 3 tries.

[Kamil Sambor (ksambor)]

I was able to reproduce the same error during deployment of env:
Ubuntu, HA, VLAN
nodes:
2 controllers, 1 compute
It was iso 467

[Vladimir Kuklin (vkuklin)]

This message is generic for failed deployment. We need to dig into logs to find what is the root cause

[Vladimir Kuklin (vkuklin)]

Kamil, your deployment failed due to insufficient memory amount. I am marking this bug as invalid as the first snapshot is related to custom ISO and the second is an environmental issue

[Tomasz 'Zen' Napierala (tzn)]

We should improve this message as it is misleading.

[Denis Egorenko (degorenko)]

Bug is reproduced on:
{"build_id": "2014-09-25_17-52-48", "ostf_sha": "a93cce54f41ee45301775d3d5e6861fd61388470", "build_number": "365", "auth_required": true, "api": "1.0", "nailgun_sha": "f8c268713a09be1b11ed2d8e30735420cc4833d4", "production": "docker", "fuelmain_sha": "540c6692d94f293e39159f4829be357a602c571d", "astute_sha": "a3e5da62af91b99f958ab958161d3dcec09c657b", "feature_groups": ["mirantis"], "release": "6.0", "release_versions": {"2014.2-6.0": {"VERSION": {"build_id": "2014-09-25_17-52-48", "ostf_sha": "a93cce54f41ee45301775d3d5e6861fd61388470", "build_number": "365", "api": "1.0", "nailgun_sha": "f8c268713a09be1b11ed2d8e30735420cc4833d4", "production": "docker", "fuelmain_sha": "540c6692d94f293e39159f4829be357a602c571d", "astute_sha": "a3e5da62af91b99f958ab958161d3dcec09c657b", "feature_groups": ["mirantis"], "release": "6.0", "fuellib_sha": "58cd7e6db388388155925f431b5c5828bba24fe3"}}}, "fuellib_sha": "58cd7e6db388388155925f431b5c5828bba24fe3"}

The problem is that new glanceclient hasn't deprecated command 'index'. So, '. /root/openrc && /usr/bin/glance index' return error code. Need to change this command to: '. /root/openrc && /usr/bin/glance image-list'

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/124330

[Matthew Mosesohn (raytrac3r)]

patch works, just waiting on merge

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/124330
Committed: https://git.openstack.org/cgit/stackforge/fuel-astute/commit/?id=65a5a0781d314bdcc090a61a599a8abf95989fc1
Submitter: Jenkins
Branch: master

commit 65a5a0781d314bdcc090a61a599a8abf95989fc1
Author: Denis Egorenko <email address hidden>
Date: Fri Sep 26 12:47:01 2014 +0400

    Replace deprecated glance command

    Replace deplrecated glance command, which was deleted from
    python-glanceclinet (v 0.14.1)

    Closes-bug: #1359282
    blueprint fuel-deploy-juno

    Change-Id: I34ef35abc6e630e73dda6740babe39222db40ebe

[Anastasia Palkina (apalkina)]

Verified on custom ISO with Juno #27

"build_id": "2014-10-13_16-21-11", "ostf_sha": "1795445bcd30ae26111e760b284078e5688f322f", "build_number": "27", "auth_required": true, "api": "1.0", "nailgun_sha": "e02c2356dc4912b56c0a95651c404dfdb6fa3eae", "production": "docker", "fuelmain_sha": "c50f4d4963ee08cf85055b8ea6cc5ac575f6c1d0", "astute_sha": "980456ba80b49a7e3f237a299e52d4ab214f6f72", "feature_groups": ["mirantis"], "release": "6.0", "release_versions": {"2014.2-6.0": {"VERSION": {"build_id": "2014-10-13_16-21-11", "ostf_sha": "1795445bcd30ae26111e760b284078e5688f322f", "build_number": "27", "api": "1.0", "nailgun_sha": "e02c2356dc4912b56c0a95651c404dfdb6fa3eae", "production": "docker", "fuelmain_sha": "c50f4d4963ee08cf85055b8ea6cc5ac575f6c1d0", "astute_sha": "980456ba80b49a7e3f237a299e52d4ab214f6f72", "feature_groups": ["mirantis"], "release": "6.0", "fuellib_sha": "0a805160d25b8074f0969dd6b70e29d6b3e93928"}}}, "fuellib_sha": "0a805160d25b8074f0969dd6b70e29d6b3e93928"

[Kyrylo Romanenko (kromanenko)]

Same errors in Astute logs of controller node:
2015-03-12 08:45:05 ERR
[399] Error running RPC method deploy: Disabling the upload of disk image because glance was not installed properly, trace:
["/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/post_deployment_actions/upload_cirros_image.rb:109:in `raise_cirros_error'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/post_deployment_actions/upload_cirros_image.rb:62:in `process'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/deploy_actions.rb:25:in `block in process'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/deploy_actions.rb:25:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/deploy_actions.rb:25:in `process'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/deployment_engine.rb:95:in `deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/orchestrator.rb:280:in `deploy_cluster'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/orchestrator.rb:35:in `deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/dispatcher.rb:59:in `deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/server.rb:142:in `dispatch_message'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/server.rb:103:in `block in dispatch'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/task_queue.rb:64:in `call'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/task_queue.rb:64:in `block in each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/server.rb:101:in `each_with_index'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/server.rb:101:in `dispatch'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/server.rb:85:in `block in perform_main_job'"]
2015-03-12 08:45:05 ERR
[399] 6d351418-a04e-4ea7-b769-be0ec5d4bd5e: Disabling the upload of disk image because glance was not installed properly

MOS 6.0.1:

{"build_id": "2015-03-11_12-48-06", "ostf_sha": "e15d4ebc7da62cd6793e1fcd7ce9e8f2504f788e", "build_number": "129", "auth_required": true, "api": "1.0", "nailgun_sha": "9f95a54126e340d78eda85a2226b0090db2adcbb", "production": "docker", "fuelmain_sha": "059522f2b59cfe7a2dc4d301281b142e4fe1618f", "astute_sha": "f7cda2171b0b677dfaeb59693d980a2d3ee4c3e0", "feature_groups": ["mirantis"], "release": "6.0.1", "release_versions": {"2014.2.2-6.0.1": {"VERSION": {"build_id": "2015-03-11_12-48-06", "ostf_sha": "e15d4ebc7da62cd6793e1fcd7ce9e8f2504f788e", "build_number": "129", "api": "1.0", "nailgun_sha": "9f95a54126e340d78eda85a2226b0090db2adcbb", "production": "docker", "fuelmain_sha": "059522f2b59cfe7a2dc4d301281b142e4fe1618f", "astute_sha": "f7cda2171b0b677dfaeb59693d980a2d3ee4c3e0", "feature_groups": ["mirantis"], "release": "6.0.1", "fuellib_sha": "f3e355fdd9d4996da4c092c68cf4a8ce58547b54"}}}, "fuellib_sha": "f3e355fdd9d4996da4c092c68cf4a8ce58547b54"}

2 controllers
1 compute
1 cinder lvm
1 telemetry
Neu...

[Bartłomiej Piotrowski (bpiotrowski)]

I can confirm it's still broken.

{"build_id": "2015-04-09_07-56-54", "ostf_sha": "88af8b5666599bc9031ca233a45b1316acb1f035", "build_number": "49", "auth_required": true, "api": "1.0", "nailgun_sha": "5d15f977ac1e4e034bf246cca0c7995cbd1308c8", "production": "docker", "fuelmain_sha": "9295506b6eeb760ce32643be2a2f22ee2b104d51", "astute_sha": "5c79ba7feb05ea944bbfead181160f74d7572065", "feature_groups": ["mirantis"], "release": "6.0.1", "release_versions": {"2014.2.2-6.0.1": {"VERSION": {"build_id": "2015-04-09_07-56-54", "ostf_sha": "88af8b5666599bc9031ca233a45b1316acb1f035", "build_number": "49", "api": "1.0", "nailgun_sha": "5d15f977ac1e4e034bf246cca0c7995cbd1308c8", "production": "docker", "fuelmain_sha": "9295506b6eeb760ce32643be2a2f22ee2b104d51", "astute_sha": "5c79ba7feb05ea944bbfead181160f74d7572065", "feature_groups": ["mirantis"], "release": "6.0.1", "fuellib_sha": "368a3ef05e110bb2724e229fcd416363865f4c86"}}}, "fuellib_sha": "368a3ef05e110bb2724e229fcd416363865f4c86"}

Ask Yegor Kotko for diagnostic snapshot.

[Bogdan Dobrelya (bogdando)]

Egor please elaborate reproduction steps and env details

[ruhe (ruhe)]

Re-assigned to Egor Kotko since Bogdan requested additional information from Egor.

[Tomasz 'Zen' Napierala (tzn)]

Egor,

This sits in Incomplete state too long, can we get some answer here please?

[Egor Kotko (ykotko)]

Could not reproduce on:
{"build_id": "2015-04-21_20-49-31", "ostf_sha": "88af8b5666599bc9031ca233a45b1316acb1f035", "build_number": "208", "auth_required": true, "api": "1.0", "nailgun_sha": "7a3c2af98bf36b942e1baa60362510c3b701f702", "production": "docker", "fuelmain_sha": "8bf06bf4ba9b2a7d5e20062d227467f0d923a0bb", "astute_sha": "5c79ba7feb05ea944bbfead181160f74d7572065", "feature_groups": ["mirantis"], "release": "6.0.1", "release_versions": {"2014.2.2-6.0.1": {"VERSION": {"build_id": "2015-04-21_20-49-31", "ostf_sha": "88af8b5666599bc9031ca233a45b1316acb1f035", "build_number": "208", "api": "1.0", "nailgun_sha": "7a3c2af98bf36b942e1baa60362510c3b701f702", "production": "docker", "fuelmain_sha": "8bf06bf4ba9b2a7d5e20062d227467f0d923a0bb", "astute_sha": "5c79ba7feb05ea944bbfead181160f74d7572065", "feature_groups": ["mirantis"], "release": "6.0.1", "fuellib_sha": "4a04e33685d0b625563d2a10b72e2bc6ec4f44c2"}}}, "fuellib_sha": "4a04e33685d0b625563d2a10b72e2bc6ec4f44c2"}

[Aleksey Kasatkin (alekseyk-ru)]

It is hard to reproduce. I could not get exact steps to reproduce it for sure.

[Egor Kotko (ykotko)]

Could not reproduce on:
{"build_id": "2015-04-13_22-54-31", "ostf_sha": "c2a76a60ec4ebbd78e508216c2e12787bf25e423", "build_number": "310", "release_versions": {"2014.2-6.1": {"VERSION": {"build_id": "2015-04-13_22-54-31", "ostf_sha": "c2a76a60ec4ebbd78e508216c2e12787bf25e423", "build_number": "310", "api": "1.0", "nailgun_sha": "d22c074dec091e5ddd8ea3003c37665058303cd5", "openstack_version": "2014.2-6.1", "production": "docker", "python-fuelclient_sha": "9208ff4a08dcb674ce2df132399a5aa3ddfac21c", "astute_sha": "d96a80b63198a578b2c159edbd76048819039eb0", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "335d3ed09ed79bd37e1f7a90442c4831c8845582", "fuellib_sha": "8b80657e9ceed8d59c2dff1c11e1481c7e69380e"}}}, "auth_required": true, "api": "1.0", "nailgun_sha": "d22c074dec091e5ddd8ea3003c37665058303cd5", "openstack_version": "2014.2-6.1", "production": "docker", "python-fuelclient_sha": "9208ff4a08dcb674ce2df132399a5aa3ddfac21c", "astute_sha": "d96a80b63198a578b2c159edbd76048819039eb0", "feature_groups": ["mirantis", "experimental"], "release": "6.1", "fuelmain_sha": "335d3ed09ed79bd37e1f7a90442c4831c8845582", "fuellib_sha": "8b80657e9ceed8d59c2dff1c11e1481c7e69380e"}

[Egor Kotko (ykotko)]

Agree on 100%,
sorry for delay - was working with urgent task.

On Tue, Apr 21, 2015 at 3:02 PM, Tomasz 'Zen' Napierala <
<email address hidden>> wrote:

> Egor,
>
> This sits in Incomplete state too long, can we get some answer here
> please?
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/1359282
>
> Title:
> glance was not installed properly
>
> Status in Fuel: OpenStack installer that works:
> Incomplete
> Status in Fuel for OpenStack 6.0.x series:
> Incomplete
> Status in Fuel for OpenStack 6.1.x series:
> Incomplete
>
> Bug description:
> Custom 5.1 ISO: http://jenkins-
> product.srt.mirantis.net:8080/view/custom_iso/job/custom_master_iso/55/
>
> Public network is allocated on controller only.
>
> Env: Ubuntu, non-HA, neutron-vlan, ceph, 1 ctrl + 1 comp + 2 ceph.
>
> Deployment has failed:
>
> "Deployment has failed. Error occurred while running method 'deploy'.
> Inspect Astute logs for the details "
>
> From astute log:
>
> 2014-08-20 16:25:41 ERR
>
> [439] Error running RPC method deploy: Disabling the upload of disk
> image because glance was not installed properly, trace:
>
> ["/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/post_deploy_actions/upload_cirros_image.rb:96:in
> `raise_cirros_error'",
>
> "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/post_deploy_actions/upload_cirros_image.rb:50:in
> `process'",
>
> "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/post_deploy_actions.rb:30:in
> `block in process'",
>
> "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/post_deploy_actions.rb:30:in
> `each'",
>
> "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/post_deploy_actions.rb:30:in
> `process'",
>
> "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/orchestrator.rb:45:in
> `deploy'",
>
> "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/dispatcher.rb:58:in
> `deploy'",
>
> "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/server.rb:136:in
> `dispatch_message'",
>
> "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/server.rb:99:in
> `block in dispatch'",
>
> "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/task_queue.rb:64:in
> `call'",
>
> "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/task_queue.rb:64:in
> `block in each'",
>
> "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/task_queue.rb:56:in
> `each'",
>
> "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/task_queue.rb:56:in
> `each'",
>
> "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/server.rb:97:in
> `each_with_index'",
>
> "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/server.rb:97:in
> `dispatch'",
>
> "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/server.rb:85:in
> `block in perform_main_job'"]
>
> 2014-08-20 16:25:41 ERR
>
> [439] d04491ac-0c9c-404b-a7d8-7b5ee1c0738f: Disabling the upload of
> disk image because glance was not installed properly
>
> 2014-08-20 16:25:41 ERR
>
> [439] d04491ac-0c9c-404b-a7d8-7b5ee1c0738f: cmd: ...

[Alexey Stupnikov (astupnikov)]

MOS 6.0 are no longer supported, moving to Won't Fix.