Error: 403 forbidden trying to download snapshot

[Andrey Sledzinskiy]

http://jenkins-product.srt.mirantis.net:8080/view/0_0_swarm/job/master_fuelmain.system_test.centos.thread_2/70/console

In our last system tests run we have problem with saving logs:

2014-05-22 19:27:09,360 - INFO decorators.py:30 -- Saving logs to "/home/jenkins/workspace/master_fuelmain.system_test.centos.thread_2/logs/fail_deploy_flat_stop_reset_on_provisioning-2014_05_23__00_27_09.tar.gz" file
2014-05-22 19:27:09,405 - ERROR decorators.py:37 -- HTTP Error 403: Forbidden

[Andrey Sledzinskiy (asledzinskiy)]

Issue isn't in system tests. It's problem of downloading snapshot:
1. Open support page
2. Generate diagnostic snapshot
3. Download snapshot

Actual - get 403 error

reproduced on {"build_id": "2014-05-23_03-53-39", "mirantis": "yes", "build_number": "19", "ostf_sha": "5c479f04c35127576d35526650ec83b104f9a33d", "nailgun_sha": "bd09f89ef56176f64ad5decd4128933c96cb20f4", "production": "docker", "api": "1.0", "fuelmain_sha": "db2d153e62cb2b3034d33359d7e3db9d4742c811", "astute_sha": "9a0d86918724c1153b5f70bdae008dea8572fd3e", "release": "5.0", "fuellib_sha": "2ed4fbe1e04b85e83f1010ca23be7f5da34bd492"}

[Sergii Golovatiuk (sgolovatiuk)]

Same issue on release candidate 2

[Matthew Mosesohn (raytrac3r)]

Checking in here. Looks like file permissions are all in order and nothing is unreadable.
Diagnostic snapshot link points to 10.20.0.2:8000/dump/ and not a specific filename. It looks like an internal issue to shotgun or nailgun's read of the "last" file for building the URL.

[Evgeniy L (rustyrobot)]

Something wrong with execute_shell_command in mcollective, it doesn't return path to the snapshot, as result url for downloading looks like
http://10.20.0.2:8000/dump/

[Evgeniy L (rustyrobot)]

Yep, mcollective plugin is broken, it doesn't return stdout.

http://paste.openstack.org/show/81215/

[Evgeniy L (rustyrobot)]

We must revert this patch https://review.openstack.org/#/c/94164/
It doesn't solve descried problem, also execute_shell_command worked fine for almost year I don't think that it we needed to change it right before release.

[Evgeniy L (rustyrobot)]

Here I reverted this patch

https://review.openstack.org/#/c/95149/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/95155

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/5.0
Review: https://review.openstack.org/95169

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/95155
Committed: https://git.openstack.org/cgit/stackforge/fuel-astute/commit/?id=74b5122a58738be4f638b84a6a60d6edb892d289
Submitter: Jenkins
Branch: master

commit 74b5122a58738be4f638b84a6a60d6edb892d289
Author: Vladimir Sharshov <email address hidden>
Date: Fri May 23 16:58:42 2014 +0400

    Fix problem with empty stdout/stderr answer for shell magent

    Change-Id: I76e2c984bf86aa0df3ec1762df52cb1533ad7665
    Closes-Bug: #1322475

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/95169
Committed: https://git.openstack.org/cgit/stackforge/fuel-astute/commit/?id=a7eac46348dc77fc2723c6fcc3dbc66cc1a83152
Submitter: Jenkins
Branch: stable/5.0

commit a7eac46348dc77fc2723c6fcc3dbc66cc1a83152
Author: Vladimir Sharshov <email address hidden>
Date: Fri May 23 16:58:42 2014 +0400

    Fix problem with empty stdout/stderr answer for shell magent

    Change-Id: I76e2c984bf86aa0df3ec1762df52cb1533ad7665
    Closes-Bug: #1322475

[Aleksey Kasatkin (alekseyk-ru)]

Reproduced on #221.

Here is snapshot taken after error message is shown.
(It was received with a help of ssh)

[Aleksey Kasatkin (alekseyk-ru)]

I.e. snapshot was built correctly but download link was broken - "http://10.20.0.2:8000/dump/".

[Vladimir Sharshov (vsharshov)]

Problem in the old package: ruby21-nailgun-mcagents for CentOS. Thanks for report.

[Aleksey Kasatkin (alekseyk-ru)]

Same for #5.0-22

[Mike Scherbakov (mihgen)]

According to latest system tests: http://jenkins-product.srt.mirantis.net:8080/view/5.0_swarm/job/5.0_fuelmain.system_test.centos.services_simple/5/ , issue was fixed, starting from ISO 5.0 RC4 (#24).
The reason, why issue was reproduced, was due to OSCI's lack of rebuild job for mcagents on stable/5.0 branch. It rebuilt mcagents fine for master, but job for stable/5.0 was not created to do so, according to Roman Vyalov.

[Nastya Urlapova (aurlapova)]

{
build_id: "2014-05-26_18-06-28",
mirantis: "yes",
build_number: "24",
ostf_sha: "89bbddb78132e2997d82adc5ae5db9dcb7a35bcd",
nailgun_sha: "bd09f89ef56176f64ad5decd4128933c96cb20f4",
production: "docker",
api: "1.0",
fuelmain_sha: "d7f86968880a484d51f99a9fc439ef21139ea0b0",
astute_sha: "a7eac46348dc77fc2723c6fcc3dbc66cc1a83152",
release: "5.0",
fuellib_sha: "2f79c0415159651fc1978d99bd791079d1ae4a06"
}

[Aleksey Kasatkin (alekseyk-ru)]

NOT reproduced on ISO 5.0-26.

[Kirill Omelchenko (komelchenko)]

Doesn't reproduce.
{

    "build_id": "2014-09-09_00-01-11",
    "ostf_sha": "f7b5d9d0d1cfaba5f1fe1e2c634493e92bce11db",
    "build_number": "505",
    "auth_required": true,
    "api": "1.0",
    "nailgun_sha": "7683df5722975c1cae48a1a3efdad872b4aaace6",
    "production": "docker",
    "fuelmain_sha": "6cdd8c3deaa5e52806a5c75c4177f3b41d157a22",
    "astute_sha": "b622d9b36dbdd1e03b282b9ee5b7435ba649e711",
    "feature_groups": [
        "mirantis"
    ],
    "release": "5.1",
    "fuellib_sha": "203ef3179007cffe3236032e61ecbaf1cd20605f"

}