check-grenade-dsvm marked as FAILED - n-api/g-api Logs have errors

[Davanum Srinivas (DIMS)]

Example:
http://logs.openstack.org/81/61281/1/check/check-grenade-dsvm/f42b658/console.html

2013-12-11 00:14:33.892 | Log File: g-api
2013-12-11 00:14:33.893 | 2013-12-11 00:04:32.459 9398 ERROR glance.api.v1.upload_utils [a954dd77-c926-4ef5-916c-0589e852bb1b 4c3bf2863784478e8fc3dec275a7bdef 4af50376a7f44390b0d5790b0f3aa1f1] Received HTTP error while uploading image 88236e20-ced9-4868-b9bb-570d97edc446
2013-12-11 00:14:33.893 |
2013-12-11 00:14:33.893 | 2013-12-11 00:04:32.472 9398 ERROR glance.api.v1.upload_utils [a954dd77-c926-4ef5-916c-0589e852bb1b 4c3bf2863784478e8fc3dec275a7bdef 4af50376a7f44390b0d5790b0f3aa1f1] Unable to kill image 88236e20-ced9-4868-b9bb-570d97edc446:
2013-12-11 00:14:33.893 |
2013-12-11 00:14:34.044 | Log File: n-api
2013-12-11 00:14:34.044 | 2013-12-11 00:04:32.475 ERROR nova.image.s3 [req-64000d25-93ec-43d8-817b-f62ec9a17a16 demo demo] Failed to upload testbucket/bundle.img.manifest.xml to /tmp/tmpsS2EHo
2013-12-11 00:14:34.045 |
2013-12-11 00:14:35.542 | Logs have errors
2013-12-11 00:14:35.542 | FAILED

[Abhishek Chanda (abhishek-i)]

This might actually be a glance issue. Marked as critical since it is blocking gates.

[Joe Gordon (jogo)]

If you think it might be a glance issue and not a nova issue, then this bug isn't triaged for nova and shouldn't be marked as critical yet.

[Matt Riedemann (mriedem)]

These errors are also in the glance API logs:

2013-12-11 16:28:25.864 10055 ERROR glance.api.v1.upload_utils [ba4c79c1-ea44-4f93-9e77-6ad3e77a97c4 c528309692694ed69e2a65f623c5b220 657967d64d7a44689a3093e6838a6fa4] Received HTTP error while uploading image 4681af21-56d3-4abf-8d54-f08de29dbc5f
2013-12-11 16:28:25.881 10055 ERROR glance.api.v1.upload_utils [ba4c79c1-ea44-4f93-9e77-6ad3e77a97c4 c528309692694ed69e2a65f623c5b220 657967d64d7a44689a3093e6838a6fa4] Unable to kill image 4681af21-56d3-4abf-8d54-f08de29dbc5f:

[Matt Riedemann (mriedem)]

http://logs.openstack.org/69/59869/4/check/check-grenade-dsvm/5dea7d2/logs/new/screen-g-api.txt.gz?level=ERROR

[Jeremy Stanley (fungi)]

Also:

Image cache contained image file for image '303d6d16-e2ee-4b71-8900-61820e087714', however the registry did not contain metadata for that image!

http://logs.openstack.org/19/61419/1/check/check-tempest-dsvm-full/c9cc87f/logs/screen-g-api.txt.gz?level=ERROR

[Mark Washenberger (markwash)]

What appears to be happening is that the image is being deleted before the PUT of data has finished. But I can't find any recent glance behavior changes related to that. Is it possible that the PUT is just taking longer than before? Is it possible that what is driving the tests for this has changed slightly?

[Mark Washenberger (markwash)]

Related to https://bugs.launchpad.net/glance/+bug/1236868 but that bug has been active for a very long time so again there might be other factors in place causing this bug to surface in our tests.

[Mark Washenberger (markwash)]

Actually that bug above appears like it may be the result for admin users. For regular users, it seems like this current behavior has been in place at least since grizzly. So one other thing that might have changed here was the presence or absence of the admin role on the test account.

[Mark Washenberger (markwash)]

Marking as incomplete just because I need a little more info here about why this just started to break in order to figure out what should be fixed. AFAICT this is *normal* glance behavior, so maybe we just need to clean up the log levels?

[Davanum Srinivas (DIMS) (dims-v)]

@markwash - this is a gate bug, all the logs are in available as well as the code for the test cases...What else is needed? Failure rates are hovering between 30-45%, I don't think we should be marking this as "Incomplete" IMHO.

[Matt Riedemann (mriedem)]

2 hits in the last 7 days for:

message:"Received HTTP error while uploading image" AND filename:"logs/screen-g-api.txt"

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiUmVjZWl2ZWQgSFRUUCBlcnJvciB3aGlsZSB1cGxvYWRpbmcgaW1hZ2VcIiBBTkQgZmlsZW5hbWU6XCJsb2dzL3NjcmVlbi1nLWFwaS50eHRcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTM4Njc4OTIyOTAzNX0=

2 hits in the last 7 days for:

message:"Unable to kill image" AND filename:"logs/screen-g-api.txt"

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiVW5hYmxlIHRvIGtpbGwgaW1hZ2VcIiBBTkQgZmlsZW5hbWU6XCJsb2dzL3NjcmVlbi1nLWFwaS50eHRcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTM4Njc4OTMxMzM2Nn0=

5504 hits in the last 7 days for:

message:"however the registry did not contain metadata for that image!" AND filename:"logs/screen-g-api.txt"

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiaG93ZXZlciB0aGUgcmVnaXN0cnkgZGlkIG5vdCBjb250YWluIG1ldGFkYXRhIGZvciB0aGF0IGltYWdlIVwiIEFORCBmaWxlbmFtZTpcImxvZ3Mvc2NyZWVuLWctYXBpLnR4dFwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxMzg2Nzg5Mzc5MzY0fQ==

[Matt Riedemann (mriedem)]

Hmm, "however the registry did not contain metadata for that image" for g-api logs is in the whitelist here:

http://git.openstack.org/cgit/openstack/tempest/tree/etc/whitelist.yaml#n50

So that shouldn't be causing failures.

[Matt Riedemann (mriedem)]

I don't know that I'd consider this 'triaged' necessarily, that implies that we know what the root cause is.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/61826

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/61826
Committed: https://git.openstack.org/cgit/openstack-infra/elastic-recheck/commit/?id=7790fabe2e99cc2d41cfdb147a18f10c4dd9aa4f
Submitter: Jenkins
Branch: master

commit 7790fabe2e99cc2d41cfdb147a18f10c4dd9aa4f
Author: Matt Riedemann <email address hidden>
Date: Thu Dec 12 09:14:05 2013 -0800

    Add e-r query for bug 1259907

    There are errors in the screen-g-api.txt log but they are whitelisted
    errors already so we can't query on them since they would result in
    false negatives. See the bug report for more details.

    Related-Bug: #1259907

    Change-Id: I8553358e7316d466cbb88e1a3809dc17b5baf53f

[Sean Dague (sdague)]

Part of the problem here is that the log fail detector in tempest is actually breaking the grenade runs when it probably shouldn't. Grenade was not a primary use case for that script.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/62107
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=1159e52a2c9c51976bc3be5ad504c88fb94c2fe1
Submitter: Jenkins
Branch: master

commit 1159e52a2c9c51976bc3be5ad504c88fb94c2fe1
Author: Sean Dague <email address hidden>
Date: Fri Dec 13 18:46:21 2013 -0500

    don't fail on dirty logs with grenade

    because grenade is upgrading from old to new we might actually
    expect the logs to be dirtier than in upstream tempest. The grenade
    logs weren't scrubbed in the same ways during the development here
    as the tempest regular runs.

    Change-Id: Id1bcc2cc85e73a414d382756a65ea1d80dc10b00
    Closes-Bug: #1259907

[Jeremy Stanley (fungi)]

Seems to have been fixed in grenade.

[Joe Gordon (jogo)]

this bug is fairly old, if it comes up again we should open a new bug.

[Erno Kuvaja (jokke)]

I do agree with Joe. Marking as Invalid for glance as well.