Jenkins fails due to test_register_http_image error

[Vijendar Komalla]

Below test is failing consistently and as a result gerrit review https://review.openstack.org/#/c/52137/ is failing and this is also blocking heat DBaaS resource implementation.

======================================================================
2013-11-01 15:05:18.597 | FAIL: tempest.api.image.v1.test_images.CreateRegisterImagesTest.test_register_http_image[gate]
2013-11-01 15:05:18.598 | tempest.api.image.v1.test_images.CreateRegisterImagesTest.test_register_http_image[gate]
2013-11-01 15:05:18.598 | ----------------------------------------------------------------------
2013-11-01 15:05:18.613 | _StringException: Empty attachments:
2013-11-01 15:05:18.613 | stderr
2013-11-01 15:05:18.613 | stdout
2013-11-01 15:05:18.613 |
2013-11-01 15:05:18.614 | pythonlogging:'': {{{
2013-11-01 15:05:18.614 | 2013-11-01 14:58:46,953 Request: POST http://127.0.0.1:5000/v2.0/tokens
2013-11-01 15:05:18.614 | 2013-11-01 14:58:46,954 Request Headers: {'Content-Type': 'application/json'}
2013-11-01 15:05:18.614 | 2013-11-01 14:58:46,954 Request Body: {"auth": {"tenantName": "CreateRegisterImagesTest-tempest-815770946-tenant", "passwordCredentials": {"username": "CreateRegisterImagesTest-tempest-815770946-user", "password": "pass"}}}
2013-11-01 15:05:18.614 | 2013-11-01 14:58:47,080 Response Status: 200
2013-11-01 15:05:18.614 | 2013-11-01 14:58:47,080 Response Headers: {'content-length': '10752', 'date': 'Fri, 01 Nov 2013 14:58:47 GMT', 'content-type': 'application/json', 'vary': 'X-Auth-Token', 'connection': 'close'}
2013-11-01 15:05:18.614 | 2013-11-01 14:58:47,080 Response Body: {"access": {"token": {"issued_at": "2013-11-01T14:58:47.031340", "expires": "2013-11-02T14:58:47Z", "id": "xxxxxxxxx
2013-11-01 15:05:18.615 | 2013-11-01 14:58:47,080 Large body (10752) md5 summary: 71395c710dafa80b0bdfab04908a4c1c
2013-11-01 15:05:18.615 | 2013-11-01 14:58:47,081 Request: POST http://127.0.0.1:9292/v1/images
2013-11-01 15:05:18.615 | 2013-11-01 14:58:47,081 Request Headers: {'x-image-meta-name': 'New Http Image', 'x-glance-api-copy-from': 'http://127.0.0.1/', 'x-image-meta-container_format': 'bare', 'x-image-meta-is_public': 'True', 'x-image-meta-disk_format': 'raw', 'X-Auth-Token': '<Token omitted>'}
2013-11-01 15:05:18.615 | 2013-11-01 14:58:48,789 Response Status: 201
2013-11-01 15:05:18.616 | 2013-11-01 14:58:48,789 Glance request id req-963e5d07-338c-4faa-ad48-229b5b246af3
2013-11-01 15:05:18.616 | 2013-11-01 14:58:48,789 Response Headers: {'content-length': '425', 'location': 'http://127.0.0.1:9292/v1/images/bc1d96f3-4af1-4bbe-b513-f48a3c72fb29', 'date': 'Fri, 01 Nov 2013 14:58:48 GMT', 'content-type': 'application/json', 'connection': 'close'}
2013-11-01 15:05:18.616 | 2013-11-01 14:58:48,789 Response Body: {"image": {"status": "queued", "deleted": false, "container_format": "bare", "min_ram": 0, "updated_at": "2013-11-01T14:58:48", "owner": "08d9e854141448eb98ec9ed2076c6ad0", "min_disk": 0, "is_public": true, "deleted_at": null, "id": "bc1d96f3-4af1-4bbe-b513-f48a3c72fb29", "size": 0, "name": "New Http Image", "checksum": null, "created_at": "2013-11-01T14:58:48", "disk_format": "raw", "properties": {}, "protected": false}}
2013-11-01 15:05:18.617 | 2013-11-01 14:58:48,790 Request: HEAD http://127.0.0.1:9292/v1/images/bc1d96f3-4af1-4bbe-b513-f48a3c72fb29
2013-11-01 15:05:18.617 | 2013-11-01 14:58:48,790 Request Headers: {'X-Auth-Token': '<Token omitted>'}
2013-11-01 15:05:18.617 | 2013-11-01 14:58:48,825 Response Status: 200
2013-11-01 15:05:18.617 | 2013-11-01 14:58:48,825 Glance request id req-58dc7b3f-e61c-4316-9a80-6ad5e2e2bf01
2013-11-01 15:05:18.618 | 2013-11-01 14:58:48,825 Response Headers: {'content-length': '0', 'x-image-meta-id': 'bc1d96f3-4af1-4bbe-b513-f48a3c72fb29', 'x-image-meta-status': 'queued', 'date': 'Fri, 01 Nov 2013 14:58:48 GMT', 'x-image-meta-owner': '08d9e854141448eb98ec9ed2076c6ad0', 'x-image-meta-name': 'New Http Image', 'content-location': u'http://127.0.0.1:9292/v1/images/bc1d96f3-4af1-4bbe-b513-f48a3c72fb29', 'x-image-meta-deleted': 'False', 'x-image-meta-container_format': 'bare', 'x-image-meta-protected': 'False', 'x-image-meta-min_disk': '0', 'x-image-meta-created_at': '2013-11-01T14:58:48', 'x-image-meta-size': '0', 'connection': 'close', 'location': 'http://127.0.0.1:9292/v1/images/bc1d96f3-4af1-4bbe-b513-f48a3c72fb29', 'x-image-meta-is_public': 'True', 'x-image-meta-min_ram': '0', 'x-image-meta-updated_at': '2013-11-01T14:58:48', 'content-type': 'text/html; charset=UTF-8', 'x-image-meta-disk_format': 'raw'}
2013-11-01 15:05:18.618 | 2013-11-01 14:58:50,827 Request: HEAD http://127.0.0.1:9292/v1/images/bc1d96f3-4af1-4bbe-b513-f48a3c72fb29
2013-11-01 15:05:18.618 | 2013-11-01 14:58:50,828 Request Headers: {'X-Auth-Token': '<Token omitted>'}
2013-11-01 15:05:18.619 | 2013-11-01 14:58:50,855 Response Status: 200
2013-11-01 15:05:18.619 | 2013-11-01 14:58:50,855 Glance request id req-f6ded1cd-2ec9-4a36-b91a-21f16ac208c9
2013-11-01 15:05:18.619 | 2013-11-01 14:58:50,855 Response Headers: {'content-length': '0', 'x-image-meta-id': 'bc1d96f3-4af1-4bbe-b513-f48a3c72fb29', 'x-image-meta-status': 'killed', 'date': 'Fri, 01 Nov 2013 14:58:50 GMT', 'x-image-meta-owner': '08d9e854141448eb98ec9ed2076c6ad0', 'x-image-meta-name': 'New Http Image', 'content-location': u'http://127.0.0.1:9292/v1/images/bc1d96f3-4af1-4bbe-b513-f48a3c72fb29', 'x-image-meta-deleted': 'False', 'x-image-meta-container_format': 'bare', 'x-image-meta-protected': 'False', 'x-image-meta-min_disk': '0', 'x-image-meta-created_at': '2013-11-01T14:58:48', 'x-image-meta-size': '0', 'connection': 'close', 'location': 'http://127.0.0.1:9292/v1/images/bc1d96f3-4af1-4bbe-b513-f48a3c72fb29', 'x-image-meta-is_public': 'True', 'x-image-meta-min_ram': '0', 'x-image-meta-updated_at': '2013-11-01T14:58:48', 'content-type': 'text/html; charset=UTF-8', 'x-image-meta-disk_format': 'raw'}
2013-11-01 15:05:18.620 | 2013-11-01 14:58:51,856 Value transition from "queued" to "killed"in 2 second(s).
2013-11-01 15:05:18.620 | }}}
2013-11-01 15:05:18.620 |
2013-11-01 15:05:18.621 | Traceback (most recent call last):
2013-11-01 15:05:18.621 | File "tempest/api/image/v1/test_images.py", line 90, in test_register_http_image
2013-11-01 15:05:18.621 | self.client.wait_for_image_status(image_id, 'active')
2013-11-01 15:05:18.621 | File "tempest/services/image/v1/json/image_client.py", line 273, in wait_for_image_status
2013-11-01 15:05:18.622 | status=value)
2013-11-01 15:05:18.622 | ImageKilledException: Image bc1d96f3-4af1-4bbe-b513-f48a3c72fb29 'killed' while waiting for killed
2013-11-01 15:05:18.622 |
2013-11-01 15:05:18.624 |

[Vijendar Komalla (vijendar-komalla)]

I would like to know why this bug status changed to 'Invalid'. I am still seeing the CI failure. Please take a look at latest jenkins failure at https://review.openstack.org/#/c/52137/

Below given is the failure log

2013-11-08 15:26:23.546 | ======================================================================
2013-11-08 15:26:23.547 | FAIL: tempest.api.image.v1.test_images.CreateRegisterImagesTest.test_register_http_image[gate]
2013-11-08 15:26:23.547 | tempest.api.image.v1.test_images.CreateRegisterImagesTest.test_register_http_image[gate]
2013-11-08 15:26:23.548 | ----------------------------------------------------------------------
2013-11-08 15:26:23.548 | _StringException: Empty attachments:
2013-11-08 15:26:23.548 | stderr
2013-11-08 15:26:23.548 | stdout
2013-11-08 15:26:23.549 |
2013-11-08 15:26:23.549 | pythonlogging:'': {{{
2013-11-08 15:26:23.549 | 2013-11-08 15:16:36,332 Request: POST http://127.0.0.1:5000/v2.0/tokens
2013-11-08 15:26:23.550 | 2013-11-08 15:16:36,332 Request Headers: {'Content-Type': 'application/json'}
2013-11-08 15:26:23.550 | 2013-11-08 15:16:36,332 Request Body: {"auth": {"tenantName": "CreateRegisterImagesTest-tempest-1788601899-tenant", "passwordCredentials": {"username": "CreateRegisterImagesTest-tempest-1788601899-user", "password": "pass"}}}
2013-11-08 15:26:23.550 | 2013-11-08 15:16:36,648 Response Status: 200
2013-11-08 15:26:23.551 | 2013-11-08 15:16:36,648 Response Headers: {'content-length': '10760', 'date': 'Fri, 08 Nov 2013 15:16:36 GMT', 'content-type': 'application/json', 'vary': 'X-Auth-Token', 'connection': 'close'}
2013-11-08 15:26:23.551 | 2013-11-08 15:16:36,648 Response Body: {"access": {"token": {"issued_at": "2013-11-08T15:16:36.506582", "expires": "2013-11-09T15:16:36Z", "id": "MIISxAYJKoZIhvcNAxxxxxxx"
2013-11-08 15:26:23.551 | 2013-11-08 15:16:36,648 Large body (10760) md5 summary: 3b2c014d60229cc791a8b0c4443f6d1b
2013-11-08 15:26:23.552 | 2013-11-08 15:16:36,649 Request: POST http://127.0.0.1:9292/v1/images
2013-11-08 15:26:23.552 | 2013-11-08 15:16:36,649 Request Headers: {'x-image-meta-name': 'New Http Image', 'x-glance-api-copy-from': 'http://127.0.0.1/', 'x-image-meta-container_format': 'bare', 'x-image-meta-is_public': 'True', 'x-image-meta-disk_format': 'raw', 'X-Auth-Token': '<Token omitted>'}
2013-11-08 15:26:23.552 | 2013-11-08 15:16:39,716 Response Status: 201
2013-11-08 15:26:23.553 | 2013-11-08 15:16:39,717 Glance request id req-5672f873-c363-4b82-a8e9-241779c31375
2013-11-08 15:26:23.553 | 2013-11-08 15:16:39,717 Response Headers: {'content-length': '425', 'location': 'http://127.0.0.1:9292/v1/images/7b8028df-bb47-4455-9523-355a05062bc4', 'date': 'Fri, 08 Nov 2013 15:16:39 GMT', 'content-type': 'application/json', 'connection': 'close'}
2013-11-08 15:26:23.553 | 2013-11-08 15:16:39,717 Response Body: {"image": {"status": "queued", "deleted": false, "container_format": "bare", "min_ram": 0, "updated_at": "2013-11-08T15:16:39", "owner": "f27e07d60a014fd39015cbfabb04154c", "min_disk": 0, "is_public": true, "deleted_at": null, "id": "7b8028df-bb47-4455-9523-355a05062bc4", "size": 0, "name": "New Http Image", "checksum": null, "created_at": "2013-11-08T15:16:39", "di...

[Jeremy Stanley (fungi)]

I'm not sure why you think this is a problem with the machines rather than in existing openstack software, test software such as tempest, or introduced within your change itself. The CI systems exist to tell you when something is broken (either broken software or broken tests), and it seems to be performing this task well.

Can you elaborate on why you think our continuous integration servers are broken instead, and also why you believe David's comments on https://review.openstack.org/52137 are incorrect?

[Vijendar Komalla (vijendar-komalla)]

probably there is some issue with glance and horizon that is causing couple of test failures. There are two failures in gerrit review https://review.openstack.org/52137 one glance test is failing and one horizon test is failing.

I think there is no issue with CI servers, but due to some issues in horizon/glance tests are failing in CI. This bug is to investigate/track the glance issue and there is a separate bug to track horizon test issue https://bugs.launchpad.net/horizon/+bug/1247198

[Mark Washenberger (markwash)]

Any clue what is actually happening that is causing the test to fail? That output is pretty hard to parse.

[Vijendar Komalla (vijendar-komalla)]

Mark, please take a look at gerrit review https://review.openstack.org/#/c/52137/ for complete output (http://logs.openstack.org/37/52137/5/check/check-tempest-devstack-vm-full/03e07f1/console.html)

[Erno Kuvaja (jokke)]

As this has not been popping up since I think it would be safe to assume this was change introduced "bug" and could be closed?