[ERROR] /opt/stack/new/devstack/exercises/bundle.sh:61 Image ami-00000001 not available within 15 seconds

[Jeremy Stanley]

Nondeterministic failure on bundle exercises in devstack, as evidenced by http://logs.openstack.org/28321/1/gate/gate-tempest-devstack-vm-full/14732/console.html

2013-05-06 21:56:04.877 | Uploaded image as testbucket/bundle.img.manifest.xml
2013-05-06 21:56:04.879 | ++ euca-register testbucket/bundle.img.manifest.xml
2013-05-06 21:56:04.880 | ++ cut -f2
2013-05-06 21:56:06.639 | + AMI=ami-00000001
2013-05-06 21:56:06.676 | + die_if_not_set 57 AMI 'Failure registering testbucket/bundle.img'
2013-05-06 21:56:06.676 | + local exitcode=0
2013-05-06 21:56:06.676 | ++ set +o
2013-05-06 21:56:06.677 | ++ grep xtrace
2013-05-06 21:56:06.677 | + FXTRACE='set -o xtrace'
2013-05-06 21:56:06.677 | + set +o xtrace
2013-05-06 21:56:06.677 | + timeout 15 sh -c 'while euca-describe-images | grep ami-00000001 | grep -q available; do sleep 1; done'
2013-05-06 21:56:21.644 | + die 61 'Image ami-00000001 not available within 15 seconds'
2013-05-06 21:56:21.644 | + local exitcode=0
2013-05-06 21:56:21.644 | + set +o xtrace
2013-05-06 21:56:21.681 | [ERROR] /opt/stack/new/devstack/exercises/bundle.sh:61 Image ami-00000001 not available within 15 seconds

[Dean Troyer (dtroyer)]

Is this a case where the timeout is just too close for the worker VM performance variations?

[Fabien Boucher (fabien-boucher)]

I have the same problem that can be seen here :
http://logs.openstack.org/23/38823/3/check/gate-tempest-devstack-vm-neutron/ce18bbe/console.html.gz

...
2013-08-06 15:15:59.716 | Uploading manifest file
2013-08-06 15:15:59.716 | send: 'PUT /testbucket/bundle.img.manifest.xml HTTP/1.1\r\nHost: 127.0.0.1:3333\r\nAccept-Encoding: identity\r\nContent-MD5: CWIhSP7D8SyTfp0sGPl6ZQ==\r\nContent-Length: 2111\r\nExpect: 100-Continue\r\nDate: Tue, 06 Aug 2013 15:15:59 GMT\r\nUser-Agent: Boto/2.2.2 (linux2)\r\nContent-Type: application/xml\r\nAuthorization: AWS 3add7ac39ff744de83cce3edaf7f6695:SVts7tYjPZUG0BofT6upANBaGos=\r\nx-amz-acl: aws-exec-read\r\n\r\n'
2013-08-06 15:15:59.716 | Uploading part: bundle.img.part.00
2013-08-06 15:15:59.716 | send: 'PUT /testbucket/bundle.img.part.00 HTTP/1.1\r\nHost: 127.0.0.1:3333\r\nAccept-Encoding: identity\r\nContent-MD5: HRhMWrUV7b32MadBNVLO8A==\r\nContent-Length: 144\r\nExpect: 100-Continue\r\nDate: Tue, 06 Aug 2013 15:15:59 GMT\r\nUser-Agent: Boto/2.2.2 (linux2)\r\nContent-Type: application/octet-stream\r\nAuthorization: AWS 3add7ac39ff744de83cce3edaf7f6695:+hfYdzXbXOdiqs0mo9v+RzOyU8U=\r\nx-amz-acl: aws-exec-read\r\n\r\n'
2013-08-06 15:15:59.716 | Uploaded image as testbucket/bundle.img.manifest.xml
2013-08-06 15:15:59.718 | ++ euca-register testbucket/bundle.img.manifest.xml
2013-08-06 15:15:59.718 | ++ cut -f2
2013-08-06 15:16:00.390 | + AMI=ami-00000001
2013-08-06 15:16:00.391 | + die_if_not_set 57 AMI 'Failure registering testbucket/bundle.img'
2013-08-06 15:16:00.391 | + local exitcode=0
2013-08-06 15:16:00.391 | ++ set +o
2013-08-06 15:16:00.391 | ++ grep xtrace
2013-08-06 15:16:00.393 | + FXTRACE='set -o xtrace'
2013-08-06 15:16:00.393 | + set +o xtrace
2013-08-06 15:16:00.394 | + timeout 15 sh -c 'while euca-describe-images | grep ami-00000001 | grep -q available; do sleep 1; done'
2013-08-06 15:16:15.395 | + die 61 'Image ami-00000001 not available within 15 seconds'
2013-08-06 15:16:15.395 | + local exitcode=0
2013-08-06 15:16:15.396 | + set +o xtrace
2013-08-06 15:16:15.398 | [ERROR] /opt/stack/new/devstack/exercises/bundle.sh:61 Image ami-00000001 not available within 15 seconds
...

[Matt Riedemann (mriedem)]

Just failed again here: http://logs.openstack.org/23/57823/1/check/check-grenade-devstack-vm/7ad84a1/

[Matt Riedemann (mriedem)]

message:"[ERROR] /opt/stack/old/devstack/exercises/bundle.sh:61 Image ami-00000001 not available within 15 seconds" AND filename:"console.html"

[Matt Riedemann (mriedem)]

I pushed up an elastic-recheck query patch for this: https://review.openstack.org/#/c/58014/

It doesn't look like it's very frequent, but it has been showing up for a long time and the error is pretty specific so we might as well track it.

[Joe Gordon (jogo)]

this hasn't been seen in two weeks. Fixed here: https://review.openstack.org/#/c/66934/