./stack.sh is resulting any error "/opt/stack/devstack/functions: line 1228: : No such file or directory" on stable/grizzly branch

[Harika Vakadi]

While installing devstack on stable/grizzly branch it is running into the following error

<snip>
gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
++ for f in '"$xdir/"*-vmlinuz*' '"$xdir/"aki-*/image'
++ '[' -f '/opt/stack/devstack/files/images/cirros-0.3.1-x86_64-uec/*-vmlinuz*' ']'
++ for f in '"$xdir/"*-vmlinuz*' '"$xdir/"aki-*/image'
++ '[' -f '/opt/stack/devstack/files/images/cirros-0.3.1-x86_64-uec/aki-*/image' ']'
++ true
+ KERNEL=
++ for f in '"$xdir/"*-initrd*' '"$xdir/"ari-*/image'
++ '[' -f '/opt/stack/devstack/files/images/cirros-0.3.1-x86_64-uec/*-initrd*' ']'
++ for f in '"$xdir/"*-initrd*' '"$xdir/"ari-*/image'
++ '[' -f '/opt/stack/devstack/files/images/cirros-0.3.1-x86_64-uec/ari-*/image' ']'
++ true
+ RAMDISK=
++ for f in '"$xdir/"*.img' '"$xdir/"ami-*/image'
++ '[' -f '/opt/stack/devstack/files/images/cirros-0.3.1-x86_64-uec/*.img' ']'
++ for f in '"$xdir/"*.img' '"$xdir/"ami-*/image'
++ '[' -f '/opt/stack/devstack/files/images/cirros-0.3.1-x86_64-uec/ami-*/image' ']'
++ true
+ IMAGE=
+ [[ -z cirros-0.3.1-x86_64-uec ]]
+ '[' '' = bare ']'
+ KERNEL_ID=
+ RAMDISK_ID=
+ '[' -n '' ']'
+ '[' -n '' ']'

+ glance --os-auth-token MIIM4QYJKoZIhvcNAQcCoIIM0jCCDM4CAQExCTAHBgUrDgMCGjCCC7oGCSqGSIb3DQEHAaCCC6sEgguneyJhY2Nlc3MiOiB7InRva2VuIjogeyJpc3N1ZWRfYXQiOiAiMjAxMy0wNi0xNFQwOToxMTozMy45NDQ1NjQiLCAiZXhwaXJlcyI6ICIyMDEzLTA2LTE1VDA5OjExOjMzWiIsICJpZCI6ICJwbGFjZWhvbGRlciIsICJ0ZW5hbnQiOiB7ImRlc2NyaXB0aW9uIjogbnVsbCwgImVuYWJsZWQiOiB0cnVlLCAiaWQiOiAiNWI4ZmQ0MTQxZTE0NGQ5Zjk1ZWI2MmY2MTRlY2I1NjgiLCAibmFtZSI6ICJhZG1pbiJ9fSwgInNlcnZpY2VDYXRhbG9nIjogW3siZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzEwLjIzMy41Mi4yNDU6ODc3NC92Mi81YjhmZDQxNDFlMTQ0ZDlmOTVlYjYyZjYxNGVjYjU2OCIsICJyZWdpb24iOiAiUmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xMC4yMzMuNTIuMjQ1Ojg3NzQvdjIvNWI4ZmQ0MTQxZTE0NGQ5Zjk1ZWI2MmY2MTRlY2I1NjgiLCAiaWQiOiAiM2Q1Njc3YWRjMTIxNGM5MDliNDk3MWRkYmE5YTM2MmIiLCAicHVibGljVVJMIjogImh0dHA6Ly8xMC4yMzMuNTIuMjQ1Ojg3NzQvdjIvNWI4ZmQ0MTQxZTE0NGQ5Zjk1ZWI2MmY2MTRlY2I1NjgifV0sICJlbmRwb2ludHNfbGlua3MiOiBbXSwgInR5cGUiOiAiY29tcHV0ZSIsICJuYW1lIjogIm5vdmEifSwgeyJlbmRwb2ludHMiOiBbeyJhZG1pblVSTCI6ICJodHRwOi8vMTAuMjMzLjUyLjI0NTo5Njk2LyIsICJyZWdpb24iOiAiUmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xMC4yMzMuNTIuMjQ1Ojk2OTYvIiwgImlkIjogIjM2MzQzODRjNGQwYzRhODlhNGFlZDdkNDk5OWU2YjA4IiwgInB1YmxpY1VSTCI6ICJodHRwOi8vMTAuMjMzLjUyLjI0NTo5Njk2LyJ9XSwgImVuZHBvaW50c19saW5rcyI6IFtdLCAidHlwZSI6ICJuZXR3b3JrIiwgIm5hbWUiOiAicXVhbnR1bSJ9LCB7ImVuZHBvaW50cyI6IFt7ImFkbWluVVJMIjogImh0dHA6Ly8xMC4yMzMuNTIuMjQ1OjMzMzMiLCAicmVnaW9uIjogIlJlZ2lvbk9uZSIsICJpbnRlcm5hbFVSTCI6ICJodHRwOi8vMTAuMjMzLjUyLjI0NTozMzMzIiwgImlkIjogIjBjN2Y0YjA2NmFkYjQ4YjU4MGU4NjMxNGQ0M2NmYTlmIiwgInB1YmxpY1VSTCI6ICJodHRwOi8vMTAuMjMzLjUyLjI0NTozMzMzIn1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogInMzIiwgIm5hbWUiOiAiczMifSwgeyJlbmRwb2ludHMiOiBbeyJhZG1pblVSTCI6ICJodHRwOi8vMTAuMjMzLjUyLjI0NTo5MjkyIiwgInJlZ2lvbiI6ICJSZWdpb25PbmUiLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovLzEwLjIzMy41Mi4yNDU6OTI5MiIsICJpZCI6ICIwNDkzOGQxODkyNTY0NTNjYmUxYzczNTRiNjY3YmViYiIsICJwdWJsaWNVUkwiOiAiaHR0cDovLzEwLjIzMy41Mi4yNDU6OTI5MiJ9XSwgImVuZHBvaW50c19saW5rcyI6IFtdLCAidHlwZSI6ICJpbWFnZSIsICJuYW1lIjogImdsYW5jZSJ9LCB7ImVuZHBvaW50cyI6IFt7ImFkbWluVVJMIjogImh0dHA6Ly8xMC4yMzMuNTIuMjQ1Ojg3NzYvdjEvNWI4ZmQ0MTQxZTE0NGQ5Zjk1ZWI2MmY2MTRlY2I1NjgiLCAicmVnaW9uIjogIlJlZ2lvbk9uZSIsICJpbnRlcm5hbFVSTCI6ICJodHRwOi8vMTAuMjMzLjUyLjI0NTo4Nzc2L3YxLzViOGZkNDE0MWUxNDRkOWY5NWViNjJmNjE0ZWNiNTY4IiwgImlkIjogIjBjMjA0ZDMxNzNmNDQ2ZGM5ODgzYmJhNDQzOTg0ZjY4IiwgInB1YmxpY1VSTCI6ICJodHRwOi8vMTAuMjMzLjUyLjI0NTo4Nzc2L3YxLzViOGZkNDE0MWUxNDRkOWY5NWViNjJmNjE0ZWNiNTY4In1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogInZvbHVtZSIsICJuYW1lIjogImNpbmRlciJ9LCB7ImVuZHBvaW50cyI6IFt7ImFkbWluVVJMIjogImh0dHA6Ly8xMC4yMzMuNTIuMjQ1Ojg3NzMvc2VydmljZXMvQWRtaW4iLCAicmVnaW9uIjogIlJlZ2lvbk9uZSIsICJpbnRlcm5hbFVSTCI6ICJodHRwOi8vMTAuMjMzLjUyLjI0NTo4NzczL3NlcnZpY2VzL0Nsb3VkIiwgImlkIjogIjA4M2FiMDQ1NjFmYzRhY2RhZThmMzdlMDVmNjQwMzRhIiwgInB1YmxpY1VSTCI6ICJodHRwOi8vMTAuMjMzLjUyLjI0NTo4NzczL3NlcnZpY2VzL0Nsb3VkIn1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogImVjMiIsICJuYW1lIjogImVjMiJ9LCB7ImVuZHBvaW50cyI6IFt7ImFkbWluVVJMIjogImh0dHA6Ly8xMC4yMzMuNTIuMjQ1OjgwODAiLCAicmVnaW9uIjogIlJlZ2lvbk9uZSIsICJpbnRlcm5hbFVSTCI6ICJodHRwOi8vMTAuMjMzLjUyLjI0NTo4MDgwL3YxL0FVVEhfNWI4ZmQ0MTQxZTE0NGQ5Zjk1ZWI2MmY2MTRlY2I1NjgiLCAiaWQiOiAiMTc3ZTdjZGM1M2Y1NDgxODgwYzcwNTE3MzAzMGE4NDUiLCAicHVibGljVVJMIjogImh0dHA6Ly8xMC4yMzMuNTIuMjQ1OjgwODAvdjEvQVVUSF81YjhmZDQxNDFlMTQ0ZDlmOTVlYjYyZjYxNGVjYjU2OCJ9XSwgImVuZHBvaW50c19saW5rcyI6IFtdLCAidHlwZSI6ICJvYmplY3Qtc3RvcmUiLCAibmFtZSI6ICJzd2lmdCJ9LCB7ImVuZHBvaW50cyI6IFt7ImFkbWluVVJMIjogImh0dHA6Ly8xMC4yMzMuNTIuMjQ1OjM1MzU3L3YyLjAiLCAicmVnaW9uIjogIlJlZ2lvbk9uZSIsICJpbnRlcm5hbFVSTCI6ICJodHRwOi8vMTAuMjMzLjUyLjI0NTo1MDAwL3YyLjAiLCAiaWQiOiAiMzhmYzlkMWI1Y2NlNGNlOWFhZTAwMWQ2MGQ0ZGJiZWIiLCAicHVibGljVVJMIjogImh0dHA6Ly8xMC4yMzMuNTIuMjQ1OjUwMDAvdjIuMCJ9XSwgImVuZHBvaW50c19saW5rcyI6IFtdLCAidHlwZSI6ICJpZGVudGl0eSIsICJuYW1lIjogImtleXN0b25lIn1dLCAidXNlciI6IHsidXNlcm5hbWUiOiAiYWRtaW4iLCAicm9sZXNfbGlua3MiOiBbXSwgImlkIjogIjY0N2IzODg5ZmVmMTRlNGQ4NTE4MjZiN2FjYmVlMTdlIiwgInJvbGVzIjogW3sibmFtZSI6ICJhZG1pbiJ9XSwgIm5hbWUiOiAiYWRtaW4ifSwgIm1ldGFkYXRhIjogeyJpc19hZG1pbiI6IDAsICJyb2xlcyI6IFsiNzg1MGVjM2I0YWExNGZiMjgwZTFlMzI3YzZlMzQ3ODAiXX19fTGB-zCB-AIBATBcMFcxCzAJBgNVBAYTAlVTMQ4wDAYDVQQIEwVVbnNldDEOMAwGA1UEBxMFVW5zZXQxDjAMBgNVBAoTBVVuc2V0MRgwFgYDVQQDEw93d3cuZXhhbXBsZS5jb20CAQEwBwYFKw4DAhowDQYJKoZIhvcNAQEBBQAEgYAMDE72yHfU0WAMb6BIvQIWP+KPUuxHMJ9PD-35jHaYrpttvr-A7n6g7cj6fi4rFznHx2KCZNLa9Ot3hY7KZIbucHvEdOiQr72J2NQRyhixzGmUgUJciU+KHGd-57n+uJfO1Jd5HS7IGJyAcefTzrZW9dBnkk1QUZ7eEh3b0Upr0w== --os-image-url http://10.233.52.245:9292 image-create --name cirros-0.3.1-x86_64-uec --public --container-format ami --disk-format ami
/opt/stack/devstack/functions: line 1228: : No such file or directory
++ failed
++ local r=1
+++ jobs -p
++ kill
++ set +o xtrace

<snip>

Can any one suggest on how to follow up, ASAP

Thanks in advance
Harika

[Abhishek (mrjain)]

same issue observed while installing devstack on UBUNTU 12.04 using stack user

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
++ for f in '"$xdir/"*-vmlinuz*' '"$xdir/"aki-*/image'
++ '[' -f '/opt/stack/devstack/files/images/cirros-0.3.1-x86_64-uec/*-vmlinuz*' ']'
++ for f in '"$xdir/"*-vmlinuz*' '"$xdir/"aki-*/image'
++ '[' -f '/opt/stack/devstack/files/images/cirros-0.3.1-x86_64-uec/aki-*/image' ']'
++ true
+ KERNEL=
++ for f in '"$xdir/"*-initrd*' '"$xdir/"ari-*/image'
++ '[' -f '/opt/stack/devstack/files/images/cirros-0.3.1-x86_64-uec/*-initrd*' ']'
++ for f in '"$xdir/"*-initrd*' '"$xdir/"ari-*/image'
++ '[' -f '/opt/stack/devstack/files/images/cirros-0.3.1-x86_64-uec/ari-*/image' ']'
++ true
+ RAMDISK=
++ for f in '"$xdir/"*.img' '"$xdir/"ami-*/image'
++ '[' -f '/opt/stack/devstack/files/images/cirros-0.3.1-x86_64-uec/*.img' ']'
++ for f in '"$xdir/"*.img' '"$xdir/"ami-*/image'
++ '[' -f '/opt/stack/devstack/files/images/cirros-0.3.1-x86_64-uec/ami-*/image' ']'
++ true
+ IMAGE=
+ [[ -z cirros-0.3.1-x86_64-uec ]]
+ '[' '' = bare ']'
+ KERNEL_ID=
+ RAMDISK_ID=
+ '[' -n '' ']'
+ '[' -n '' ']'
+ glance --os-auth-token --os-image-url http://172.17.32.22:9292 image-create --name cirros-0.3.1-x86_64-uec --public --container-format ami --disk-format ami
/opt/stack/devstack/functions: line 1247: : No such file or directory
++ failed
++ local r=1
+++ jobs -p
++ kill
++ set +o xtrace

while checking manually in functions file at line 1247, it seems that glance is trying to create image for container but it is not able to locate actual image at its place.

Here is /opt/stack/devstack/functions: line 1247 :

glance --os-auth-token $token --os-image-url http://$GLANCE_HOSTPORT image-create --name "${IMAGE_NAME%.img}" --public --container-format ami --disk-format ami ${KERNEL_ID:+--property kernel_id=$KERNEL_ID} ${RAMDISK_ID:+--property ramdisk_id=$RAMDISK_ID} < "${IMAGE}"

Do we have any workaround for the same ?

Thanks!
Abhishek

[Harika Vakadi (harika-vakadi)]

Yes Abhishek, I have the same work around

[adapaka bhavaniprasad (adapaka-prasad)]

I am also facing the same issue.

[Jerry Zhao (zhaoxinyu)]

same here when running tempest neutron test with havana trunk code. not every time though.

[Jerry Zhao (zhaoxinyu)]

It seems to me that the image file is corrupted or something during download.

2013-10-10 10:58:26 + tar -zxf /opt/stack/new/devstack/files/cirros-0.3.1-x86_64-uec.tar.gz -C /opt/stack/new/devstack/files/images/cirros-0.3.1-x86_64-uec
2013-10-10 10:58:26
2013-10-10 10:58:26 gzip: stdin: not in gzip format
2013-10-10 10:58:26 tar: Child returned status 1
2013-10-10 10:58:26 tar: Error is not recoverable: exiting now

In a successful build, the log looks like this:
2013-10-10 01:17:55 2013-10-10 01:17:55 (8.62 MB/s) - `/opt/stack/new/devstack/files/cirros-0.3.1-x86_64-uec.tar.gz' saved [8633554/8633554]

In a failed build, however, looks like this:
2013-10-10 10:58:26 2013-10-10 10:58:26 (123 MB/s) - `/opt/stack/new/devstack/files/cirros-0.3.1-x86_64-uec.tar.gz' saved [1209/1209]

It is obvious that the download was interrupted.

[Jerry Zhao (zhaoxinyu)]

It is a network issue causing image files incomplete. Thus, image can not be created in glance. I would check network conditions if devstack is in a private environment or recheck/reverify in review.o.o to give the test a second chance.

In the long run, I think a md5 check and retry logic is needed in devstack.

[Jerry Zhao (zhaoxinyu)]

Just found out that the root cause is cirros image source site hostname is resolved to 82.98.86.161, which returned wrong content (both length and type).

Below is the failure log, please note the content length and type:

2013-10-10 23:47:53 + wget -c http://download.cirros-cloud.net/0.3.1/cirros-0.3.1-x86_64-uec.tar.gz -O /opt/stack/new/devstack/files/cirros-0.3.1-x86_64-uec.tar.gz
2013-10-10 23:47:53 --2013-10-10 23:47:42-- http://download.cirros-cloud.net/0.3.1/cirros-0.3.1-x86_64-uec.tar.gz
2013-10-10 23:47:53 Resolving download.cirros-cloud.net (download.cirros-cloud.net)... 69.163.202.251
2013-10-10 23:47:53 Connecting to download.cirros-cloud.net (download.cirros-cloud.net)|69.163.202.251|:80... connected.
2013-10-10 23:47:53 HTTP request sent, awaiting response... 302 Found
2013-10-10 23:47:53 Location: http://cdn.download.cirros-cloud.net/0.3.1/cirros-0.3.1-x86_64-uec.tar.gz [following]
2013-10-10 23:47:53 --2013-10-10 23:47:42-- http://cdn.download.cirros-cloud.net/0.3.1/cirros-0.3.1-x86_64-uec.tar.gz
2013-10-10 23:47:53 Resolving cdn.download.cirros-cloud.net (cdn.download.cirros-cloud.net)... 82.98.86.161, 2600:5801:8001:1::d112:2e2a, 2600:5801:8001:1::d112:2e51
2013-10-10 23:47:53 Connecting to cdn.download.cirros-cloud.net (cdn.download.cirros-cloud.net)|82.98.86.161|:80... connected.
2013-10-10 23:47:53 HTTP request sent, awaiting response... 200 OK
2013-10-10 23:47:53 Length: 1209 (1.2K) [text/html]
2013-10-10 23:47:53 Saving to: `/opt/stack/new/devstack/files/cirros-0.3.1-x86_64-uec.tar.gz'
2013-10-10 23:47:53
2013-10-10 23:47:53 0K . 100% 135M=0s

And here is the correct source:

2013-10-11 03:48:10 + wget -c http://download.cirros-cloud.net/0.3.1/cirros-0.3.1-x86_64-uec.tar.gz -O /opt/stack/new/devstack/files/cirros-0.3.1-x86_64-uec.tar.gz
2013-10-11 03:48:10 --2013-10-11 03:48:10-- http://download.cirros-cloud.net/0.3.1/cirros-0.3.1-x86_64-uec.tar.gz
2013-10-11 03:48:10 Resolving download.cirros-cloud.net (download.cirros-cloud.net)... 69.163.202.251
2013-10-11 03:48:10 Connecting to download.cirros-cloud.net (download.cirros-cloud.net)|69.163.202.251|:80... connected.
2013-10-11 03:48:10 HTTP request sent, awaiting response... 302 Found
2013-10-11 03:48:10 Location: http://cdn.download.cirros-cloud.net/0.3.1/cirros-0.3.1-x86_64-uec.tar.gz [following]
2013-10-11 03:48:10 --2013-10-11 03:48:10-- http://cdn.download.cirros-cloud.net/0.3.1/cirros-0.3.1-x86_64-uec.tar.gz
2013-10-11 03:48:10 Resolving cdn.download.cirros-cloud.net (cdn.download.cirros-cloud.net)... 67.131.38.42, 67.131.38.17, 2600:809:600::3f50:43b, ...
2013-10-11 03:48:10 Connecting to cdn.download.cirros-cloud.net (cdn.download.cirros-cloud.net)|67.131.38.42|:80... connected.
2013-10-11 03:48:10 HTTP request sent, awaiting response... 200 OK
2013-10-11 03:48:10 Length: 8633554 (8.2M) [application/x-tar]
2013-10-11 03:48:10 Saving to: `/opt/stack/new/devstack/files/cirros-0.3.1-x86_64-uec.tar.gz'

Another right source:

2013-10-10 23:47:04 + wget -c http://download.cirros-cloud.net/0.3.1/cirros-0.3.1-x86_64-uec.tar.gz -O /opt/stack/new/devstack/files/cirros-0.3.1-x86_64-uec.tar.gz
2013-10-10 23:47:04 --2013-10-10 23:46:58-- http://download.cirros-cloud.net/0....

[Jerry Zhao (zhaoxinyu)]

Convert back to bug because the cirros image source is so flaky recently and causes many gate rechecks.
I think there better be a checksum mechanism or check content types(html content on failure instead of gzip type on success) to detect and retry for image download. Even a local mirror for the cirros image also works.

[Jerry Zhao (zhaoxinyu)]

The CDN for cirros image is not the root cause. In some cases, the source site download.cirros-cloud.net is also resolved as 82.98.86.161, which sometimes is an IP result of the CNAME CDN site cdn.download.cirros-cloud.net.

2013-10-16 18:15:56 + IMAGE_FNAME=cirros-0.3.1-x86_64-uec.tar.gz
2013-10-16 18:15:56 + [[ ! -f /opt/stack/new/devstack/files/cirros-0.3.1-x86_64-uec.tar.gz ]]
2013-10-16 18:15:56 + wget -c http://download.cirros-cloud.net/0.3.1/cirros-0.3.1-x86_64-uec.tar.gz -O /opt/stack/new/devstack/files/cirros-0.3.1-x86_64-uec.tar.gz
2013-10-16 18:15:56 --2013-10-16 18:15:21-- http://download.cirros-cloud.net/0.3.1/cirros-0.3.1-x86_64-uec.tar.gz
2013-10-16 18:15:56 Resolving download.cirros-cloud.net (download.cirros-cloud.net)... 82.98.86.161
2013-10-16 18:15:56 Connecting to download.cirros-cloud.net (download.cirros-cloud.net)|82.98.86.161|:80... connected.
2013-10-16 18:15:56 HTTP request sent, awaiting response... 200 OK
2013-10-16 18:15:56 Length: 1209 (1.2K) [text/html]
2013-10-16 18:15:56 Saving to: `/opt/stack/new/devstack/files/cirros-0.3.1-x86_64-uec.tar.gz'
2013-10-16 18:15:56
2013-10-16 18:15:56 0K . 100% 159M=0s

[Dean Troyer (dtroyer)]

Is there a link to a known failed gate check due to this? If it is intermittent we should get it added to the elastic-recheck list.

[Launchpad Janitor (janitor)]

[Expired for devstack because there has been no activity for 60 days.]