| 2013-10-17 21:03:02 |
Scott Moser |
description |
Since 2013-10-11 we started seeing errors running the MAAS integration tests for saucy[1].
The checksum of the downloaded image wasn't matching the checksum expected[2]. maas-import-pxe-files failed with the following traceback:
Traceback (most recent call last):
File "/usr/sbin/maas-import-ephemerals", line 26, in <module>
main(args)
File "/usr/lib/python2.7/dist-packages/provisioningserver/import_images/ephemerals_script.py", line 450, in main
target.sync(source, args.path)
File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/__init__.py", line 85, in sync
return self.sync_index(reader, path, data, content)
File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/__init__.py", line 230, in sync_index
self.sync(reader, path=epath)
File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/__init__.py", line 83, in sync
return self.sync_products(reader, path, data, content)
File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/__init__.py", line 304, in sync_products
self.insert_item(item, src, target, pgree, ipath_cs)
File "/usr/lib/python2.7/dist-packages/provisioningserver/import_images/ephemerals_script.py", line 249, in insert_item
data, src, target, pedigree, contentsource)
File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/__init__.py", line 388, in insert_item
size=data.get('size'))
File "/usr/lib/python2.7/dist-packages/simplestreams/objectstores/__init__.py", line 147, in insert
cksum.hexdigest(), cksum.expected))
Exception: unexpected checksum 'sha256' on precise/release-20131010/precise-daily-maas-amd64.tar.gz (found: 3d99e069e34d031f924689a3c28ea43d4046d8e9a79a2ea9014d35bcbcf57a16 expected: 2c7ab801ffd7e8673b77be828335bac29538f102dbc6baacd4ccc4c0d309a11b)
}}}
Initially it was thought this could be caused by a misconfigured cache in the QA lab, but after re-installing and reconfiguring (and checking with wget that the cache was returning the correct objects for a given URL, as well as checking the checksum of the file matched the one in the json file), the error was still present.
Further investigation[3] raised the suspicion that the downloaded image is being handled by multiple processes. Turns out the code for simplestreams and import_ephemerals.py is not multi process safe, and multiple import-ephemerals processes working with the same datadir can cause problems like this.
[1] http://10.189.74.2:8080/view/MAAS/job/saucy-adt-maas-daily/220/ARCH=amd64,label=lenovo-RD230-01/console
[2] http://10.189.74.2:8080/view/MAAS/job/saucy-adt-maas-manual/95/ARCH=amd64,label=lenovo-RD230-01/console
[3] http://pastebin.ubuntu.com/6247125/ |
== Begin SRU Info ==
[Impact]
maas-import-ephemerals is the program that is executed to download images from http://maas.ubuntu.com which are then used for the commissioning, enlistment, and curtin install environment.
This bug causes that download to fail in most cases with an http proxy between the user and maas.ubuntu.com. It is also reported to fail under cases where there is no http_proxy set (but may be a proxy in between).
[Test Case]
The bug could be reproduced by simply doing:
PROXY=http://localhost:3128 # or some valid proxy
http_proxy=$PROXY python /usr/bin/sstream-mirror -vvv http://maas.ubuntu.com/images/ephemeral/daily/streams/v1/index.json out.d arch=amd64 release=quantal
You would see the download start and then hang. A suitable proxy can be installed just by 'apt-get install squid3' and then using http://localhost:3128.
[Regression Potential]
Regression potential should be very small. Previously each connection we made was doing an open and close and then open again. This was wreaking havoc with urllib3 as used by python-requests. Now, we're only opening and closing a single time during the mirror, and even then,
a.) the url that is being downloaded will change (by having a '/' inserted into its path) where previously it would be the same url
b.) step 'a' will only occur if the base url provided by the user did not already end in '/'.
== End SRU Info ==
Since 2013-10-11 we started seeing errors running the MAAS integration tests for saucy[1].
The checksum of the downloaded image wasn't matching the checksum expected[2]. maas-import-pxe-files failed with the following traceback:
Traceback (most recent call last):
File "/usr/sbin/maas-import-ephemerals", line 26, in <module>
main(args)
File "/usr/lib/python2.7/dist-packages/provisioningserver/import_images/ephemerals_script.py", line 450, in main
target.sync(source, args.path)
File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/__init__.py", line 85, in sync
return self.sync_index(reader, path, data, content)
File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/__init__.py", line 230, in sync_index
self.sync(reader, path=epath)
File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/__init__.py", line 83, in sync
return self.sync_products(reader, path, data, content)
File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/__init__.py", line 304, in sync_products
self.insert_item(item, src, target, pgree, ipath_cs)
File "/usr/lib/python2.7/dist-packages/provisioningserver/import_images/ephemerals_script.py", line 249, in insert_item
data, src, target, pedigree, contentsource)
File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/__init__.py", line 388, in insert_item
size=data.get('size'))
File "/usr/lib/python2.7/dist-packages/simplestreams/objectstores/__init__.py", line 147, in insert
cksum.hexdigest(), cksum.expected))
Exception: unexpected checksum 'sha256' on precise/release-20131010/precise-daily-maas-amd64.tar.gz (found: 3d99e069e34d031f924689a3c28ea43d4046d8e9a79a2ea9014d35bcbcf57a16 expected: 2c7ab801ffd7e8673b77be828335bac29538f102dbc6baacd4ccc4c0d309a11b)
}}}
Initially it was thought this could be caused by a misconfigured cache in the QA lab, but after re-installing and reconfiguring (and checking with wget that the cache was returning the correct objects for a given URL, as well as checking the checksum of the file matched the one in the json file), the error was still present.
Further investigation[3] raised the suspicion that the downloaded image is being handled by multiple processes. Turns out the code for simplestreams and import_ephemerals.py is not multi process safe, and multiple import-ephemerals processes working with the same datadir can cause problems like this.
[1] http://10.189.74.2:8080/view/MAAS/job/saucy-adt-maas-daily/220/ARCH=amd64,label=lenovo-RD230-01/console
[2] http://10.189.74.2:8080/view/MAAS/job/saucy-adt-maas-manual/95/ARCH=amd64,label=lenovo-RD230-01/console
[3] http://pastebin.ubuntu.com/6247125/ |
|
