Cached copy of layers used on build

[Frode Nordahl]

When doing a charm build charm-tools is using cached copies of layers even though a more recent version of the layer is available in the upstream layer repository.

For example in the referenced patch series 5b0928f6695c18a21160ab47958c15efde88ac12 was selected from cache and not aa5bc57aea7e37d718da866082ed601230470bcb which was available upstream for ``layer-openstack`` at the time of build.

I do not know if this caching is occurring locally by charm-tools on the task executor or if it is operating behind a caching proxy server.

https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_smoke/openstack/charm-octavia-dashboard/677887/1/11752/consoleText.charm_build_30944.txt
https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_smoke/openstack/charm-nova-cell-controller/677905/1/11761/consoleText.charm_build_30953.txt
https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_smoke/openstack/charm-neutron-dynamic-routing/677906/1/11762/consoleText.charm_build_30954.txt
https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_smoke/openstack/charm-barbican-vault/677907/1/11763/consoleText.charm_build_30955.txt
https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_smoke/openstack/charm-masakari-monitors/677904/1/11759/consoleText.charm_build_30951.txt
https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_smoke/openstack/charm-barbican/677908/1/11764/consoleText.charm_build_30956.txt
https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_smoke/openstack/charm-ceph-rbd-mirror/677902/1/11758/consoleText.charm_build_30950.txt
https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_smoke/openstack/charm-designate/677911/1/11767/consoleText.charm_build_30959.txt
https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_smoke/openstack/charm-aodh/677910/1/11766/consoleText.charm_build_30958.txt
https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_smoke/openstack/charm-masakari/677909/1/11765/consoleText.charm_build_30957.txt
https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_smoke/openstack/charm-octavia-diskimage-retrofit/677913/1/11769/consoleText.charm_build_30961.txt
https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_smoke/openstack/charm-gnocchi/677912/1/11768/consoleText.charm_build_30960.txt
https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_smoke/openstack/charm-manila/677914/1/11770/consoleText.charm_build_30962.txt

[Frode Nordahl (fnordahl)]

I confirmed locally that if the layer is present in ~/.cache/charm/layer, charm-tools will use it without updating from upstream.

[Frode Nordahl (fnordahl)]

When building with a version of charm-tools with the unreleased https://github.com/juju/charm-tools/commit/1ffd2b3130690f588c03ee748d106d020e76db54 commit in it this would not have been a problem, as it cleans up after itself.

However, this would probably raise build concurrency issues like we see in bug 1733984.

I feel we need some more sandboxing around the build process with clean per instance caches for pip, charm-tools etc.