Comment 3 for bug 1464771

After much mucking about with all of this i found that old keys, stale apt cache and speed were all the enemy here. The container build process is running fast enough that its not guaranteed that networking is online within a container before running apt-get update/install commands. While "it should" work there are cases when a slow host can not get a container and its network up fast enough to deal with the running of online commands in a timely order. This causes random failures and in some cases apt simply hangs for a very long time before eventually giving up.

So to resolve this issue I broke up the container build task which contained a blob command to "do all the things" into smaller chunks. The change makes the role more idomatic ansible, its now easier to debug / read and provides deployers tags on a given task to isolate retry efforts.