Track pip package download retries (note this often does not cause jobs to fail)

[Clark Boylan]

Pip will make up to 5 attempts to download a package. If it fails you get log messages like:

2017-08-04 13:43:05.623 | Retrying (Retry(total=4, connect=None, read=None, redirect=None)) after connection broken by 'ReadTimeoutError("HTTPConnectionPool(host='mirror.lon1.citycloud.openstack.org', port=8080): Read timed out. (read timeout=60.0)",)': /pypi/simple/sphinxcontrib-pecanwsme/

Because we will then retry 4 more times after this failure chances are that we eventually succeed. This means this error is often not fatal. But we want to track occurences of this as it may point to larger underlying network issues.

This bug is being created to track this specific mostly not fatal problem so that it is clear when looking at the elastic-recheck list that this bug while common is 1) likely not the cause of the actual failure so please do look into it more and 2) is not as high a priority as the bugs that actually cause failures per 1).

[Sorin Sbarnea (ssbarnea)]

We are currently using local mirrors in most places which accelerates and lower the chance of getting an error but even these do fail from time to time, causing lots of build failures.

Due to this I am working and implementing a fallback mecanism what should make pip look first at local mirror but also try to use official mirrors, so temporary glitches would not affect the builds.

One such change is at https://launchpad.net/bugs/1708686 but we need to test it better as there are some known side effects on how this may affect pip behavior.

Related:
* https://github.com/pypa/pip/issues/6045

[Sorin Sbarnea (ssbarnea)]

Correction: https://review.openstack.org/#/c/620630/