Move the retry mechanism to the client

[Fernando Correa Neto]

In order to have the package reporter to retry the 'apt-get update' operation a few times before it gives up, the client must have a retry mechanism.

[Free Ekanayaka (free.ekanayaka)]

Note that if you refer to canonical.twisted.retry in the server, I'm not convinced it's a good choice. That machinery implements blind retries, and it's a workaround that we should eventually drop. That code also has the problem of using the Twisted reactor directly, instead of the wrapper that the client uses (landscape.reactor.Landscape.Reactor).

I'd probably go for implementing retries by hand for this specific case, using landscape.reactor.LandscapeReactor.call_later, and be sure to not blindly retry and instead check for expected error conditions (like the apt db is locked).

Later on we could see if we have enough use cases and enough data to modify canonical.twisted.retry with some information about the failures that should be retried, but that's something I'd not do now.

[Dean Henrichsmeyer (dean)]

As discussed in standup today, I agree with free. I prefer to implement a
by-hand work around for specifically the lock. No need for transient
retries to try and deal with things like network connectivity.

[Fernando Correa Neto (fcorrea)]

Changes were made in the client code so that it could retry upon receiving a error code 100, instead of blindly retrying.
It was landed in r808.