[New charm] Ubuntu repository cache for cloud partners

[Robert C Jennings]

This charm provides a partial caching proxy mirror of the
Ubuntu Software Repository. This is intended for deployment in cloud
environments to provide a cloud-local repository.

This is a hybrid mirror / cache. Repository metadata, data under the
ubuntu/dists/ directory, is copied from upstream Ubuntu mirrors and
checked to ensure that it is consistent. Requests for package files in
/ubuntu/pool are forwarded internally to squid-deb-proxy, which keeps a
local cache of .deb files as they are requested. This approach minimized
load on the upstream archive server, improves performance, and requires
less disk space than a static archive mirror.

[Robert C Jennings (rcj)]

Code branch @ lp:~rcj/charms/trusty/ubuntu-repository-cache/trunk. The TODO file lists the outstanding work for the charm, but the charm is quite functional and ready for review. The BUGS file lists a couple known issues that are being worked, but they should not be blockers.

[Robert C Jennings (rcj)]

Deploying this charm to test will pull a lot of data, to reduce the content mirrored during testing you can apply patches/mirror-0001-devel_reduce_dists.patch.

[Whit Morriss (whitmo)]

Robert, thanks for the great looking charm. Nice use of charm helpers, very clean and well organized w/ good if incomplete docs. All in all well written and useful charm.

Couple, three nits to pick:

Tests: Currently the amulet tests fail when running under bundletester (gives a 404 when the test attempts an http connection). Also consider the amount of python you have in the hooks, I'd like to see a bit more unit coverage.

Docs: "juju add-relationship" needs to be changed to "juju add-relation" (make this mistake all the time myself). Also I noted a number of "hooks" intended for running via juju run such as code to kick off the mirror, but do not see these documented.

so -1 for now, but looks really good. Thanks again Robert!

-w

[Whit Morriss (whitmo)]

Hi robert, also ran into this error adding the relations:

"juju add-relation ntp ubuntu-repository-cache
ERROR cannot add relation "ntp:juju-info ubuntu-repository-cache:juju-info": principal and subordinate services' series must match"

we may need to get a version of the ntp subordinate promulgated for trusty?

-w

[Robert C Jennings (rcj)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 09/15/2014 01:42 PM, Whit Morriss wrote:
> Hi robert, also ran into this error adding the relations:
>
> "juju add-relation ntp ubuntu-repository-cache ERROR cannot add
> relation "ntp:juju-info ubuntu-repository-cache:juju-info":
> principal and subordinate services' series must match"
>
> we may need to get a version of the ntp subordinate promulgated
> for trusty?

Thanks for the review. I'll work on addressing issues.
There is a recommended NTP subordinate (cs:trusty/ntp-6), could your
default series be causing issues?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBCgAGBQJUGCrSAAoJEHQMPZ7t8u1zBLUP/0hWZli7gdfskCZRqsaMfpk+
NBvUvtKnYWvP1/wkJFtqU+nVKysuN/DFbGR2tyJz6iJ1PgsNI8kjlmNlZLnsDwBj
rGKXxrgxBz8NfCKiS9NgabJHwB/uzxFrNVlZbkc41QsRsPYQGBMyBZkni5lFuzea
tKYSGMkfzBZLsHQ4pRZGj4DFyu8LfWKSp+KTj+Q2AtC0XwaybbEYSsjdP61WVwix
xSz6lI0H3iUHsBUvD99WbkInbvRblL0MA2RBiBSi9pnZI4sEXsRwUIEbFwtv5f8G
CZ8Wgnh+Ytb75V6/6fkRT/Fib4kL5HGqMPmCc9N8bDv7MLD/caLGMwnmquVQKwcp
SXWIVHcqMfy8nYm1EUxE+jDW3Unkf41lc5w1Mvy7Xy4aWiHerYoW5xMoC1b2M+7x
Qj9spPetvQm9RTlpFINTXdcZtSaKuDpmQLZ8x+4CEwYgxCmqjoeHOKwaZbIYp8kJ
8H3s2BNF/OYAaV4qZx48uwyvj/XKvbNYuWRMPlybkj52J+g5iwD7Wj0UeioqRQ1v
ppRY9UEmWcOga+8La3a63KqGqTVp7usiJc0Ghjge2iocaARhNPwQJhndfHTJT3h0
4kr9qMFl51y3LoCuuQbTQLiIySFy2hjFShjM78kxjwCCRT5NKti8m7mCLi37/ZhQ
jIJhfoCrw65Y1NNzdGBj
=hscK
-----END PGP SIGNATURE-----

[Whit Morriss (whitmo)]

Thanks Robert, and sorry about the delayed response.
LGTM +1.

I was able to stand it up and connect via http to a running repostory. I still would prefer to see some unit tests, but nice to see the integration tests working.

Thanks again,

-w

[Ben Howard (utlemming)]

After deploying to Azure, I get:

Ign http://juju-azure-8w2jyvuidv.cloudapp.net trusty-updates/universe Translation-en
W: Failed to fetch http://juju-azure-8w2jyvuidv.cloudapp.net/ubuntu/dists/trusty/main/source/Sources 503 Service Unavailable

W: Failed to fetch http://juju-azure-8w2jyvuidv.cloudapp.net/ubuntu/dists/trusty/universe/source/Sources 503 Service Unavailable

W: Failed to fetch http://juju-azure-8w2jyvuidv.cloudapp.net/ubuntu/dists/trusty/main/binary-amd64/Packages 503 Service Unavailable

W: Failed to fetch http://juju-azure-8w2jyvuidv.cloudapp.net/ubuntu/dists/trusty/universe/binary-amd64/Packages 503 Service Unavailable

W: Failed to fetch http://juju-azure-8w2jyvuidv.cloudapp.net/ubuntu/dists/trusty-updates/main/source/Sources 503 Service Unavailable
W: Failed to fetch http://juju-azure-8w2jyvuidv.cloudapp.net/ubuntu/dists/trusty-updates/universe/source/Sources 503 Service Unavailable
W: Failed to fetch http://juju-azure-8w2jyvuidv.cloudapp.net/ubuntu/dists/trusty-updates/main/binary-amd64/Packages 503 Service Unavailable
W: Failed to fetch http://juju-azure-8w2jyvuidv.cloudapp.net/ubuntu/dists/trusty-updates/universe/binary-amd64/Packages 503 Service Unavailable
E: Some index files failed to download. They have been ignored, or old ones used instead.

[Ben Howard (utlemming)]

Looks like after leaving the instances overnight things started to work. After chatting offline the 503's are caused by setting sync-on-start to false.

[Robert C Jennings (rcj)]

This charm is rolling out into production with a few additions based on IS requirements. We would like to provide this to our partners as well through the charm store. This had been blocked on an amulet issue in bug #1430488 which was blocking testing. With the patches for that bug applied to amulet, testing of this charm should work.

Please re-review this charm for charmstore inclusion, thanks.

[Antonio Rosales (arosales)]

Per comment 9 @rcj is looking for a re-review of this charm. Thus, I am moving the status of this bug to "Fix Committed" so it shows up in the the charm review queue @ http://review.juju.solutions/

-thanks,
Antonio

[Adam Israel (aisrael)]

Hi Robert,

That amulet bug should be fixed in amulet 1.10.1, cut a week ago. I did a full review of the charm during that testing, with the patched amulet. I was able to fully deploy and run all tests successfully. You've got a +1 from me.

[Tim Van Steenburgh (tvansteenburgh)]

Looking through the logs of the latest test runs (http://reports.vapour.ws/charm-test-details/charm-bundle-test-parent-245), I see that all failures seem to stem from the python-jujuclient losing the websocket connection to the juju api server. Looking into the all-machines log for each provider, I found something similar in each. Here's the log from the HP cloud (start reading on line 721):

http://pastebin.ubuntu.com/11098434/

Sure looks like something bad is happening, but I can't diagnose beyond that...

[Tim Van Steenburgh (tvansteenburgh)]

Opened bug against juju-core: https://bugs.launchpad.net/juju-core/+bug/1454359

[Antonio Rosales (arosales)]

Given we are hitting a Juju Core bug (see comment 13) AdamI and RobertJ have manually tested this charm and tests have passed.

Since code reviews and manually testing have been successful suggesting to the ~charmers team to promote this charm to recommended.

-thanks,
Antonio

[Tim Van Steenburgh (tvansteenburgh)]

Charm promulgated, thanks rcj!