juju should support an apt proxy or alternate mirror for private clouds

[Kapil Thangavelu]

private clouds / data centers may not allow public internet access, being able to configure an environment property defining a apt mirror/proxy for instances in the cloud would alleviate this. We should be able to put this into place via cloud-init.

[Scott Moser (smoser)]

I do not think that juju is the correct place to "support an apt proxy".
Just as cloud-init selects a mirror correctly, JuJu should let cloud-init do apt proxy configuration (if applicable).

The most generic solution for this is to allow users to provide user-data parts to cloud-init.

Then, to support apt proxy selection, the user (with no specific juju support) could add a boothook like:
#!/bin/sh
echo 'Acquire { HTTP { Proxy "http://my-local-proxy:3128/"; }; };' > /etc/apt/apt.conf.d/local-proxy

I also just opened cloud-init bug 897688, which will make specification of apt_proxy supported via cloud-config.

[Kapil Thangavelu (hazmat)]

Hi Scott,

The goal is to support this via cloud-init, with juju just passing cloud init the right options. ie. we want to use cloud-init support for this functionality, and just populate user-data appropriately if the option is specified on the juju environment. Juju doesn't expose cloud-init to end users, so it has to take care of configuring it as needed.

[Scott Moser (smoser)]

I'm suggesting that JuJu should allow the end user to specify cloud-init "parts" or a single multi-part file that would then be passed through either prior to the juju generated part or post it (that would have to be thought through) .

This would provide a generic method for local customizations that are going to be necessary.

[Gustavo Niemeyer (niemeyer)]

As we discussed, this should be a first class option, for a few reasons:

1) It's simpler. The interface for proxy selection should be:

        juju bootstrap --http-proxy=$URL

2) We want control over that URL.

    Even though we'll be using cloud-init for the moment, we'll also want to enable the user
    to *change* that URL with a command like "juju set-env http-proxy=$URL", and that should
    affect both the future machines, and the *current* machines, which cloud-init does not
    do.

With those said, at some point we may want to enable a generic mechanism for preparing
machines, but this bug isn't justification for it.

[Scott Moser (smoser)]

As of trunk at revision 486 [1], you can now have cloud-config like:
   apt_proxy: http://my.apt.proxy:3128

That will do what you'd expect.
[1] http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/revision/486

[Jorge (correajl)]

It will be very import to Juju allow users to specify some cloud_configs, as we can do creating instances with --user-data-file.

[Clint Byrum (clint-fewbar)]

Note that the case of ongoing-configurability can be handled fine with a subordinate "apt-proxy" charm that would be related to all services. So one way to get a lightweight implementation for this would just be to push apt_proxy: into cloud-init and then have an included apt-proxy subordinate charm that automatically gets related and pushed onto every node.

[Jamie Strandboge (jdstrand)]

Just to add my two cents-- I only just started playing with juju for its MIR review, but I'm betting others would want to test drive it too. As such, the LXC option is free of cost (and hassles if you don't have an amazon account already). Many enterprises will have a local mirror or apt proxy and having these options in juju would greatly speed up this trial period. I have to admit that I thought that juju was broken because it took so long for my units to come out of state pending. While this could be handled in documentation (though it would have to be everywhere-- and that is hard to control), for people with access to a local mirror or apt proxy, this would be a great speed-up and improve first impressions. I imagine it would also speed up live demonstrations.

[Clint Byrum (clint-fewbar)]

Kapil, propose merge now or forever hold your peace. I intend to drop this from the 0.6 milestone if its not in review by tomorrow morning.

[Данило Шеган (danilo)]

This is not very high on the roadmap for juju-core.

[Sidnei da Silva (sidnei)]

Looking at https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/974509 it seems like if the private cloud has a 'ubuntu-mirror' dns name, then cloud-init might pick that up as the mirror.

More specifically, for local provider with lxc, a ubuntu-mirror entry could be pushed to dnsmasq on the host such that it is picked up automatically.

[Sidnei da Silva (sidnei)]

In other news, while we could pick up the proxy from the host during cloud-init setup, it seems like cloud-init doesn't support setting an https proxy?

[Kapil Thangavelu (hazmat)]

fwiw this is a typical requirement in private clouds.

On Fri, Oct 11, 2013 at 7:52 PM, Curtis Hovey <email address hidden> wrote:

> ** Changed in: juju-core
> Importance: Medium => Low
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/897645
>
> Title:
> juju should support an apt proxy or alternate mirror for private
> clouds
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/cloud-init/+bug/897645/+subscriptions
>

[Curtis Hovey (sinzui)]

I think this feature needs to be solved for 14.04.

[Jonathan Davies (jpds)]

For private clouds, this can be done in MAAS and MAAS settings has all the options for this.

[Kapil Thangavelu (hazmat)]

how does one enable this behavior?

On Tue, Nov 12, 2013 at 7:16 PM, Dave Cheney <email address hidden>wrote:

> ** Changed in: juju-core
> Status: Triaged => Fix Released
>
> ** Changed in: juju-core
> Milestone: None => 1.17.0
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/897645
>
> Title:
> juju should support an apt proxy or alternate mirror for private
> clouds
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/cloud-init/+bug/897645/+subscriptions
>

[Dave Cheney (dave-cheney)]

If the machine you are bootstrapping on uses the apt proxy, we
magically sniff those settings and pass them to the environment via
cloud init.

On Thu, Nov 14, 2013 at 2:23 AM, Kapil Thangavelu
<email address hidden> wrote:
> how does one enable this behavior?
>
>
> On Tue, Nov 12, 2013 at 7:16 PM, Dave Cheney <email address hidden>wrote:
>
>> ** Changed in: juju-core
>> Status: Triaged => Fix Released
>>
>> ** Changed in: juju-core
>> Milestone: None => 1.17.0
>>
>> --
>> You received this bug notification because you are subscribed to the bug
>> report.
>> https://bugs.launchpad.net/bugs/897645
>>
>> Title:
>> juju should support an apt proxy or alternate mirror for private
>> clouds
>>
>> To manage notifications about this bug go to:
>> https://bugs.launchpad.net/cloud-init/+bug/897645/+subscriptions
>>
>
> --
> You received this bug notification because you are subscribed to juju-
> core.
> Matching subscriptions: MOAR JUJU SPAM!
> https://bugs.launchpad.net/bugs/897645
>
> Title:
> juju should support an apt proxy or alternate mirror for private
> clouds
>
> Status in Init scripts for use on cloud images:
> Fix Released
> Status in The Eilt project:
> New
> Status in pyjuju:
> Triaged
> Status in juju-core:
> Fix Released
>
> Bug description:
> private clouds / data centers may not allow public internet access,
> being able to configure an environment property defining a apt
> mirror/proxy for instances in the cloud would alleviate this. We
> should be able to put this into place via cloud-init.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/cloud-init/+bug/897645/+subscriptions

[Paul Gear (paulgear)]

The apt-*-proxy settings allow juju to work in egress-restricted environments, but it still does not pick up the mirror settings from the local host. In the parts of the world where we pay for bandwidth outside of our ISP's network, this is a very important setting to have.

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/2227