setting up system proxy via cloud-config might be nice

Bug #1089405 reported by Scott Moser on 2012-12-12
74
This bug affects 15 people
Affects Status Importance Assigned to Milestone
cloud-init
Wishlist
Unassigned

Bug Description

as seen in bug 1089389 it might be nice to set system wide proxy information. There seems to be reasonable explanation of how you'd do that at [1].
Config could look like:

system_proxy:
 http_proxy: http://foo.bar:3128
 https_proxy: https://foo.host:3128
 ftp_proxy: ...
 no_proxy: ...

cloud-init would/should take just the lower case and write /etc/environment correctly.

Some tricky things here:
 a.) cloud-init should then adjust its own environment to use these settings (and subprocesses)
 b.) what about the datasource? ie, should / would access to http://169.254.169.254 be proxied?

[1] http://askubuntu.com/questions/150210/how-do-i-set-systemwide-proxy-servers-in-xubuntu-lubuntu-or-ubuntu-studio

Tags: sts Edit Tag help
Scott Moser (smoser) on 2012-12-12
Changed in cloud-init:
status: New → Triaged
importance: Undecided → Wishlist
description: updated
Geronimo Orozco (gorozco) wrote :

+1 add repo behind proxy or installing packages is a nightmare

Dustin Kirkland  (kirkland) wrote :

Please. I need this.

Jeroen T. Vermeulen (jtv) wrote :

Given the comments, can we raise the priority on this one?

+1
Deploying a machine with a private IP only (EC2/VPC) would benefit from this

Maitreya Ranganath (maitreya) wrote :

+1

However, I found a workaround to proceed without these properties within cloud-init.
Setting environment variables within /etc/profile or /etc/environment did not work since cloud-init is started by init(8) and does not inherit these variables.
I had to use the config files in /etc/init/cloud-*.conf to set the relevant environment variables before invoking the cloud init scripts.

For example in /etc/init/cloud-config.conf:

# cloud-config - Handle applying the settings specified in cloud-config
description "Handle applying cloud-config"

start on (filesystem and started rsyslog)
console output
task

env HTTP_PROXY=http://x.y.z.w:8080/
env HTTPS_PROXY=http://x.y.z.w:8080/
env NO_PROXY=169.254.169.254

env http_proxy=http://x.y.z.w:8080/
env https_proxy=http://x.y.z.w:8080/
env no_proxy=169.254.169.254

export HTTP_PROXY
export HTTPS_PROXY
export NO_PROXY
export http_proxy
export https_proxy
export no_proxy

exec cloud-init-cfg all config

This set up the environment for all the scripts executed by cloud-init, including curl, wget, apt-get. I was able to successfully test this in AWS EC2 in a purely private Subnet and sending requests to the Internet via my on-prem proxy server. The 'no_proxy' environment variables are to allow the AWS CLI and SDK to retrieve and use data and credentials from the EC2 instance meta-data service.

Sam Stoelinga (sammiestoel) wrote :

+1 for priority raise

attaboystrong (attaboystrong) wrote :

Adding our experience based on Maitreya Ranganath (maitreya)'s answer:

The file /etc/init.d/cloud-config had below entries.

# If there exist sysconfig/default variable override files use it...
[ -f /etc/sysconfig/cloud-init ] && . /etc/sysconfig/cloud-init
[ -f /etc/default/cloud-init ] && . /etc/default/cloud-init

so we created /etc/default/cloud-init that contains below proxy variables.
export HTTP_PROXY=http://x.y.z.w:8080/
export HTTPS_PROXY=http://x.y.z.w:8080/
export http_proxy=http://x.y.z.w:8080/
export https_proxy=http://x.y.z.w:8080/

As a result, the file /etc/default/cloud-init is executed when /etc/init.d/cloud-config is executed during boot. This provides access to internet to be able to download stuff required in clout-init process.

Graham Leggett (minfrin-y) wrote :

In reference to "so we created /etc/default/cloud-init that contains below proxy variables" - how did you do that?

I am assuming you didn't use cloud-init to do it, or did you?

If not, what did you use?

Peter Schroeter (schroeter) wrote :

+1 for priority raise also, have bumped into this issue a couple times now. I also think this feature would be quite useful as well. VPCs and other locked downed environments are only going to increase in popularity so it'd useful having an easy way of pulling down scripts/installing packages in that scenario via cloud-config.

This was also an issue in our case - but there's a solution.
In case of CentOS 7.1 and yum updates this is one way:

#cloud-config
runcmd:
 - echo "proxy=http://proxy.example.com:8080" >> /etc/yum.conf
 - echo proxy_username=user123 >> /etc/yum.conf
 - echo proxy_password=pwd456 >> /etc/yum.conf
 - echo "http_proxy=http://user123:<email address hidden>:8080" > /etc/profile.d/proxy.sh && chmod 755 /etc/profile.d/proxy.sh
 - echo "https_proxy=http://user123:<email address hidden>:8080" >> /etc/profile.d/proxy.sh
 - echo "no_proxy=localhost,127.0.0.1,169.254.169.254,.sock,.example.com" >> /etc/profile.d/proxy.sh
 - yum update -y
...

Sorry, in my previous script the "export" keyword was missing for http_proxy, https_proxy and no_proxy:
...
 - echo "export http_proxy=http://user123:<email address hidden>:8080" > /etc/profile.d/proxy.sh && chmod 755 /etc/profile.d/proxy.sh
 - echo "export https_proxy=http://user123:<email address hidden>:8080" >> /etc/profile.d/proxy.sh
 - echo "export no_proxy=localhost,127.0.0.1,169.254.169.254,.sock,.example.com" >> /etc/profile.d/proxy.sh
...

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

An alternative might be to use a boot hook.

#cloud-boothook
#!/bin/sh

#### Ubuntu/Debian

# Add proxies to Apt config
cat > /etc/apt/apt.conf.d/00-proxy <<EOC
Acquire::http { Proxy "http://proxy.example.com:8080"; };
Acquire::https { Proxy "http://proxy.example.com:8080"; };
Acquire::http::Pipeline-Depth "23";
Acquire::Languages "none";
EOC

# Add it to the environment
cat >> /etc/environment <<EOC
http_proxy="http://myproxy.server.com:8080/"
https_proxy="http://myproxy.server.com:8080/"
ftp_proxy="http://myproxy.server.com:8080/"
no_proxy="localhost,127.0.0.1,localaddress,.localdomain.com"
HTTP_PROXY="http://myproxy.server.com:8080/"
HTTPS_PROXY="http://myproxy.server.com:8080/"
FTP_PROXY="http://myproxy.server.com:8080/"
NO_PROXY="localhost,127.0.0.1,localaddress,.localdomain.com"
EOC

# YUM
cat > /etc/yum.conf <<EOC
proxy=http://proxy.example.com:8080"
echo proxy_username=user123
proxy_password=pwd456
EOC
-----BEGIN PGP SIGNATURE-----

iQIbBAEBCgAGBQJWN3EQAAoJEETWil5UBqhmIhYP9RuvBvysYbP4rXVa01UkWJQq
pQOa3gg1Mp2izTz6mQHCDhOWitNGLy2/bTW1y5mq2XXC/P+EUyy0FLX3YmXObUSa
hnEy6QCkHI1SAwqPCkSOE2dgcWQE3//8AW6xb4uTA+klaEqQAZ5Npa+WqXZlRFkU
+o8EEhExGEUn/E983wmYL6HABMHEVc52RX4THi8WsIkcHkRnoipu1wPnyP0EwBdI
uioMLyEHa7eWRp6BKQN6Gol7ElDMGccKdjxMb2mh0zk9Jo8ckK/IDQZvAAmpbqrB
mmNSiNd7vxE9Qu98I2sxXqQNkMCyLxL1Z6zXwoAQK4MRz1BBEJxckpKJCkVqbNL9
rEgtsl+GOBCEVk8ukyMipH9l+bhPj49tfn+jkHjotI9Dll4ajXF+TuSSpx/1gXAM
/i3uRGiWEzx85ZMSSsYBn+X7XnYtzQgMLMB84+JKGkEwcS/3beZfGQeemrziEHc8
CaBjV2gM6SXJbHr95RNJyULD9o6UzRzt63eSMPfnacYSFuHmCn2k3HMxM6xumwwK
HqiZEKMRRdVupDowzxQ+yLeDZG5nXa4wKDuPFHgFqBRDLgfvoakmuSGoJ3Eb6cde
A4VU3HyZjEN8DBiiF9HVpQXjkjefpnnIGCFkZV3fybY85bLROsKa7AuQveRc0MJ3
eT2lFZkKHQ/qOU200zg=
=SlyK
-----END PGP SIGNATURE-----

tags: added: sts

At the root of this issue is the inability to define a global environment variable for cloud-init.

A patchwork of workarounds can be made, but as long as we can't define an environment variable that is inherited by all processes spawned by cloud-init, there will always be an issue waiting to happen.

Scott Moser (smoser) wrote :

I'll say that I would accept patches for setting up environment for cloud-init.
Something in cloud-config that just did:

environment:
 env_name1: env_name2

and then cloud-init on start up would just read those and basically:
  os.environ.update(cfg.get(environment), {})

or something to that affect.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers