ppa will fail to install under proxy environment

[Yoshi Kadokawa]

I have properly configured juju-http(s)-proxy and no-proxy in the model-config,
however, it seems the adding ppa process does not use that value, and therefore fails to add the ppa with network timeout.

Traceback (most recent call last):
  File "/usr/bin/add-apt-repository", line 364, in <module>
    sys.exit(0 if addaptrepo.main() else 1)
  File "/usr/bin/add-apt-repository", line 347, in main
    shortcut = handler(source, **shortcut_params)
  File "/usr/lib/python3/dist-packages/softwareproperties/shortcuts.py", line 40, in shortcut_handler
    return handler(shortcut, **kwargs)
  File "/usr/lib/python3/dist-packages/softwareproperties/ppa.py", line 82, in __init__
    if self.lpppa.publish_debug_symbols:
  File "/usr/lib/python3/dist-packages/softwareproperties/ppa.py", line 120, in lpppa
    self._lpppa = self.lpteam.getPPAByName(name=self.ppaname)
  File "/usr/lib/python3/dist-packages/softwareproperties/ppa.py", line 107, in lpteam
    self._lpteam = self.lp.people(self.teamname)
  File "/usr/lib/python3/dist-packages/softwareproperties/ppa.py", line 98, in lp
    self._lp = login_func("%s.%s" % (self.__module__, self.__class__.__name__),
  File "/usr/lib/python3/dist-packages/launchpadlib/launchpad.py", line 494, in login_anonymously
    return cls(
  File "/usr/lib/python3/dist-packages/launchpadlib/launchpad.py", line 230, in __init__
    super(Launchpad, self).__init__(
  File "/usr/lib/python3/dist-packages/lazr/restfulclient/resource.py", line 472, in __init__
    self._wadl = self._browser.get_wadl_application(self._root_uri)
  File "/usr/lib/python3/dist-packages/lazr/restfulclient/_browser.py", line 447, in get_wadl_application
    response, content = self._request(url, media_type=wadl_type)
  File "/usr/lib/python3/dist-packages/lazr/restfulclient/_browser.py", line 389, in _request
    response, content = self._request_and_retry(
  File "/usr/lib/python3/dist-packages/lazr/restfulclient/_browser.py", line 359, in _request_and_retry
    response, content = self._connection.request(
  File "/usr/lib/python3/dist-packages/httplib2/__init__.py", line 1725, in request
    (response, content) = self._request(
  File "/usr/lib/python3/dist-packages/launchpadlib/launchpad.py", line 144, in _request
    response, content = super(LaunchpadOAuthAwareHttp, self)._request(
  File "/usr/lib/python3/dist-packages/lazr/restfulclient/_browser.py", line 184, in _request
    return super(RestfulHttp, self)._request(
  File "/usr/lib/python3/dist-packages/httplib2/__init__.py", line 1441, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/usr/lib/python3/dist-packages/httplib2/__init__.py", line 1363, in _conn_request
    conn.connect()
  File "/usr/lib/python3/dist-packages/httplib2/__init__.py", line 1153, in connect
    sock.connect((self.host, self.port))
TimeoutError: [Errno 110] Connection timed out

And it also does not support to configure mirrored repository, like passing the deb-line and the GPG key.

At the moment, there is no method to configure and install landscape in offline environment and behind proxy environment.

[Nobuto Murata (nobuto)]

At this revision, we lost an ability of installing from a mirrored repository:
https://git.launchpad.net/landscape-charm/commit/config.yaml?h=main&id=01f72f99b84fcf8e7c8eef81883abb12209e3461

Installation from a mirror server can happen where a child Landscape doesn't have internet access but to fetch packages from a parent Landscape in DMZ in some enterprise requirements.

[Nobuto Murata (nobuto)]

Subscribing ~field-high. Looks like we lost some of the abilities for customer deployment in the charm rewrite.

[Mitch Burton (mitchburton)]

I've tested with these additional juju model-config settings. Please confirm if you have these ones set as well, meaning apt-http-proxy, apt-https-proxy. My local proxy is at 10.76.244.153:8080 and I can confirm traffic crossing the proxy:

apt-ftp-proxy model http://10.76.244.153:8080/
apt-http-proxy model http://10.76.244.153:8080/
apt-https-proxy model http://10.76.244.153:8080/
apt-mirror default ""
apt-no-proxy default ""
ftp-proxy model http://10.76.244.153:8080/
http-proxy model http://10.76.244.153:8080/
https-proxy model http://10.76.244.153:8080/
no-proxy default 127.0.0.1,localhost,::1

Sample of traffic logs from proxy during juju deploy:

10.76.244.203 - - [06/Jul/2023:21:08:37 +0000] "CONNECT api.launchpad.net:443 HTTP/1.1" 200 215160 "-" "-"
10.76.244.203 - - [06/Jul/2023:21:08:45 +0000] "GET http://archive.ubuntu.com/ubuntu/pool/main/l/language-pack-en-base/language-pack-en-base_22.04%2b20230209_all.deb HTTP/1.1" 200 438391 "-" "Debian APT-HTTP/1.3 (2.4.9) non-interactive"

Of course, this is just testing with a proxy, not in a DMZ.

Installation succeeds and the landscape-server application reaches the status "Waiting on relations"

[Nobuto Murata (nobuto)]

The thing is setting http-proxy globally in one model has side effects. That's why juju-http-proxy was born to give a more granular control over what traffic should be proxied or not.

The legacy charm had http-proxy and https-proxy in the charm config itself so it was okay. And the new charm has http_proxy and https_proxy but I don't see an implementation in the charm to leverage those values at all.

So the issue here is there is no way to use proxy if we use juju-http-proxy instead of the legacy model global http-proxy which is full of side-effect.

[Nobuto Murata (nobuto)]

I cannot find the mention of "legacy" or "deprecated" in the official doc any longer somehow. But it's clearly mentioned as LegacyProxySettings.

https://github.com/juju/juju/blob/0cbfebb898b895ca95016df3ee09e6c105ed1b63/environs/config/config.go#L989-L1028

[Mitch Burton (mitchburton)]

Here's the settings I used. This works with the current edge version of the charm:

apt-ftp-proxy model http://10.76.244.153:8080/
apt-http-proxy model http://10.76.244.153:8080/
apt-https-proxy model http://10.76.244.153:8080/
apt-mirror default ""
apt-no-proxy default ""
ftp-proxy default ""
http-proxy default ""
https-proxy default ""
no-proxy default 127.0.0.1,localhost,::1
juju-ftp-proxy model http://10.76.244.153:8080/
juju-http-proxy model http://10.76.244.153:8080/
juju-https-proxy model http://10.76.244.153:8080/
juju-no-proxy default 127.0.0.1,localhost,::1

This is using both the apt-*-proxy settings and the juju-*-proxy settings and works fine for me. Please confirm (as I asked previously) that the apt-*-proxy settings are set, because these are used for add-apt-repository and other apt-related networking.

[Yoshi Kadokawa (yoshikadokawa)]

I believe that all of the necessary proxy configs are set in model-config.
Here are the proxy related configs I have set.

apt-http-proxy: http://192.168.1.28:8000
apt-https-proxy: http://192.168.1.28:8000
snap-http-proxy: http://192.168.1.28:8000
snap-https-proxy: http://192.168.1.28:8000
juju-http-proxy: http://192.168.1.28:8000
juju-https-proxy: http://192.168.1.28:8000
no-proxy: 127.0.0.1,localhost,192.168.1.0/24

We are deploying a full LMA stack in this model, which includes landscape, elasticsearch, graylog, grafana, prometheus, etc.
I can confirm other applications are succeeding with adding ppa and installing apt packages, so I believe the proxy configuration are okay.

[Nobuto Murata (nobuto)]

@mitchburton, thanks for looking into. We will mark this as Incomplete for the time being as we need a bit more testing on our side.

[Nobuto Murata (nobuto)]

Hmm, it was fairly straightforward to reproduce on a clean environment.

- MAAS provider (192.168.151.0/24 is the network managed by MAAS and 192.168.151.1 is MAAS)
- Juju 2.9.43-ubuntu-amd64
- landscape-dense-maas bundle from the charmstore

cat <<EOF | tee cloudinit-userdata.yaml
apt-http-proxy: http://192.168.151.1:8000/
apt-https-proxy: http://192.168.151.1:8000/
juju-http-proxy: http://192.168.151.1:8000/
juju-https-proxy: http://192.168.151.1:8000/
juju-no-proxy: 127.0.0.1,localhost,::1,192.168.151.0/24
snap-http-proxy: http://192.168.151.1:8000/
snap-https-proxy: http://192.168.151.1:8000/

cloudinit-userdata: |
  preruncmd:
  - ufw default allow # don't block anything by default
  - ufw reject out http # block outgoing 80/tcp connection without proxy
  - ufw reject out https # block outgoing 443/tcp connection without proxy
  - ufw enable
EOF

juju add-model landscape-test --config cloudinit-userdata.yaml

juju deploy landscape-dense-maas && time juju-wait -w

-> ERROR:root:landscape-server/0 failed: workload status is error

[juju debug-log -i landscape-server/0 --replay]
unit-landscape-server-0: 11:03:00 DEBUG unit.landscape-server/0.install Cannot add PPA: 'ppa:~landscape/ubuntu/self-hosted-beta'.
unit-landscape-server-0: 11:03:00 DEBUG unit.landscape-server/0.install ERROR: '~landscape' user or team does not exist.

$ juju ssh landscape-server/0 -- sudo ufw status verbose
Status: active
Logging: on (low)
Default: allow (incoming), allow (outgoing), deny (routed)
New profiles: skip

To Action From
-- ------ ----
80/tcp REJECT OUT Anywhere
443/tcp REJECT OUT Anywhere
80/tcp (v6) REJECT OUT Anywhere (v6)
443/tcp (v6) REJECT OUT Anywhere (v6)

[Nobuto Murata (nobuto)]

I also tried latest/edge landscape charm and it failed as well.

unit-landscape-server-0: 11:44:21 DEBUG unit.landscape-server/0.juju-log Initializing SQLite local storage: /var/lib/juju/agents/unit-landscape-server-0/charm/.unit-state.db.
unit-landscape-server-0: 11:44:21 DEBUG unit.landscape-server/0.juju-log Emitting Juju event install.
unit-landscape-server-0: 11:44:22 WARNING unit.landscape-server/0.install Traceback (most recent call last):
unit-landscape-server-0: 11:44:22 WARNING unit.landscape-server/0.install File "/usr/bin/add-apt-repository", line 364, in <module>
unit-landscape-server-0: 11:44:22 WARNING unit.landscape-server/0.install sys.exit(0 if addaptrepo.main() else 1)
unit-landscape-server-0: 11:44:22 WARNING unit.landscape-server/0.install File "/usr/bin/add-apt-repository", line 347, in main
unit-landscape-server-0: 11:44:22 WARNING unit.landscape-server/0.install shortcut = handler(source, **shortcut_params)
unit-landscape-server-0: 11:44:22 WARNING unit.landscape-server/0.install File "/usr/lib/python3/dist-packages/softwareproperties/shortcuts.py", line 40, in shortcut_handler
unit-landscape-server-0: 11:44:22 WARNING unit.landscape-server/0.install return handler(shortcut, **kwargs)
unit-landscape-server-0: 11:44:22 WARNING unit.landscape-server/0.install File "/usr/lib/python3/dist-packages/softwareproperties/ppa.py", line 82, in __init__
unit-landscape-server-0: 11:44:22 WARNING unit.landscape-server/0.install if self.lpppa.publish_debug_symbols:
unit-landscape-server-0: 11:44:22 WARNING unit.landscape-server/0.install File "/usr/lib/python3/dist-packages/softwareproperties/ppa.py", line 120, in lpppa
unit-landscape-server-0: 11:44:22 WARNING unit.landscape-server/0.install self._lpppa = self.lpteam.getPPAByName(name=self.ppaname)
unit-landscape-server-0: 11:44:22 WARNING unit.landscape-server/0.install File "/usr/lib/python3/dist-packages/softwareproperties/ppa.py", line 107, in lpteam
unit-landscape-server-0: 11:44:22 WARNING unit.landscape-server/0.install self._lpteam = self.lp.people(self.teamname)
unit-landscape-server-0: 11:44:22 WARNING unit.landscape-server/0.install File "/usr/lib/python3/dist-packages/softwareproperties/ppa.py", line 98, in lp
unit-landscape-server-0: 11:44:22 WARNING unit.landscape-server/0.install self._lp = login_func("%s.%s" % (self.__module__, self.__class__.__name__),
unit-landscape-server-0: 11:44:22 WARNING unit.landscape-server/0.install File "/usr/lib/python3/dist-packages/launchpadlib/launchpad.py", line 494, in login_anonymously
unit-landscape-server-0: 11:44:22 WARNING unit.landscape-server/0.install return cls(
unit-landscape-server-0: 11:44:22 WARNING unit.landscape-server/0.install File "/usr/lib/python3/dist-packages/launchpadlib/launchpad.py", line 230, in __init__
unit-landscape-server-0: 11:44:22 WARNING unit.landscape-server/0.install super(Launchpad, self).__init__(
unit-landscape-server-0: 11:44:22 WARNING unit.landscape-server/0.install File "/usr/lib/python3/dist-packages/lazr/restfulclient/resource.py", line 472, in __init__
unit-landscape-server-0: 11:44:22 WARNING unit.landscape-server/0.install self....

[Aymen Frikha (aym-frikha)]

I also hit that bug, but I got an error related to the timeout (similar to the description of this bug), not network unreachable. When i do debug-hook and run the hook manually, it worked correctly.

[Paul Goins (vultaire)]

We've hit this problem as well on the current stable branch, and will be trying to work around it by deploying landscape and dependencies to a separate "landscape" model where we can set the legacy-style proxy settings with limited impact to other apps.

Also, it's worth noting that this issue is kind of "hidden". What I mean is: the charm is effectively broken once the add-apt-repository call fails, yet that just causes a blocked status and it's not until several hooks later that things "really" break. We happened to check "juju show-status-log $unit" and could find the root cause there.

[Paul Goins (vultaire)]

For the record: using the legacy proxy settings seems to allow for a clean deploy. However, if I then try to deploy landscape-client, the proxy settings seem to get in the way. It appears the environment-level proxy settings are then stored in /etc/landscape/client.conf (without any "no-proxy"-type setting), and this ends up breaking things when that proxy shouldn't be used to reach landscape. I also then have to manually edit the /etc/landscape/client.conf files to remove such proxy references in order for registration to succeed.

On the other hand, the juju-*-proxy settings do not result in such config settings in the landscape-client charm.

Long story short: the current implementation is problematic if the customer requires a proxy for external access, yet the proxy shouldn't be used for accessing landscape via landscape-client. If we could avoid needing to use the legacy proxy settings, it would help.