regression: offline installation fails for charms that depend on packages that use easy_install

[Pedro Guimarães]

Using Octavia from: cs:~openstack-charmers-next/octavia.

Running on an environment with very strict access to internet.

When deploying Octavia,it fails installation with the message as in the bottom of bug report.

For this environment, I am using only apt-http(s)-proxy and juju-http(s)-proxy as configurations for the proxy. Proxy does allow me to reach for pypi.org. It seems that I am forced to use also http(s)-proxy settings on juju model-config for octavia to work (will test next).

However, that type of installation, using access to pypi.org, won't work on air-gapped scenarios neither.

(...)
Looking in links: wheelhouse
Collecting charmhelpers
Collecting statsd
Collecting PyYAML
Collecting psutil
Collecting os-service-types
    Complete output from command python setup.py egg_info:
    Download error on https://pypi.org/simple/pbr/: [Errno -2] Name or service not known -- Some packages may not be found!
    Couldn't find index page for 'pbr' (maybe misspelled?)
    Download error on https://pypi.org/simple/: [Errno -2] Name or service not known -- Some packages may not be found!
    No local packages or working download links found for pbr>=2.0.0
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-zsrnx_ia/os-service-types/setup.py", line 29, in <module>
        pbr=True)
      File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/setuptools/__init__.py", line 144, in setup
        _install_setup_requires(attrs)
      File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/setuptools/__init__.py", line 139, in _install_s
etup_requires
        dist.fetch_build_eggs(dist.setup_requires)
      File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/setuptools/dist.py", line 720, in fetch_build_eg
gs
        replace_conflicting=True,
      File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 782, in resolve
        replace_conflicting=replace_conflicting
      File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1065, in best_m
atch
        return self.obtain(req, installer)
      File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1077, in obtain
        return installer(requirement)
      File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/setuptools/dist.py", line 787, in fetch_build_eg
g
        return cmd.easy_install(req)
      File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/setuptools/command/easy_install.py", line 673, i
n easy_install
        raise DistutilsError(msg)
    distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('pbr>=2.0.0')

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-zsrnx_ia/os-service-types/
Traceback (most recent call last):
  File "./hooks/install", line 8, in <module>
    basic.bootstrap_charm_deps()
  File "lib/charms/layer/basic.py", line 189, in bootstrap_charm_deps
    '--no-cache-dir', '-f', 'wheelhouse'] + list(pkgs))
  File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/var/lib/juju/agents/unit-octavia-0/.venv/bin/pip', 'install', '-U', '--force-reinstall'
, '--no-index', '--no-cache-dir', '-f', 'wheelhouse', 'charmhelpers', 'statsd', 'PyYAML', 'psutil', 'os-service-types', 'ovs', 'py
thon-neutronclient', 'alembic', 'pbr', 'vine', 'os-client-config', 'MarkupSafe', 'oslo.versionedobjects', 'waitress', 'oslo.utils'
, 'oslo.db', 'charms.reactive', 'zipp', 'oslo.concurrency', 'python-novaclient', 'Tempita', 'pytz', 'requests', 'SQLAlchemy', 'dec
orator', 'python-dateutil', 'testresources', 'soupsieve', 'python-editor', 'netifaces', 'pecan', 'jsonpatch', 'oslo.messaging', 'c
achetools', 'futurist', 'python-keystoneclient', 'keystoneauth1', 'dnspython', 'linecache2', 'oslo.i18n', 'munch', 'fixtures', 'py
perclip', 'certifi', 'urllib3', 'openstacksdk', 'os-ken', 'requestsexceptions', 'pyparsing', 'cffi', 'appdirs', 'jsonpointer', 'ne
taddr', 'Jinja2', 'stevedore', 'WebTest', 'sortedcontainers', 'oslo.middleware', 'msgpack', 'eventlet', 'oslo.log', 'kombu', 'osc-
lib', 'wrapt', 'sqlparse', 'Paste', 'Mako', 'PasteDeploy', 'testscenarios', 'python-mimeparse', 'cliff', 'wcwidth', 'wheel', 'os-t
raits', 'cryptography', 'iso8601', 'debtcollector', 'cmd2', 'traceback2', 'jmespath', 'testtools', 'amqp', 'oslo.service', 'pycpar
ser', 'importlib-metadata', 'repoze.lru', 'extras', 'unittest2', 'idna', 'charms.openstack', 'fasteners', 'tinyrpc', 'greenlet', '
pytest-runner', 'simplejson', 'monotonic', 'argparse', 'sqlalchemy-migrate', 'Routes', 'chardet', 'logutils', 'rfc3986', 'oslo.con
text', 'pyaml', 'osprofiler', 'Babel', 'dogpile.cache', 'oslo.config', 'beautifulsoup4', 'prettytable', 'yappi', 'WebOb', 'neutron
-lib', 'setproctitle', 'oslo.policy', 'pyinotify', 'six', 'oslo.serialization']' returned non-zero exit status 1.

[Frode Nordahl (fnordahl)]

This is most likely caused by a regression in layer-basic and should be fixed by rebuilding the charm with a version of layer-basic that contains https://github.com/juju-solutions/layer-basic/pull/172q

[Pedro Guimarães (pguimaraes)]

Hi fnordahl, yes I was suspecting of layer-basic, just surprised me other charms did not have it happening, such as Barbican's. I am using openstack-charmers-next, so i guess latest version is also having this issue.

Can we get that done before 20.05 release?

[Dmitrii Shcherbakov (dmitriis)]

It looks like the regression mentioned in #1 only affects charms that have dependencies that use easy_install.

https://review.opendev.org/#/q/topic:bug/1879184+(status:open+OR+status:merged)

So it isn't octavia-specific so I am going to rename the bug.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/728755

[Dmitrii Shcherbakov (dmitriis)]

I went through all reactive openstack charms to find out which charms depend on pbr and added them to the bug as well, except for the OVN charms mentioned in LP: #1878626.

[Pedro Guimarães (pguimaraes)]

Hi @dmitriis, I have your change as a charm on my namespace.
When I run your installation, I am seeing now: https://pastebin.canonical.com/p/MfpvPBjzZn/

It is still failing on install hook, but for other reason

[Pedro Guimarães (pguimaraes)]

As far as I understood, we have a couple of issues when we get an env with very strict network access here:
1) pip is always searching for packages online: there is no configuration on ~/.pydistutils.cfg to prevent that. From what dmitriis pointed me, this rollback fixes it: https://github.com/juju-solutions/layer-basic/pull/169/files

Then, 2) appdirs-1.4.4 seems not to be working. I am rolled back to 1.4.3 (which is used on the stable version of this charm) and I could install. Otherwise, 1.4.4 fails with: https://pastebin.canonical.com/p/MfpvPBjzZn/

[Frode Nordahl (fnordahl)]

Please use one of the built artifacts from our CI in your test, the resulting charm you get from a charm build depends heavily on your build environment.

It is also inaccurate to say that pip searches for packages online (it does not, just try to execute it with -vv), the problem is with some individual python packages setup scripts that pip executes. Which act the one or other way depend on how the upstream dependency is packaged.

From our testing with the OVN charms the problem disappeared after [2] was landed.

This build [0] should contain the relevant pieces of [1] rolled back [2].

0: https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_smoke/openstack/charm-octavia/728751/1/16628/octavia-charm_build-33946.tar.bz2
1: https://github.com/juju-solutions/layer-basic/pull/169/files
2: https://github.com/juju-solutions/layer-basic/pull/172/files

[Dmitrii Shcherbakov (dmitriis)]

So the particular issue Pedro is running into now is due to a sequence of events:

1) There was a piece of code added to appdirs 1.4.4 (compared to 1.4.3 which was pulled for previous charm builds):
https://github.com/ActiveState/appdirs/blame/1.4.4/setup.py#L24-L28

It contains a seemingly unharmful piece of code to read the content of appdirs.py which contains unicode characters:
https://github.com/ActiveState/appdirs/blob/1.4.4/appdirs.py#L4

2) During the installation from wheelhouse, correct dependencies are used and so the layer-basic fix is sound in that regard.

3) setup.py gets called and thus the code from appdirs 1.4.4 tries to read the lines of appdirs.py containing unicode characters. This results in the following:

https://pastebin.canonical.com/p/MfpvPBjzZn/

2020-05-19 05:12:55 DEBUG install Complete output from command python setup.py egg_info:
2020-05-19 05:12:55 DEBUG install Traceback (most recent call last):
2020-05-19 05:12:55 DEBUG install File "<string>", line 1, in <module>
2020-05-19 05:12:55 DEBUG install File "/tmp/pip-install-v5e474hs/appdirs/setup.py", line 25, in <module>
2020-05-19 05:12:55 DEBUG install for line in read("appdirs.py").splitlines():
2020-05-19 05:12:55 DEBUG install File "/tmp/pip-install-v5e474hs/appdirs/setup.py", line 19, in read
2020-05-19 05:12:55 DEBUG install out = "\n" + inf.read().replace("\r\n", "\n")
2020-05-19 05:12:55 DEBUG install File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/encodings/ascii.py", line 26, in decode
2020-05-19 05:12:55 DEBUG install return codecs.ascii_decode(input, self.errors)[0]
2020-05-19 05:12:55 DEBUG install UnicodeDecodeError: 'ascii' codec can't decode byte 0xc8 in position 129: ordinal not in range(128)
2020-05-19 05:12:55 DEBUG install
2020-05-19 05:12:55 DEBUG install ----------------------------------------
2020-05-19 05:12:55 DEBUG install Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-v5e474hs/appdirs/

Now, to why this happens:

* on bionic python3.6 is used and it is affected by the following behavior if LANG is not set (this doesn't happen on 3.8 for example):

Python 3.6.9 (default, Apr 18 2020, 01:56:04)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import locale
>>> locale.getpreferredencoding()
'ANSI_X3.4-1968'

https://docs.python.org/3/library/functions.html#open
"encoding is the name of the encoding used to decode or encode the file. This should only be used in text mode. The default encoding is platform dependent (whatever ***locale.getpreferredencoding() returns***), but any text encoding supported by Python can be used. See the codecs module for the list of supported encodings."

- basically, ASCII is used if LANG is not set with python 3.6.

* Pedro's environment has juju 2.6.x which doesn't have this Juju commit that adds LANG=C.UTF-8 to all hook environments https://github.com/juju/juju/commit/d9a55e9e900cd66e83d88b5643d7e1a2560c160c

git tag --contains=d9a55e9e900cd66e83d88b5643d7e1a2560c160c | grep 2.6 ; echo $?
1
git tag --contains=d9a55e9e900cd66e83d88b5643d7e1a2560c160c...

[Pedro Guimarães (pguimaraes)]

Hi Dmitrii, just to confirm this is the version of octavia that is working with no issues for me:

https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_smoke/openstack/charm-octavia/728751/1/16628/octavia-charm_build-33946.tar.bz2

As you shared with me.
Can you confirm this is the fix that will be available on the charmers-next (or if it is already available)?
I am currently using a local charm from that tar.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/728751
Committed: https://git.openstack.org/cgit/openstack/charm-octavia/commit/?id=8caeb381a3f3547bd2ec2a3e844ac789feffb3a5
Submitter: Zuul
Branch: master

commit 8caeb381a3f3547bd2ec2a3e844ac789feffb3a5
Author: Dmitrii Shcherbakov <email address hidden>
Date: Mon May 18 09:16:45 2020 +0300

    Rebuild to include a layer-basic fix

    This change makes a regression fix for layer-basic to be included.

    https://github.com/juju-solutions/layer-basic/pull/172

    Change-Id: Iab0d7a4d210b519f7101cd57bdaca2cbdda6f747
    Closes-Bug: #1879184

[Dmitrii Shcherbakov (dmitriis)]

Hi Pedro,

The fix will be available with the 20.05 release and soon in openstack-charmers-next (see #11).

[OpenStack Infra (hudson-openstack)]

Change abandoned by Dmitrii Shcherbakov (<email address hidden>) on branch: master
Review: https://review.opendev.org/728755
Reason: This commit also resulted in a rebuild: https://opendev.org/openstack/charm-octavia-diskimage-retrofit/commit/4dd8ffe72309955701ee4b80ea066ec18fd122c2

Abandoning.