Failed build requirements_constraints.txt in a muti OS release

[Gilles Mocellin]

Hello,

I'm in a temporary situation with 2 controllers(and repo servers) upgrade from Ubuntu 16 to 18 (OpenStack Rocky), and one remaining Ubuntu 16 controller, to be able to distrubute ubuntu 16 components to remaining ubuntu 16 compute nodes.

I had some problems and decide to do a minor upgrade from 18.1.11 to 18.1.18.
And I cannot pass the build step.
My master_repos is a Bubuntu 18 one.
When it tries to install/build requirements_constraints.txt, it tries to download from a Ubuntu 16 URL !

Feb 08 14:34:50 Collecting cython<0.28 (from -c /var/www/repo/os-releases/18.1.18/ubuntu-18.04-x86_64/requirements_constraints.txt (line 57))
Feb 08 14:34:53 Exception:
Feb 08 14:34:53 Traceback (most recent call last):
Feb 08 14:34:53 File "/openstack/venvs/repo-build-18.1.18/local/lib/python2.7/site-packages/pip/_internal/basecommand.py", line 141, in main
Feb 08 14:34:53 status = self.run(options, args)
Feb 08 14:34:53 File "/openstack/venvs/repo-build-18.1.18/local/lib/python2.7/site-packages/pip/_internal/commands/wheel.py", line 161, in run
Feb 08 14:34:53 resolver.resolve(requirement_set)
Feb 08 14:34:53 File "/openstack/venvs/repo-build-18.1.18/local/lib/python2.7/site-packages/pip/_internal/resolve.py", line 102, in resolve
Feb 08 14:34:53 self._resolve_one(requirement_set, req)
Feb 08 14:34:53 File "/openstack/venvs/repo-build-18.1.18/local/lib/python2.7/site-packages/pip/_internal/resolve.py", line 256, in _resolve_one
Feb 08 14:34:53 abstract_dist = self._get_abstract_dist_for(req_to_install)
Feb 08 14:34:53 File "/openstack/venvs/repo-build-18.1.18/local/lib/python2.7/site-packages/pip/_internal/resolve.py", line 209, in _get_abstract_dist_for
Feb 08 14:34:53 self.require_hashes
Feb 08 14:34:53 File "/openstack/venvs/repo-build-18.1.18/local/lib/python2.7/site-packages/pip/_internal/operations/prepare.py", line 283, in prepare_linked_requirement
Feb 08 14:34:53 progress_bar=self.progress_bar
Feb 08 14:34:53 File "/openstack/venvs/repo-build-18.1.18/local/lib/python2.7/site-packages/pip/_internal/download.py", line 823, in unpack_url
Feb 08 14:34:53 unpack_file_url(link, location, download_dir, hashes=hashes)
Feb 08 14:34:53 File "/openstack/venvs/repo-build-18.1.18/local/lib/python2.7/site-packages/pip/_internal/download.py", line 728, in unpack_file_url
Feb 08 14:34:53 unpack_file(from_path, location, content_type, link)
Feb 08 14:34:53 File "/openstack/venvs/repo-build-18.1.18/local/lib/python2.7/site-packages/pip/_internal/utils/misc.py", line 578, in unpack_file
Feb 08 14:34:53 flatten=not filename.endswith('.whl')
Feb 08 14:34:53 File "/openstack/venvs/repo-build-18.1.18/local/lib/python2.7/site-packages/pip/_internal/utils/misc.py", line 461, in unzip_file
Feb 08 14:34:53 zipfp = open(filename, 'rb')
Feb 08 14:34:53 IOError: [Errno 2] No such file or directory: '/var/www/repo/pools/ubuntu-16.04-x86_64/cython/cython-0.27.3-cp27-cp27mu-manylinux1_x86_64.whl'

It should be from ubuntu-18.04-x86_64.

If I force to build only on my ubuntu 16 repo container, it fails on cradox build, but here it seems to be a lacking module :

Feb 08 13:42:09 Collecting cradox>=2.0.0 (from gnocchi->-c /var/www/repo/os-releases/18.1.18/ubuntu-16.04-x86_64/requirements_constraints.txt (line 13)) [13/5629]
Feb 08 13:42:09 Using cached https://files.pythonhosted.org/packages/30/a8/f4790a3f5af7c7616dda7bd5cddbe31d3cf7c35319b700ff49e14ffa8153/cradox-2.1.2.tar.gz
Feb 08 13:42:10 Complete output from command python setup.py egg_info:
Feb 08 13:42:10 Traceback (most recent call last):
Feb 08 13:42:10 File "<string>", line 1, in <module>
Feb 08 13:42:10 File "/tmp/openstack-builder/cradox/setup.py", line 73, in <module>
Feb 08 13:42:10 ext_modules=[Extension("cradox", ["cradox.pyx"], libraries=["rados"])],
Feb 08 13:42:10 File "/openstack/venvs/repo-build-18.1.18/local/lib/python2.7/site-packages/setuptools/__init__.py", line 130, in setup
Feb 08 13:42:10 _install_setup_requires(attrs)
Feb 08 13:42:10 File "/openstack/venvs/repo-build-18.1.18/local/lib/python2.7/site-packages/setuptools/__init__.py", line 125, in _install_setup_requires
Feb 08 13:42:10 dist.fetch_build_eggs(dist.setup_requires)
Feb 08 13:42:10 File "/openstack/venvs/repo-build-18.1.18/local/lib/python2.7/site-packages/setuptools/dist.py", line 514, in fetch_build_eggs
Feb 08 13:42:10 replace_conflicting=True,
Feb 08 13:42:10 File "/openstack/venvs/repo-build-18.1.18/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 773, in resolve
Feb 08 13:42:10 replace_conflicting=replace_conflicting
Feb 08 13:42:10 File "/openstack/venvs/repo-build-18.1.18/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1056, in best_match
Feb 08 13:42:10 return self.obtain(req, installer)
Feb 08 13:42:10 File "/openstack/venvs/repo-build-18.1.18/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1068, in obtain
Feb 08 13:42:10 return installer(requirement)
Feb 08 13:42:10 File "/openstack/venvs/repo-build-18.1.18/local/lib/python2.7/site-packages/setuptools/dist.py", line 581, in fetch_build_egg
Feb 08 13:42:10 return cmd.easy_install(req)
Feb 08 13:42:10 File "/openstack/venvs/repo-build-18.1.18/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 676, in easy_install
Feb 08 13:42:10 return self.install_item(spec, dist.location, tmpdir, deps)
Feb 08 13:42:10 File "/openstack/venvs/repo-build-18.1.18/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 702, in install_item
Feb 08 13:42:10 dists = self.install_eggs(spec, download, tmpdir)
Feb 08 13:42:10 File "/openstack/venvs/repo-build-18.1.18/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 887, in install_eggs
Feb 08 13:42:10 return self.build_and_install(setup_script, setup_base)
Feb 08 13:42:10 File "/openstack/venvs/repo-build-18.1.18/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1155, in build_and_install
Feb 08 13:42:10 self.run_setup(setup_script, setup_base, args)
Feb 08 13:42:10 File "/openstack/venvs/repo-build-18.1.18/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1141, in run_setup
Feb 08 13:42:10 run_setup(setup_script, args)
Feb 08 13:42:10 File "/openstack/venvs/repo-build-18.1.18/local/lib/python2.7/site-packages/setuptools/sandbox.py", line 253, in run_setup
Feb 08 13:42:10 raise
Feb 08 13:42:10 File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__
Feb 08 13:42:10 self.gen.throw(type, value, traceback)
Feb 08 13:42:10 File "/openstack/venvs/repo-build-18.1.18/local/lib/python2.7/site-packages/setuptools/sandbox.py", line 195, in setup_context
Feb 08 13:42:10 yield
Feb 08 13:42:10 File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__
Feb 08 13:42:10 self.gen.throw(type, value, traceback)
Feb 08 13:42:10 File "/openstack/venvs/repo-build-18.1.18/local/lib/python2.7/site-packages/setuptools/sandbox.py", line 166, in save_modules
Feb 08 13:42:10 saved_exc.resume()
Feb 08 13:42:10 File "/openstack/venvs/repo-build-18.1.18/local/lib/python2.7/site-packages/setuptools/sandbox.py", line 141, in resume
Feb 08 13:42:10 six.reraise(type, exc, self._tb)
Feb 08 13:42:10 File "/openstack/venvs/repo-build-18.1.18/local/lib/python2.7/site-packages/setuptools/sandbox.py", line 154, in save_modules
Feb 08 13:42:10 yield saved
Feb 08 13:42:10 File "/openstack/venvs/repo-build-18.1.18/local/lib/python2.7/site-packages/setuptools/sandbox.py", line 195, in setup_context
Feb 08 13:42:10 yield
Feb 08 13:42:10 File "/openstack/venvs/repo-build-18.1.18/local/lib/python2.7/site-packages/setuptools/sandbox.py", line 250, in run_setup
Feb 08 13:42:10 _execfile(setup_script, ns)
Feb 08 13:42:10 File "/openstack/venvs/repo-build-18.1.18/local/lib/python2.7/site-packages/setuptools/sandbox.py", line 45, in _execfile
Feb 08 13:42:10 exec(code, globals, locals)
Feb 08 13:42:10 File "/tmp/easy_install-jrNL5E/Jinja2-3.0.0a1/setup.py", line 6, in <module>
Feb 08 13:42:10
Feb 08 13:42:10 File "/openstack/venvs/repo-build-18.1.18/local/lib/python2.7/site-packages/setuptools/sandbox.py", line 419, in _open
Feb 08 13:42:10 return _open(path, mode, *args, **kw)
Feb 08 13:42:10 TypeError: 'encoding' is an invalid keyword argument for this function

[Gilles Mocellin (gilles-mocellin)]

Jinja 3.0.0a is download now on requirements, instead of 2.10.

cradox/test-requirements.txt does not specify a version and fails.

I also see gnocchi python requirements does not specify version in /var/www/repo/openstackgit/gnocchi/setup.cfg.

[Gilles Mocellin (gilles-mocellin)]

I have upgrade my 3 controllers to Ubuntu 18 and rebase on openstack-ansible to 18.1.11.

I'm still having the same error on rebuilding everything.

[Dmitriy Rabotyagov (noonedeadpunk)]

Hi,

I assume the issue here is that all repo containers could potentially be left running on older Ubuntu 16.04, since OS on containers is different from controller anyway.

However, there were slightly more issues on older OpenStack-Ansible versions with repo containers, as content might be overriden due to how lsyncd works.

Though, starting with Yoge we have improved a lot multi-distro and multi-arch support by deprecating lsyncd in favor of glusterfs or any other shared FS of choice.