OpenStack Swift-proxy-server do not start

[Daniel Pham]

I installed OpenStack version Rocky with Ansible.
I run all 3 playbooks without any errors, but during the test I discovered that Swift was down. I was unable to store an image.

Swift services are not all up, swift-proxy-server was down so I tried to restart it manually.

This following TraceBack are seen in my log when I manually start swift-proxy-server :

root@storage1-swift-proxy-container:/var/log# /openstack/venvs/swift-18.1.21.dev5/bin/swift-proxy-server /etc/swift/proxy-server/proxy-server.conf
Traceback (most recent call last):
  File "/openstack/venvs/swift-18.1.21.dev5/bin/swift-proxy-server", line 23, in <module>
    sys.exit(run_wsgi(conf_file, 'proxy-server', **options))
  File "/openstack/venvs/swift-18.1.21.dev5/lib/python2.7/site-packages/swift/common/wsgi.py", line 1029, in run_wsgi
    loadapp(conf_path, global_conf=global_conf)
  File "/openstack/venvs/swift-18.1.21.dev5/lib/python2.7/site-packages/swift/common/wsgi.py", line 394, in loadapp
    ctx = loadcontext(loadwsgi.APP, conf_file, global_conf=global_conf)
  File "/openstack/venvs/swift-18.1.21.dev5/lib/python2.7/site-packages/swift/common/wsgi.py", line 378, in loadcontext
    global_conf=global_conf)
  File "/openstack/venvs/swift-18.1.21.dev5/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 296, in loadcontext
    global_conf=global_conf)
  File "/openstack/venvs/swift-18.1.21.dev5/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 320, in _loadconfig
    return loader.get_context(object_type, name, global_conf)
  File "/openstack/venvs/swift-18.1.21.dev5/lib/python2.7/site-packages/swift/common/wsgi.py", line 68, in get_context
    object_type, name=name, global_conf=global_conf)
  File "/openstack/venvs/swift-18.1.21.dev5/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 450, in get_context
    global_additions=global_additions)
  File "/openstack/venvs/swift-18.1.21.dev5/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 562, in _pipeline_app_context
    for name in pipeline[:-1]]
  File "/openstack/venvs/swift-18.1.21.dev5/lib/python2.7/site-packages/swift/common/wsgi.py", line 68, in get_context
    object_type, name=name, global_conf=global_conf)
  File "/openstack/venvs/swift-18.1.21.dev5/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 454, in get_context
    section)
  File "/openstack/venvs/swift-18.1.21.dev5/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 476, in _context_from_use
    object_type, name=use, global_conf=global_conf)
  File "/openstack/venvs/swift-18.1.21.dev5/lib/python2.7/site-packages/swift/common/wsgi.py", line 68, in get_context
    object_type, name=name, global_conf=global_conf)
  File "/openstack/venvs/swift-18.1.21.dev5/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 406, in get_context
    global_conf=global_conf)
  File "/openstack/venvs/swift-18.1.21.dev5/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 296, in loadcontext
    global_conf=global_conf)
  File "/openstack/venvs/swift-18.1.21.dev5/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 328, in _loadegg
    return loader.get_context(object_type, name, global_conf)
  File "/openstack/venvs/swift-18.1.21.dev5/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 620, in get_context
    object_type, name=name)
  File "/openstack/venvs/swift-18.1.21.dev5/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 640, in find_egg_entry_point
    pkg_resources.require(self.spec)
  File "/openstack/venvs/swift-18.1.21.dev5/lib/python2.7/site-packages/pkg_resources/__init__.py", line 891, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/openstack/venvs/swift-18.1.21.dev5/lib/python2.7/site-packages/pkg_resources/__init__.py", line 782, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (swift 0.0.0 (/openstack/venvs/swift-18.1.21.dev5/lib/python2.7/site-packages), Requirement.parse('swift>=2.14.0'), set(['swift3']))

As we can see in the last traceback, Swift version is "0.0.0" so the Requirement.parse did not fulfill the condition. I don't know why my repo-container pulled those versions.

Moreover it appear there are several version "0.0.0" for the other OpenStack services, but those services are running without error.
We can see the version of the service into the repo container :

root@infra1-repo-container:~# find /var/www/** -name "*0.0.0*"
/var/www/repo/os-releases/18.1.21.dev5/ubuntu-18.04-x86_64/networking_sfc-0.0.0-py2-none-any.whl
/var/www/repo/os-releases/18.1.21.dev5/ubuntu-18.04-x86_64/designate_dashboard-0.0.0-py2-none-any.whl
/var/www/repo/os-releases/18.1.21.dev5/ubuntu-18.04-x86_64/horizon-0.0.0-py2-none-any.whl
/var/www/repo/os-releases/18.1.21.dev5/ubuntu-18.04-x86_64/trove_dashboard-0.0.0-py2-none-any.whl
/var/www/repo/os-releases/18.1.21.dev5/ubuntu-18.04-x86_64/octavia_dashboard-0.0.0-py2-none-any.whl
/var/www/repo/os-releases/18.1.21.dev5/ubuntu-18.04-x86_64/keystone-0.0.0-py2-none-any.whl
/var/www/repo/os-releases/18.1.21.dev5/ubuntu-18.04-x86_64/cinder-0.0.0-py2-none-any.whl
/var/www/repo/os-releases/18.1.21.dev5/ubuntu-18.04-x86_64/magnum_ui-0.0.0-py2-none-any.whl
/var/www/repo/os-releases/18.1.21.dev5/ubuntu-18.04-x86_64/setuptools-40.0.0-py2.py3-none-any.whl
/var/www/repo/os-releases/18.1.21.dev5/ubuntu-18.04-x86_64/ceilometer-0.0.0-py2-none-any.whl
/var/www/repo/os-releases/18.1.21.dev5/ubuntu-18.04-x86_64/openstack_heat-0.0.0-py2-none-any.whl
/var/www/repo/os-releases/18.1.21.dev5/ubuntu-18.04-x86_64/nova_powervm-0.0.0-py2-none-any.whl
/var/www/repo/os-releases/18.1.21.dev5/ubuntu-18.04-x86_64/swift-0.0.0-py2-none-any.whl
/var/www/repo/os-releases/18.1.21.dev5/ubuntu-18.04-x86_64/neutron_fwaas_dashboard-0.0.0-py2-none-any.whl
/var/www/repo/os-releases/18.1.21.dev5/ubuntu-18.04-x86_64/nova_lxd-18.0.0.0rc1-py2-none-any.whl
/var/www/repo/os-releases/18.1.21.dev5/ubuntu-18.04-x86_64/nova-0.0.0-py2.py3-none-any.whl
/var/www/repo/os-releases/18.1.21.dev5/ubuntu-18.04-x86_64/neutron-0.0.0-py2.py3-none-any.whl
/var/www/repo/pools/ubuntu-18.04-x86_64/swift/swift-0.0.0-py2-none-any.whl
/var/www/repo/pools/ubuntu-18.04-x86_64/openstack_heat/openstack_heat-0.0.0-py2-none-any.whl
/var/www/repo/pools/ubuntu-18.04-x86_64/ceilometer/ceilometer-0.0.0-py2-none-any.whl
/var/www/repo/pools/ubuntu-18.04-x86_64/magnum_ui/magnum_ui-0.0.0-py2-none-any.whl
/var/www/repo/pools/ubuntu-18.04-x86_64/nova_powervm/nova_powervm-0.0.0-py2-none-any.whl
/var/www/repo/pools/ubuntu-18.04-x86_64/neutron/neutron-0.0.0-py2.py3-none-any.whl
/var/www/repo/pools/ubuntu-18.04-x86_64/octavia_dashboard/octavia_dashboard-0.0.0-py2-none-any.whl
/var/www/repo/pools/ubuntu-18.04-x86_64/designate_dashboard/designate_dashboard-0.0.0-py2-none-any.whl
/var/www/repo/pools/ubuntu-18.04-x86_64/setuptools/setuptools-40.0.0-py2.py3-none-any.whl
/var/www/repo/pools/ubuntu-18.04-x86_64/networking_sfc/networking_sfc-0.0.0-py2-none-any.whl
/var/www/repo/pools/ubuntu-18.04-x86_64/cinder/cinder-0.0.0-py2-none-any.whl
/var/www/repo/pools/ubuntu-18.04-x86_64/trove_dashboard/trove_dashboard-0.0.0-py2-none-any.whl
/var/www/repo/pools/ubuntu-18.04-x86_64/keystone/keystone-0.0.0-py2-none-any.whl
/var/www/repo/pools/ubuntu-18.04-x86_64/horizon/horizon-0.0.0-py2-none-any.whl
/var/www/repo/pools/ubuntu-18.04-x86_64/nova/nova-0.0.0-py2.py3-none-any.whl
/var/www/repo/pools/ubuntu-18.04-x86_64/neutron_fwaas_dashboard/neutron_fwaas_dashboard-0.0.0-py2-none-any.whl
/var/www/repo/pools/ubuntu-18.04-x86_64/nova_lxd/nova_lxd-18.0.0.0rc1-py2-none-any.whl
/var/www/repo/links/ubuntu-18.04-x86_64/networking_sfc-0.0.0-py2-none-any.whl
/var/www/repo/links/ubuntu-18.04-x86_64/designate_dashboard-0.0.0-py2-none-any.whl
/var/www/repo/links/ubuntu-18.04-x86_64/horizon-0.0.0-py2-none-any.whl
/var/www/repo/links/ubuntu-18.04-x86_64/trove_dashboard-0.0.0-py2-none-any.whl
/var/www/repo/links/ubuntu-18.04-x86_64/octavia_dashboard-0.0.0-py2-none-any.whl
/var/www/repo/links/ubuntu-18.04-x86_64/keystone-0.0.0-py2-none-any.whl
/var/www/repo/links/ubuntu-18.04-x86_64/cinder-0.0.0-py2-none-any.whl
/var/www/repo/links/ubuntu-18.04-x86_64/magnum_ui-0.0.0-py2-none-any.whl
/var/www/repo/links/ubuntu-18.04-x86_64/setuptools-40.0.0-py2.py3-none-any.whl
/var/www/repo/links/ubuntu-18.04-x86_64/ceilometer-0.0.0-py2-none-any.whl
/var/www/repo/links/ubuntu-18.04-x86_64/openstack_heat-0.0.0-py2-none-any.whl
/var/www/repo/links/ubuntu-18.04-x86_64/nova_powervm-0.0.0-py2-none-any.whl
/var/www/repo/links/ubuntu-18.04-x86_64/swift-0.0.0-py2-none-any.whl
/var/www/repo/links/ubuntu-18.04-x86_64/neutron_fwaas_dashboard-0.0.0-py2-none-any.whl
/var/www/repo/links/ubuntu-18.04-x86_64/nova_lxd-18.0.0.0rc1-py2-none-any.whl
/var/www/repo/links/ubuntu-18.04-x86_64/nova-0.0.0-py2.py3-none-any.whl
/var/www/repo/links/ubuntu-18.04-x86_64/neutron-0.0.0-py2.py3-none-any.whl

Step to reproduce :

https://docs.openstack.org/project-deploy-guide/openstack-ansible/rocky/deploymenthost.html

DragonFlow is no longer maintained so you have to patch it :
https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/745377/

ubuntu repo do not have version "18.04.1-base-amd64.tar.gz" anymore so you have to override :
lxc_hosts_container_image_url: "http://cdimage.ubuntu.com/ubuntu-base/releases/18.04/release/ubuntu-base-18.04.5-base-amd64.tar.gz"
into the user_variables.yml file

[Jonathan Rosser (jrosser)]

Hi Daniel,

The Rocky branch is in extended-maintenance so we are no longer back-porting bug fixes to it. I would strongly recommend using a more recent maintained branch.

Canonical deleted the ubuntu images that these older branches rely on, unfortunately something out of our control for openstack-ansible. There is a fix for this in the Stein branch and later:

https://github.com/openstack/openstack-ansible-lxc_hosts/commit/46e10d21f15fa7ec50c35e800125278e270f46b6

You can apply this patch locally to try to fix the LXC image version. I'll leave the bug open for now for the 0.0.0 version issue, but really the best advice is to move to a more recent supported branch.

[Dmitriy Rabotyagov (noonedeadpunk)]

Hi,

Can you kindly provide us with what tag are you trying to use? Was it rocky-em?

Also you can try re-defining openstack_release: 18.3.0 (or smth like that) - and it should set correct venv_tag as well.

[Daniel Pham (shin-ichi)]

Hello again,

Thank you for your replies, I used the patch for the LXC image version instead of override this variable.

I used the stable/rocky branch of openstack-ansible to configure my inventory of machines :
git clone -b stable/rocky https://opendev.org/openstack/openstack-ansible.git
Did I have to use the tag rocky-em instead of the stable branch ?

I'm not sure to understand "re-defining openstack_release".
I override the variable into my user_variables.yml file with your version number, but if it did not work :
 Where can I find the list of openstack_release version number ?

During my research I found out that the job "Backup the default pip_install_upper_constraints"
into the 2nd playbook "setup-infrastructure" fetch a list of requirements.

This list seems wrong :

root@os-infra:~# curl http://<internal_lb_vip_address>:8181/os-releases/18.1.21.dev5/ubuntu-18.04-x86_64/requirements_absolute_requirements.txt | grep 0.0.0
networking_sfc==0.0.0
designate_dashboard==0.0.0
horizon==0.0.0
trove_dashboard==0.0.0
octavia_dashboard==0.0.0
keystone==0.0.0
cinder==0.0.0
magnum_ui==0.0.0
setuptools==40.0.0
ceilometer==0.0.0
openstack_heat==0.0.0
nova_powervm==0.0.0
swift==0.0.0
neutron_fwaas_dashboard==0.0.0
nova_lxd==18.0.0.0rc1
nova==0.0.0
neutron==0.0.0

[Daniel Pham (shin-ichi)]

Hello again,

I cloned openstack/openstack-ansible.git with the tag rocky-em instead of the branch stable/rocky
After the second playbook it apear that the correct version of OpenStack components was correct

root@infra1-repo-container-9be89d4e:~# ls /var/www/repo/os-releases/18.1.20/ubuntu-18.04-x86_64/ | grep 0.0.0
nova_lxd-18.0.0.0rc1-py2-none-any.whl
setuptools-40.0.0-py2.py3-none-any.whl

They areno more weird .whl file.

Thank you for your hint
Have a nice day

[Dmitriy Rabotyagov (noonedeadpunk)]

Yeah, the thing is that we're detecting version with pbr based on the state of the openstack-ansible repo. In case it can't detect it, things may go weird.
I'd say that jsut setting `openstack_release` should override that behaviour, but maybe it isn't.

Anyway good that you've figured that out!