HTTP 500 Internal Server Error

[Viorel-Cosmin Miron]

Hi most of the error is just this noticed after full deployment.

http://paste.openstack.org/show/787764/

We tried several reconfigurations and bootstrapping from zero all, nothing helped.

[Viorel-Cosmin Miron (uhl-hosting)]

Added globals.yml

[Viorel-Cosmin Miron (uhl-hosting)]

Added multinode

[Viorel-Cosmin Miron (uhl-hosting)]

Added init-runonce

[Viorel-Cosmin Miron (uhl-hosting)]

##### System Identification #####
PRETTY_NAME="Ubuntu 18.04.3 LTS"
VERSION_ID="18.04"
##### Docker Version #####
Docker version 19.03.5, build 633a0ea838
##### Docker Info #####
Client:
 Debug Mode: false

Server:
 Containers: 82
  Running: 81
  Paused: 0
  Stopped: 1
 Images: 117
 Server Version: 19.03.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
 runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 4.15.0-72-generic
 Operating System: Ubuntu 18.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 31.31GiB
 Name: ss.uhl.local
 ID: TAAV:RUAJ:PZE5:XUIV:3S34:AS4P:5ZTL:BFB3:BSLN:OUVJ:G6KW:SK7Y
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Cluster Store: etcd://172.22.100.1:2379
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: API is accessible on http://172.22.100.1:2375 without encryption.
         Access to the remote API is equivalent to root access on the host. Refer
         to the 'Docker daemon attack surface' section in the documentation for
         more information: https://docs.docker.com/engine/security/security/#docker-daemon-attack-surface
WARNING: No swap limit support
##### Ansible Version #####
ansible 2.9.1
  config file = /root/kolla-ansible_setup/ansible.cfg
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /root/venv3/lib/python3.6/site-packages/ansible
  executable location = /root/venv3/bin/ansible
  python version = 3.6.9 (default, Nov 7 2019, 10:44:02) [GCC 8.3.0]
##### List installed packages, including editables #####
Package Version
---------------------- ----------
ansible 2.9.1
appdirs 1.4.3
attrs 19.3.0
Babel 2.7.0
certifi 2019.11.28
cffi 1.13.2
chardet 3.0.4
cliff 2.16.0
cmd2 0.8.9
cryptography 2.8
debtcollector 1.22.0
decorator 4.4.1
docker 4.1.0
dogpile.cache 0.9.0
gitdb2 2.0.6
GitPython 3.0.5
idna 2.8
importlib-metadata 1.3.0
iso8601 0.1.12
Jinja2 2.10.3
jmespath 0.9.4
jsonpatch 1.24
jsonpointer 2.0
jsonschema 3.2.0
keystoneauth1 3.18.0
kolla 9.0.0
kolla-ansible 9.0.0
MarkupSafe ...

[Radosław Piliszek (yoctozepto)]

Please include glance-api logs.

[Viorel-Cosmin Miron (uhl-hosting)]

Here I run with --debug on while creating the image. Let me know if this suffice.

(venv3) root@ss:~/kolla-ansible_setup# ./init-runonce
Checking for locally available cirros image.
Creating glance image.
START with options: --debug image create --disk-format qcow2 --container-format bare --public --property os_type=linux --file .//cirros-0.4.0-x86_64-disk.img cirros
options: Namespace(access_token='***', access_token_endpoint='', access_token_type='', application_credential_id='', application_credential_name='', application_credential_secret='***', auth_methods='', auth_type='', auth_url='http://172.22.100.10:35357/v3', cacert=None, cert='', client_id='', client_secret='***', cloud='', code='', debug=True, default_domain='default', default_domain_id='', default_domain_name='', deferred_help=False, discovery_endpoint='', domain_id='', domain_name='', endpoint='', identity_provider='', insecure=None, interface='internal', key='', log_file=None, openid_scope='', os_beta_command=False, os_compute_api_version='', os_container_infra_api_version='1', os_identity_api_version='3', os_image_api_version='', os_network_api_version='', os_object_api_version='', os_project_id=None, os_project_name=None, os_tacker_api_version='1', os_volume_api_version='', passcode='', password='***', project_domain_id='', project_domain_name='Default', project_id='', project_name='admin', protocol='', redirect_uri='', region_name='RegionOne', remote_project_domain_id='', remote_project_domain_name='', remote_project_id='', remote_project_name='', service_provider='', system_scope='', timing=False, token='***', trust_id='', user_domain_id='', user_domain_name='Default', user_id='', username='admin', verbose_level=3, verify=None)
Auth plugin password selected
auth_config_hook(): {'api_timeout': None, 'verify': True, 'cacert': None, 'cert': None, 'key': None, 'baremetal_status_code_retries': '5', 'baremetal_introspection_status_code_retries': '5', 'image_status_code_retries': '5', 'disable_vendor_agent': {}, 'interface': 'internal', 'floating_ip_source': 'neutron', 'image_api_use_tasks': False, 'image_format': 'qcow2', 'message': '', 'network_api_version': '2', 'object_store_api_version': '1', 'secgroup_source': 'neutron', 'status': 'active', 'auth': {'user_domain_name': 'Default', 'project_domain_name': 'Default', 'project_name': 'admin'}, 'verbose_level': 3, 'deferred_help': False, 'debug': True, 'region_name': 'RegionOne', 'default_domain': 'default', 'timing': False, 'auth_url': 'http://172.22.100.10:35357/v3', 'username': 'admin', 'password': '***', 'beta_command': False, 'identity_api_version': '3', 'tacker_api_version': '1', 'container_infra_api_version': '1', 'auth_type': 'password', 'networks': []}
defaults: {'api_timeout': None, 'verify': True, 'cacert': None, 'cert': None, 'key': None, 'auth_type': 'password', 'baremetal_status_code_retries': 5, 'baremetal_introspection_status_code_retries': 5, 'image_status_code_retries': 5, 'disable_vendor_agent': {}, 'interface': None, 'floating_ip_source': 'neutron', 'image_api_use_tasks': False, 'image_format': 'qcow2', 'message': '', 'network_api_version': '2', 'object_store_api_version': '1', 'secgroup_source': 'neutron', '...

[Radosław Piliszek (yoctozepto)]

No, we need service logs. In /var/log/kolla/glance/glance-api.log

[Viorel-Cosmin Miron (uhl-hosting)]

Added the log as requested.

[Radosław Piliszek (yoctozepto)]

So it's up to swift now. Please attach all swift logs.

[Viorel-Cosmin Miron (uhl-hosting)]

Added swift log

[Radosław Piliszek (yoctozepto)]

"DB doesn't exist" is pretty self-explanatory. :-)

Did you configure swift rings properly?

[Viorel-Cosmin Miron (uhl-hosting)]

This got messed up when I moved from Stein to Train?

I did not made any changed to the swift rings, now I have disabled Swift and will try again. Is there a proper guide for Train and rings? I get some errors on latest code snippet to create the cells.

[Viorel-Cosmin Miron (uhl-hosting)]

It seems something is keeping in lock some docker value, every once and than it fails, hard to reproduce.
- to have a quick workaround I have redefined the control server to a new server.
- In swift it fails due to broken rings, so this can be closer to a missconfiguration.

[Radosław Piliszek (yoctozepto)]

Well, Stein->Train upgrade seems to be just fine for Swift: https://review.opendev.org/700268
CI tests pre- and post-upgrade both swift via glance and swift directly plus misc tests.

[Launchpad Janitor (janitor)]

[Expired for kolla-ansible because there has been no activity for 60 days.]