Containerized Horizon: unable to connect

[Cédric Jeanneret deactivated]

Dear Stackers,

While finishing a pike BM -> pike Container migration, the last issue I get is horizon. It doesn't work as expected:
- it shows up the login form
- it accepts the login (at least, keystone seems to be happy in the backend)
- it crashes right after that with a nice stack trace:

Request Method: POST
Request URL: https://HOST/dashboard/auth/login/
Django Version: 1.8.14
Exception Type: OperationalError
Exception Value:
unable to open database file
Exception Location: /usr/lib/python2.7/site-packages/django/db/backends/sqlite3/base.py in get_new_connection, line 204
Python Executable: /usr/bin/python
Python Version: 2.7.5
Python Path:
['/usr/share/openstack-dashboard',
 '/usr/lib64/python27.zip',
 '/usr/lib64/python2.7',
 '/usr/lib64/python2.7/plat-linux2',
 '/usr/lib64/python2.7/lib-tk',
 '/usr/lib64/python2.7/lib-old',
 '/usr/lib64/python2.7/lib-dynload',
 '/usr/lib64/python2.7/site-packages',
 '/usr/lib/python2.7/site-packages',
 '/usr/share/openstack-dashboard/openstack_dashboard']
Server time: lun, 23 Avr 2018 07:26:18 +0000

Apparently, it wants to access some sqlite3 database file somewhere:
Traceback:
File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response
  132. response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/usr/lib/python2.7/site-packages/django/views/decorators/debug.py" in sensitive_post_parameters_wrapper
  76. return view(request, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/django/utils/decorators.py" in _wrapped_view
  110. response = view_func(request, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/django/views/decorators/cache.py" in _wrapped_view_func
  57. response = view_func(request, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/openstack_auth/views.py" in login
  104. **kwargs)
File "/usr/lib/python2.7/site-packages/django/views/decorators/debug.py" in sensitive_post_parameters_wrapper
  76. return view(request, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/django/utils/decorators.py" in _wrapped_view
  110. response = view_func(request, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/django/views/decorators/cache.py" in _wrapped_view_func
  57. response = view_func(request, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/django/contrib/auth/views.py" in login
  51. auth_login(request, form.get_user())
File "/usr/lib/python2.7/site-packages/django/contrib/auth/__init__.py" in login
  110. request.session.cycle_key()
File "/usr/lib/python2.7/site-packages/django/contrib/sessions/backends/base.py" in cycle_key
  285. self.create()
File "/usr/lib/python2.7/site-packages/django/contrib/sessions/backends/db.py" in create
  37. self._session_key = self._get_new_session_key()
File "/usr/lib/python2.7/site-packages/django/contrib/sessions/backends/base.py" in _get_new_session_key
  155. if not self.exists(session_key):
File "/usr/lib/python2.7/site-packages/django/contrib/sessions/backends/db.py" in exists
  33. return Session.objects.filter(session_key=session_key).exists()
File "/usr/lib/python2.7/site-packages/django/db/models/query.py" in exists
  586. return self.query.has_results(using=self.db)
File "/usr/lib/python2.7/site-packages/django/db/models/sql/query.py" in has_results
  484. return compiler.has_results()
File "/usr/lib/python2.7/site-packages/django/db/models/sql/compiler.py" in has_results
  811. return bool(self.execute_sql(SINGLE))
File "/usr/lib/python2.7/site-packages/django/db/models/sql/compiler.py" in execute_sql
  838. cursor = self.connection.cursor()
File "/usr/lib/python2.7/site-packages/django/db/backends/base/base.py" in cursor
  162. cursor = self.make_debug_cursor(self._cursor())
File "/usr/lib/python2.7/site-packages/django/db/backends/base/base.py" in _cursor
  135. self.ensure_connection()
File "/usr/lib/python2.7/site-packages/django/db/backends/base/base.py" in ensure_connection
  130. self.connect()
File "/usr/lib/python2.7/site-packages/django/db/utils.py" in __exit__
  98. six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/usr/lib/python2.7/site-packages/django/db/backends/base/base.py" in ensure_connection
  130. self.connect()
File "/usr/lib/python2.7/site-packages/django/db/backends/base/base.py" in connect
  119. self.connection = self.get_new_connection(conn_params)
File "/usr/lib/python2.7/site-packages/django/db/backends/sqlite3/base.py" in get_new_connection
  204. conn = Database.connect(**conn_params)

Exception Type: OperationalError at /auth/login/
Exception Value: unable to open database file

Although I was pretty sure it was using the mysql cluster… Maybe I overlooked something in horizon configuration?

Care to provide some hints?

Thank you!

Cheers,

C.

[Cédric Jeanneret deactivated (cjeanneret-c2c-deactivated)]

Any news regarding this issue? For now, we're still unable to connect to Horizon dashboard…

Thank you!

[Courtney Oakley (courts3003)]

Hi Cedric,
    Yep. I have had exactly the same problem while testing TripleO Pike 2-Stack (and hence containerised) configuration on my Data Centre. 2-Stack is Openstack on 2 hosts. It tests the functionality and configuration far more thoroughly than a single host.

Kolla containerised Horizon is broken.

The problem is caused by the Kolla (Puppet) Configuration files for containerised Horizon being incorrect. Horizon is automatically assuming that it has the default sqlite as its database and no Mariadb database tables are configured, nor the MySQL Session Backend database configuration statements added to the local_settings.py file.

So in summary Kolla configured Horizon does not know that it needs to use Mariadb (MySQL) as its backend database.

I have a FIX. I could rewrite the Kolla scripts, but unfortunately fixing Openstack containerisation bugs is not my day job. So this way of fixing the problem will have to do for now.

1) Access the docker container for horizon on your primary controller.

sudo docker exec-ti horizon /bin/bash

2) Now you are in the container, find the file with the local settings.

find / -name local_settings.py

It should be in /usr/share/openstack-dashboard/openstack-dashboard/local/local_settings.py.

sudo vi /usr/share/openstack-dashboard/openstack-dashboard/local/local_settings.py

3) Edit the local_settings.py file, add the following underneath the existing SESSION_ENGINE = statement and comment it out.

SESSION_ENGINE = 'django.contrib.sessions.backends.cached_db'
DATABASES = {
    'default': {
        # Database configuration here
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'dash',
        'USER': 'dash',
        'PASSWORD': 'DASH_DBPASS',
        'HOST': '<same IP address as the Memcached server>',
        'default-character-set': 'utf8'
    }
}

4) All done with that now you have to configure the Mariadb database. This will give you the MariaDB password and access.

exit the horizon container; 'exit' will do.

sudo on your controller. 'sudo -s'

MYSQLDBPASS=$(cat /var/lib/config-data/mysql/etc/puppet/hieradata/service_configs.json | grep mysql | grep root_password | awk -F": " '{print $2}' | awk -F"\"" '{print $2}')

gives you your MariaDB password. Login below with and follow the commands (ignore the mysql prompt on lines 2,3,4 and 5):-

mysql -uroot -p$MYSQLDBPASS

mysql> CREATE DATABASE dash;

mysql> GRANT ALL PRIVILEGES ON dash.* TO 'dash'@'%' IDENTIFIED BY 'DASH_DBPASS';

mysql> GRANT ALL PRIVILEGES ON dash.* TO 'dash'@'localhost' IDENTIFIED BY 'DASH_DBPASS';

mysql> quit

5) Back to the horizon container to generate the MariaDB tables and let horizon know about the MySQL database.

sudo docker exec -ti horizon /bin/bash

6) Now from the horizon container:

/usr/share/openstack-dashboard/manage.py syncdb

Answer 'no' to the request to create another superuser and exit the horizon container:

7) Now the problem is with the puppet generated configuration file used by Kolla, so unless you edit it also then ever time you restart the container the MySQL configuration will be lost.

sudo vi /var/lib/config-data/puppet-generated/horizon/etc/openstack-dashboard/local_settings

and mak...

[Bogdan Dobrelya (bogdando)]

Thank you for the hints! Those steps should be defined in t-h-t http://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/docker/services/horizon.yaml , see puppet_config, kolla_config, docker_config. In order to persist the puppet generated changes, we bind-mount configuration, like that local_settings.py, into containers, so they keep being immutable. The manage db steps are normally defined as "init containers" steps in the docker_config. And the missing mysql backend, user and grants, need to be placed into puppets.

I know there had been a lot of fixes for Horizon in Queens dev cycle. May be the stable/pike misses some of them.

[Bogdan Dobrelya (bogdando)]

Note, for the fixes for Horizon in Queens dev cycle I mostly mean puppet and tripleo heat templates configuration/templates

[Cédric Jeanneret deactivated (cjeanneret-c2c-deactivated)]

Hello Courtney, Bogdan,

Thanks for the tips - any chance to get some merge in pike from queens tree? That would be really nice :).

Cheers,

C.

[Remo Mattei (remo-o)]

any suggestions on how to fix it in PIKE, I just tried those steps somehow it still call sqlite3..

[Remo Mattei (remo-o)]

I found a workaround that you do not need to make the changes for MySQL.

1) edit the
/usr/share/openstack-tripleo-heat-templates/docker/services/horizon.yaml

turn off the following:
environment:
              - KOLLA_CONFIG_STRATEGY=COPY_ALWAYS
              # Installed plugins:
              - ENABLE_CLOUDKITTY=yes
              - ENABLE_IRONIC=yes
              - ENABLE_MAGNUM=yes
              - ENABLE_MANILA=yes
              - ENABLE_MURANO=yes
              - ENABLE_MISTRAL=yes
              - ENABLE_NEUTRON_LBAAS=yes
              - ENABLE_SAHARA=yes
              - ENABLE_TROVE=yes

to
 environment:
              - KOLLA_CONFIG_STRATEGY=COPY_ALWAYS
              # Installed plugins:
              # turn to no cloudkitty, manila, murano
              - ENABLE_CLOUDKITTY=no
              - ENABLE_IRONIC=yes
              - ENABLE_MAGNUM=no
              - ENABLE_MANILA=no
              - ENABLE_MURANO=no
              - ENABLE_MISTRAL=yes
              - ENABLE_NEUTRON_LBAAS=yes
              - ENABLE_SAHARA=yes
              - ENABLE_TROVE=no

I also created a new container but that should not matter, but the new container does not have the pkg installed of the services. Nevertheless, once I repush the update from this file the UI (Horizon) worked.

Hope this helps.
Remo

[Juan Antonio Osorio Robles (juan-osorio-robles)]

This is addressed by Remo's patch https://review.openstack.org/#/c/613186/

[OpenStack Infra (hudson-openstack)]

Change abandoned by Juan Antonio Osorio Robles (<email address hidden>) on branch: master
Review: https://review.openstack.org/613186
Reason: clearing up the gate to free up resources. I'll restore this when we fix up the issues in the gate.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Remo Mattei (<email address hidden>) on branch: master
Review: https://review.openstack.org/613186
Reason: Juan will restore it as soon as the fix is in place.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/613186
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=65b12d1d8f50e697b3c8d3231071458fbe9702dd
Submitter: Zuul
Branch: master

commit 65b12d1d8f50e697b3c8d3231071458fbe9702dd
Author: Remo Mattei <email address hidden>
Date: Wed Oct 24 19:53:35 2018 -0700

    Turning off extra UIs from Horizon fixes also SQLite issue

    * Cloudkitty
    * Murano
    * Trove
    * Manila

    Closes-Bug: #1766184
    Change-Id: Idce8619138870ac561894ff4fea433ddf1e5c4f9

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/613942

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/613943

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/613943
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=ba38de6193b4db9b8aa903ac7924e902d58dbf3a
Submitter: Zuul
Branch: stable/queens

commit ba38de6193b4db9b8aa903ac7924e902d58dbf3a
Author: Remo Mattei <email address hidden>
Date: Wed Oct 24 19:53:35 2018 -0700

    Turning off extra UIs from Horizon fixes also SQLite issue

    * Cloudkitty
    * Murano
    * Trove
    * Manila

    Closes-Bug: #1766184
    Change-Id: Idce8619138870ac561894ff4fea433ddf1e5c4f9
    (cherry picked from commit 65b12d1d8f50e697b3c8d3231071458fbe9702dd)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/613942
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=d0752e31e51ff55e49c3db572362f33a71ba5bb2
Submitter: Zuul
Branch: stable/rocky

commit d0752e31e51ff55e49c3db572362f33a71ba5bb2
Author: Remo Mattei <email address hidden>
Date: Wed Oct 24 19:53:35 2018 -0700

    Turning off extra UIs from Horizon fixes also SQLite issue

    * Cloudkitty
    * Murano
    * Trove
    * Manila

    Closes-Bug: #1766184
    Change-Id: Idce8619138870ac561894ff4fea433ddf1e5c4f9
    (cherry picked from commit 65b12d1d8f50e697b3c8d3231071458fbe9702dd)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 10.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 8.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 9.1.0 release.

[Oleg Borisenko (al-foo)]

Perhaps, I do not understand something, but the removal of the four subsystems cannot be called a "fix" (and moreover "fix for SQLite issue").

It would be much more fair to call it kludge so anyone looking for Trove in TripleO deployments could understand that nothing was fixed actually.

[Goutham Pacha Ravi (gouthamr)]

While TripleO has no support for Trove, CloudKitty, Murano or Magnum; the "fix" for this bug disabled an optional overcloud service: manila. Please see LP https://bugs.launchpad.net/tripleo/+bug/1832302 for that issue.