Asset issue with multiple horizon instances

[Ian Cordasco]

On a customer deployment, we have 5 horizon containers across 5 nodes. In the horizon_common role we run manage.py collectstatic --noinput and manage.py compress --force. Since the assets should be identical across all of the machines, the compressor *should* generate files with exactly the same name. We've run into instances where that's not the case so if the response is made by the horizon app on node0 but the assets are served from node2 (and node2's assets are named differently) then the page will appear unstyled because the browser received a 404 trying to retrieve the assets.

The current in-production solution is to set [1] COMPRESS_OFFLINE = True in local_settings.py. This, however, may not be an appropriate solution since this setting will prevent django_compressor from dynamically creating missing asset files [2]. Also, the problem we're seeing seems that it may be a problem upstream [3]. Also, for what it's worth, it seems that Horizon [4] sets a 'hash' for COMPRESS_CSS_HASHING_METHOD by default which is not a documented option [5].

There are a few problems I can think of:

- mtime is involved in the hashing of the files so the asset names will be different if the mtime has a delta
- the files are different and the response is served from a different container than the assets are served from
- we're running the commands in the wrong order (should be running compress then collectstatic) (read the bugs linked for others doing this)
- we should be setting COMPRESS_OFFLINE = True in our local_settings.py to prevent re-compression on the fly
- we should be setting a documented option for COMPRESS_CSS_HASHING_METHOD to ensure it works as expected.

Which of these possibilities (or combination thereof) is actually causing this problem is unclear at the moment.

[1]: http://django-compressor.readthedocs.org/en/latest/settings/#django.conf.settings.COMPRESS_OFFLINE
[2]: https://github.com/django-compressor/django-compressor/issues/443
[3]: https://github.com/django-compressor/django-compressor/issues/426
[4]: https://github.com/openstack/horizon/blob/16ef57c94bd6f593d2e995039f8642aae2ec009e/openstack_dashboard/settings.py#L212
[5]: http://django-compressor.readthedocs.org/en/latest/settings/#django.conf.settings.COMPRESS_CSS_HASHING_METHOD

[Matt Thompson (mattt416)]

Horizon defaults COMPRESS_CSS_HASHING_METHOD to 'hash' (not documented, but is treated the same as 'content'):

https://github.com/openstack/horizon/blob/stable/icehouse/openstack_dashboard/settings.py#L145
https://github.com/openstack/horizon/blob/stable/juno/openstack_dashboard/settings.py#L212

In my lab running os-ansible-deployment master (deployed 15/12/2014), I uninstalled all horizon-related pip packages, removed /usr/local/lib/python2.7/dist-packages/static, and re-ran 'horizon-manage.py collectstatic --noinput; horizon-manage.py compress --force' on a single container. I then verified that all files recreated in /usr/local/lib/python2.7/dist-packages/static were identical to those in another horizon container. When I changed COMPRESS_CSS_HASHING_METHOD to 'mtime' and reran the above, the contents of /usr/local/lib/python2.7/dist-packages/static were no longer identical.

Regarding the order in which we run collectstatic/compress, I believe this should be fine. There is actually a pending review [1] to add some documentation to horizon to clarify how this is done (and the order they outline matches the order in which we do it).

[1]: https://review.openstack.org/#/c/141885/

[Ian Cordasco (icordasc)]

So after further looking into this issue today, it appears the containers were upgraded from 10.0.0 to 10.1rc1. During the upgrade, the manifest.json (located in /usr/local/lib/python2.7/dist-packages/static/dashboard) were regenerated and that for those to take effect apache needs to be properly restarted. It would seem that at least one container did not properly restart apache. Once apache was restarted on all of the containers, they all were serving assets named exactly the same from all of the containers. The only difference between the manifests were ordering (which does not cause this bug).

If this is indeed encountered again, the containers/apache should be restarted to ensure that the manifest.json is properly reloaded and used to render templates which rely on assets.

[Ian Cordasco (icordasc)]

I'm going to attempt to get a lab together to see if I can reproduce this and confirm that it's a problem with apache not being properly restarted.