[SRU] Fixing empty create swift container dialog after upgrading horizon from newton to ocata

[Seyeong Kim]

[Impact]

After upgrading openstack-dashboard from newton to ocata
Create Container dialog show us empty dialog.

I'm sure this is needed for ocata UCA
but not sure pike or newer version.
because upgrading 2 steps is not recommended.
Please let me know if the patch for the other releases are also needed.
e.g from newton to pike or from xenial to Z A B?

[Test Case]

1. deploy newton env with object store
2. check if create container dialog show us proper contents ( yes )
3. upgrade newton to coat
4. check if create container dialog show us proper contents ( no )

[Regression Potential]
After this patch, every installing or upgrading try to delete specific directories and files that no need in ocata anymore. affected file are related to newton only but if ocata has changes in that directory, that file also can be deleted. but this case unlikely happens.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "lp1775170_ocata.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Corey Bryant (corey.bryant)]

Hi Seyeong,

Something is wrong because the upgrade from newton->ocata should already drop static assets from /usr/share/openstack-dashboard/horizon/static/ and install them in /var/lib/openstack-dashboard/static. I don't think the attached patch is the correct fix. Is the existing preinst script wrong or incomplete?

Thanks,
Corey

[Seyeong Kim (seyeongkim)]

@corey

oh Thanks, I haven't noticed that. because there are remained files.

so, In this case, rm -rf in preinst didn't work.

I'll analyze this further.

Thanks.

[Seyeong Kim (seyeongkim)]

Hello Corey

I've analyzed those.

In preinst, below line there.

rm -rf /usr/share/openstack-dashboard/static/ || :
rm -rf /var/lib/openstack-dashboard/static/ || :

it is not the same as

find /usr/share/openstack-dashboard/horizon/static/horizon/lib/ ! -name 'jquery.bootstrap.wizard.js' -delete || :

because /usr/share/openstack-dashboard/static is link to /usr/share/openstack-dahsboard/openstack_dashboard/static

I moved delete code to preinst instead of postinst(don't need to restart apache2 again)

Could you please review this?

Thanks a lot!

[Corey Bryant (corey.bryant)]

Seyeong,

Can you tell if the files in /usr/share/openstack-dahsboard/openstack_dashboard/static get installed in /var/lib/openstack-dashboard/static/? If so, we likely want to add 'rm -rf /usr/share/openstack-dashboard/horizon/static/ || :' where we currently run 'rm -rf /usr/share/openstack-dashboard/static/ || :'. So we'd remove the contents of both directories prior to collect/compress.

Thanks,
Corey

[Seyeong Kim (seyeongkim)]

Hello Corey,

If I don't remove horizon/static,

compressed js has incomplete js file so UI show us javascript error when clicking container button.
I think it can't get template for this dialog.

I checked compressed js has more code( and broken ) than pure ocata

I don't remember exact code so I'll reproduce this and will attache that compressed part
and I'll also check your request.

[Seyeong Kim (seyeongkim)]

Hello Corey

I paste directory structure and result of python manage.py collectstatic --no-input

https://pastebin.ubuntu.com/p/vjgmjHcTf6/

[Corey Bryant (corey.bryant)]

I've just compared jquery.bootstrap.wizard.js files from 2 installations:
1) installed openstack-dashboard newton and upgraded to ocata
2) installed openstack-dashboard ocata

/usr/share/openstack-dashboard/horizon/static/horizon/lib/jquery/jquery.bootstrap.wizard.js
and
/var/lib/openstack-dashboard/static/horizon/lib/jquery/jquery.bootstrap.wizard.js
are the same for both installations (after upgrade to ocata for 1st installation).

Note that this is just an openstack-dashboard package install. I currently don't have a test environment for charms but will try again with charms once our test environment is back up and running.

[Seyeong Kim (seyeongkim)]

right, source code itself doesn't have any differences ( if compare same file name )

but compressed js is different(&broken) because extra file is copied to /var/lib

and it seems extra code inserted to compressed js (& broken, you can see broken(incomplete) source in compressed.js )

[Corey Bryant (corey.bryant)]

Seyeong, I took a look at this a bit today and was able to reproduce it. I think we can add the following. I'm curious if this exists for any other releases or if any similar issues exist elsewhere. Either way it seems like we'd want to fix this through for ocata and above.

diff --git a/debian/openstack-dashboard.preinst b/debian/openstack-dashboard.preinst
index c74e321e..03e9754f 100644
--- a/debian/openstack-dashboard.preinst
+++ b/debian/openstack-dashboard.preinst
@@ -15,6 +15,7 @@ dpkg-maintscript-helper mv_conffile \
   1:2013.2~b2-0ubuntu3 -- "$@"

 rm -rf /usr/share/openstack-dashboard/static/ || :
+rm -rf /usr/share/openstack-dashboard/horizon/static/ || :
 rm -rf /var/lib/openstack-dashboard/static/ || :

 #DEBHELPER#

[Seyeong Kim (seyeongkim)]

Corey, Yes, possible.

I thought upgrading 2 release at once is not recommended so I wanted to ask somebody it is needed.

I'll test from Newton to Pike and Queens.

[Corey Bryant (corey.bryant)]

Sorry, I wasn't referring to skipping a release on upgrade. I was referring to Ocata->Pike, Pike->Queens.

[Seyeong Kim (seyeongkim)]

ah I see, will check ASAP

[Seyeong Kim (seyeongkim)]

Hello Corey,

I tested Ocata->Pike, Pike->Queens.

and There is no issue like this.

I could see container dialog properly.

Please let me know if you need anything.

Thanks.

[Seyeong Kim (seyeongkim)]

Hello, Could you please proceed this? or I can do it.

Please let me know if you are ok.

Thanks.

[Corey Bryant (corey.bryant)]

I've triaged this for all releases >= ocata. While there are no issues surfacing currently for any release other than ocata, it seems we should make this change in all releases >= ocata.

[Corey Bryant (corey.bryant)]

Actually, we only rm /usr/share/openstack-dashboard/static/ in <= ocata, so I think we can limit this fix to ocata, where we're seeing the issue.

[Corey Bryant (corey.bryant)]

Please disregard comment #18.

[Corey Bryant (corey.bryant)]

Hello Seyeong, or anyone else affected,

Accepted horizon into ocata-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:ocata-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-ocata-needed to verification-ocata-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-ocata-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Seyeong Kim (seyeongkim)]

Hello,

I verified this issue fixed.

Below are steps I followed

1. deploy newton env.
2. upgrade it to ocata.
3. check if there is ui error ( it was )
4. upgrade pkg from -proposed ocata
5. check if there is ui error ( it's gone )

Thanks a lot

root@juju-4a8232-0-lxd-5:~# dpkg -l | grep dashboard
ii openstack-dashboard 3:11.0.4-0ubuntu1~cloud2.5 all Django web interface for OpenStack
ii openstack-dashboard-ubuntu-theme 3:11.0.4-0ubuntu1~cloud2.5 all Transitional dummy package for Ubuntu theme for Horizon

[Seyeong Kim (seyeongkim)]

Hello corey,

Could you please check this case if you have time?

Thanks a lot

[Corey Bryant (corey.bryant)]

I've also given an ocata-proposed deployment some manual testing and it looks good.

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for horizon has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package horizon - 3:11.0.4-0ubuntu1~cloud2.5
---------------

 horizon (3:11.0.4-0ubuntu1~cloud2.5) xenial-ocata; urgency=medium
 .
   * d/openstack-dashboard.preinst: Prior to installing package, ensure
     /usr/share/openstack-dashboard/horizon/static/ is removed. This ensures
     there are no conflicts on upgrades to Ocata (LP: #1775170).

[Jared Baker (shubjero)]

This particular bug of the container creation modal being blank has plagued us for several OpenStack releases now. We are running Queens on Ubuntu 16.04 currently (http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-updates/queens/main amd64 Packages)

python-django-horizon 3:13.0.1-0ubuntu2~cloud0

We are essentially seeing the same (or at least similar) problem. When we run python manage.py collectstatic -c && python manage.py compress some of the dynamicly generated javascript files are incomplete/truncated and causes this container creation modal to be blank.

We have no idea whats causing the javascript to be generated in an incomplete way and has been very difficult for us to reproduce in a lab environment. This issue is mostly unique to our production environment however in the past we have been able to create this issue in the lab (this was probably on ocata).

Does anyone have any guidance on this?

Thanks!

[Corey Bryant (corey.bryant)]

Hi Jared,

Are you adding any of your own static assets? Otherwise the package should collect/compress static assets on it's own in the postinst script.

For reference the fix for newton->ocata was:
https://git.launchpad.net/~ubuntu-server-dev/ubuntu/+source/horizon/commit/?id=5d1b77991c657774e0d172f89cedad44aa46a62d

It's possible this or a similar fix is needed on other releases as well.

If you can provide some simple steps to allow me to recreate that'd be helpful.

Thanks,
Corey

[Jared Baker (shubjero)]

We implement the following customizations of the openstack-horizon dashboard:

- /usr/share/openstack-dashboard/openstack_dashboard/templates/_login_footer.html
- /usr/share/openstack-dashboard/openstack_dashboard/templates/_footer.html
- /usr/share/openstack-dashboard/openstack_dashboard/static/dashboard/img/logo.svg
- /usr/share/openstack-dashboard/openstack_dashboard/static/dashboard/img/logo-splash.svg

We serve the openstack dashboard out of '/' instead of Ubuntu's default of '/horizon' and we do so by modifying the following:

- /etc/openstack-dashboard/local_settings.py
   WEBROOT = '/'
   LOGIN_URL = WEBROOT + 'auth/login/'
   LOGOUT_URL = WEBROOT + 'auth/logout/'

- /etc/apache2/conf-available/openstack-dashboard.conf
   WSGIScriptAlias / /usr/share/openstack-dashboard/openstack_dashboard/wsgi/django.wsgi process-group=horizon
   Alias /static /var/lib/openstack-dashboard/static/
   Alias /horizon/static /var/lib/openstack-dashboard/static/

[Jared Baker (shubjero)]

Just to be clear, I've tried removing the following directories

rm -rf /usr/share/openstack-dashboard/
rm -rf /var/lib/openstack-dashboard/static/

followed by an apt-get install --reinstall openstack-dashboard python-django-horizon

Leaving the dashboard uncustomized, I am still getting a blank create container window.

Chrome developer console when loading '/project/containers/' shows:

Uncaught SyntaxError: Invalid or unexpected token for 95f4db675ac3.js:1

[Corey Bryant (corey.bryant)]

You might also try re-running the post-install script after the package is installed and after you've added your customizations:

sudo sh /var/lib/dpkg/info/openstack-dashboard.postinst configure

[Corey Bryant (corey.bryant)]

Jared, If I were to recreate this without your additional customizations, would I need to upgrade from pike->queens or just install openstack-dashboard for queens, or something else?

[Jared Baker (shubjero)]

I don't think it's necessary to go through the process of a pike -> queens upgrade to reproduce the issue. Reason I say that is because we can take our current install and purge & install the openstack-dashboard & related packages and without our customizations we still experience the problem. That being said we are unable to reproduce this issue from a fresh install in our lab environment by taking a fresh ubuntu 16.04 install and layering on the openstack packages. This has been good and bad. Bad because we are unable to reproduce the issue in our lab. Good because we can take this working environment and copy the dynamically generated javascript files from '/var/lib/openstack-dashboard/static/dashboard/js' to our production environment so that the create container modal works properly.

Is there any way to get a more verbose output from the generation (collectstatic and compress) of those javascript files to see where it might be breaking?

[Corey Bryant (corey.bryant)]

It looks like you can use -v3:
  -v {0,1,2,3}, --verbosity {0,1,2,3}
                        Verbosity level; 0=minimal output, 1=normal output,
                        2=verbose output, 3=very verbose output

You mentioned 95f4db675ac3.js had an error. You might compare good vs bad versions of that file and maybe that'll help narrow things down.

[Jared Baker (shubjero)]

Hey Corey,

I believe I have found the source of my problem with bad javascript being generated at the compress and collect static stage and it has to do with whats in my /etc/openstack-dashboard/local_settings.py, specifically my memcached section. We run 3 controllers and 3 instances of memcached. If we configure our local_settings.py with the 3 memcached servers, bad javascript is generated at the compress and collectstatic stage. If we just use a single memcached server, the javascript generation is just fine. Any idea why this might be? Is our syntax for the cache stanza incorrect? There's no examples provided on the OpenStack site on how multiple memcached servers would look in terms of syntax. Either way, even the 'bad' stanza as written below seems to work in our lab (in terms of javascript generation)

GOOD:
SESSION_ENGINE = 'django.contrib.sessions.backends.cache'
CACHES = {
   'default': {
       'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
       'LOCATION': '172.25.4.17:11211',
   },
}

BAD:
SESSION_ENGINE = 'django.contrib.sessions.backends.cache'
CACHES = {
   'default': {
       'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
       'LOCATION': ['172.25.4.17:11211','172.25.4.18:11211','172.25.4.19:11211'],
   },
}

[Jared Baker (shubjero)]

I did a bit more investigating and it seems to me like the contents of the defined memcached server can influence the generation of the javascript in '/var/lib/openstack-dashboard/static/dashboard/js'

Is that assumption correct? If so, maybe the collect/compress should flush the memcached contents?

[Corey Bryant (corey.bryant)]

Good work finding that out, Jared. I think we need some input from upstream horizon developers to determine if this is an upstream bug or not. I'll add upstream horizon to this bug.