Juju gui is not loading. Because: "Error":"unit not found"

[lithium]

When I login in juju gui it stick with message "start adding charms!", actually i have over 15 charms deployed and working well with juju cli

If I start juju gui in debug mode:

/usr/bin/python /usr/local/bin/runserver.py --logging="debug" --guiroot="/var/lib/juju-gui/juju-gui/build-prod" --sslpath="/etc/ssl/juju-gui" --charmworldurl="https://manage.jujucharms.com/" --apiurl="wss://xxxx:17070" --apiversion="go"

I can see this message when i try to load juju gui web:

[D 150815 20:43:22 handlers:193] GET /ws/environment/60533555-9829-40fe-83d4-1a7de8f32e99/api (xxx) client -> juju: {"Type":"AllWatcher","Request":"Next","Id":"1","Params":{},"RequestId":5}

[D 150815 20:43:22 handlers:213] GET /ws/environment/60533555-9829-40fe-83d4-1a7de8f32e99/api (xxx) juju -> client: {"RequestId":5,"Error":"unit not found","ErrorCode":"not found","Response":{}}

It's a juju problem? or a juju gui problem?

[lithium (rudicba)]

Using juju debug-log i see:

machine-0: 2015-08-15 20:53:15 ERROR juju.apiserver.common resource.go:119 error stopping *state.Multiwatcher resource: unit not found

This problem just happen using GUI, CLI works fine

[lithium (rudicba)]

Forget: this happend in:

juju 1.23 and in juju 1.24

[james beedy (jamesbeedy)]

This exists in 1.24.5 and (after upgrade) in 1.24.6.

This issue did not initially exist, only after I destroyed the juju-gui and redeployed.

Screen shot of juju-gui: http://cl.ly/image/0h1N3P3k0Z28

Output of http://pastebin.ubuntu.com/12544350/ : http://paste.ubuntu.com/12544391/

[Brad Crittenden (bac)]

I've tried reproducing this problem without success.

$ juju version
1.24.6-vivid-amd64

I deployed the gui and mysql to both a local provider and to ec2. In both cases I was able to connect to the gui.

On ec2 I then did

% juju destroy-service juju-gui
% juju deploy cs:trusty/juju-gui --to=0
% juju expose juju-gui

After it came back up I was able to connect with no problem and could see my environment with mysql.

I repeated the above on ec2 but this time did not put the gui on machine 0 but let it create a new machine. In that scenario I could still connect.

I have not yet tried establishing the environment with an older juju and then upgrading and reconnecting. Perhaps I will try that next.

Further suggestions on how to reproduce are welcome.

[Edwin Gnichtel (ned-4)]

We have a production Juju deployment that has been hit with this bug. I'd be happy to provide whatever log or debug information from either machine 0 or the juju GUI machine/unit.

This began after we upgraded from 1.23.3 -> and persists in 1.25.0 (which is our currently deployed version, along with rev 41 of the Trusty GUI charm). We have a dev environment which was freshly deployed with 1.24.7 which has not exhibited this behavior upon upgrade to 1.25.0.

Whenever attempting to load the WebGUI, we see this repeated in the machine-0.log:

2015-11-10 02:27:16 ERROR juju.apiserver.common resource.go:119 error stopping *state.Multiwatcher resource: unit not found

I'd greatly appreciate suggestions on a workaround and would be happy to provide whatever info is needed from our side to see this fixed.

Thanks!

[Brad Crittenden (bac)]

Thanks for the additional information Edwin. I'm trying again to reproduce and will contact you if I need more information.

[Brad Crittenden (bac)]

Hi Edwin,

I tried several times to reproduce the problem and still cannot. I took careful notes the last time and have logged them in the attachment. Could you have a look and see if there are other steps I should take to reproduce the problem? Thanks.

[Edwin Gnichtel (ned-4)]

Let me see if we can develop steps on our side to reproduce this; we will have to take a little time standing up test environment with the same moving parts as our production environment (with the same upgrade path). Unfortunately this hasn't occurred on any of our developers' juju-local setups or in our existing Dev-lab environment, though the upgrade paths were not identical to our prod environment which could be why this only occurred there (exactly where we wouldn't want it to show up - murphy's law).

In the meantime, would it help to have us set "juju set-env logging-config=juju=DEBUG" and post logs?

Thanks,

-N

[Brad Crittenden (bac)]

Edwin, thank you for trying to make a minimal environment to reproduce.

If you can post logs (make sure they are redacted of any access keys or secrets) that would be great.

[mahmoh (mahmoh)]

Hi Brad, Edwin's setup is using MAAS 1.7.6 (which he can't upgrade because of another bug), Brad could try and reproduce with MAAS? Also, they're prod env has been upgraded a few times, there may be history there that's also impacting this bug. If access is needed to help debug pls advise out of band.

[Cheryl Jennings (cherylj)]

I was able to recreate the problem captured in the screen shot in comment #3 with the following steps:

1 - bootstrap with 1.22.8
2 - deploy juju-gui and another service
3 - upgrade to juju 1.25.0
4 - refresh the gui

I thought this might be related to bug #1516989, but when I ran the script to correct the unit statuses in the database, it didn't fix the gui problem. I tried refreshing the gui, and also redploying juju-gui with the same result.

Bootstrapping again with some additional debugging to try and track the problem down.

[Cheryl Jennings (cherylj)]

Ah, okay, I was just missing a step. Once I applied the script to fix the unit statuses, I had to restart jujud on machine-0, refresh juju-gui and I was able to see the units again.

The script to address bug #1516989 is here: http://paste.ubuntu.com/14474251/

Steps to resolve this issue:
- On the state server, install mongodb-clients (installs the mongo tool to allow us to manually modify the database)
- Run the insert-statuses.sh script on the state server (machine 0): ./insert-statuses.sh 0 (if your state server is not machine 0, change 0 to be the correct machine number)
- Restart jujud on your state server (you can just kill -9 the process and it will restart)
- Refresh the juju-gui