juju-deployer/jujuclient incompatibility with 1.21.3

[James Page]

I've started seeing this problem since 1.21.3 was released - things where OK with 1.21.1.

$ juju-deployer -c basic-next.yaml -S -d -L
2015-02-25 08:43:17 [DEBUG] deployer.cli: Using runtime GoEnvironment on maas
2015-02-25 08:43:17 [INFO] deployer.cli: Using deployment basic
2015-02-25 08:43:17 [INFO] deployer.cli: Starting deployment of basic
2015-02-25 08:43:17 [DEBUG] deployer.import: Getting charms...
2015-02-25 08:43:17 [DEBUG] deployer.charm: Cache dir /home/shared/.juju/.deployer-store-cache/cs_trusty_ntp-6
2015-02-25 08:43:17 [DEBUG] deployer.deploy: Resolving configuration
2015-02-25 08:43:17 [DEBUG] deployer.env: Connecting to environment...
2015-02-25 08:43:18 [DEBUG] deployer.env: Connected to environment
2015-02-25 08:43:18 [INFO] deployer.import: Deploying services...
2015-02-25 08:43:18 [DEBUG] deployer.import: <deployer.env.go.GoEnvironment object at 0x7f5485cb46d0>
Traceback (most recent call last):
  File "/usr/local/bin/juju-deployer", line 9, in <module>
    load_entry_point('juju-deployer==0.4.3', 'console_scripts', 'juju-deployer')()
  File "/usr/local/lib/python2.7/dist-packages/deployer/cli.py", line 130, in main
    run()
  File "/usr/local/lib/python2.7/dist-packages/deployer/cli.py", line 228, in run
    importer.Importer(env, deployment, options).run()
  File "/usr/local/lib/python2.7/dist-packages/deployer/action/importer.py", line 196, in run
    self.deploy_services()
  File "/usr/local/lib/python2.7/dist-packages/deployer/action/importer.py", line 72, in deploy_services
    env_status = self.env.status()
  File "/usr/local/lib/python2.7/dist-packages/deployer/env/go.py", line 203, in status
    return self.client.get_stat()
  File "/usr/local/lib/python2.7/dist-packages/jujuclient.py", line 600, in get_stat
    return StatusTranslator().run(watch)
  File "/usr/local/lib/python2.7/dist-packages/jujuclient.py", line 1000, in run
    self._unit(d)
  File "/usr/local/lib/python2.7/dist-packages/jujuclient.py", line 1032, in _unit
    for p in ports:
TypeError: 'NoneType' object is not iterable

[James Page (james-page)]

Seems related to bug 1420403

[Martin Packman (gz)]

Two watcher related changes in since 1.21.1

<https://github.com/juju/juju/pull/1558>

<https://github.com/juju/juju/pull/1595>

Seems the megawatcher that python-jujuclient uses has a backwards incompatible change.

[James Page (james-page)]

Confirming that the client version is 1.21.3, using juju-deployer as installed using pip (should be up-to-date).

Specifically I'm using the branch lp:openstack-charm-testing (0mq.yaml) but any deployer configuration should do the same.

[Dimiter Naydenov (dimitern)]

I can't reproduce this with 1.21.3 or 1.22-beta4 in a freshly bootstrapped environment. I also asked frankban to do a separate test with juju-quickstart which uses the same 4.3.0 version of python-jujuclient. I quadruple-checked both fixes mentioned in comment #2 have actually landed and are inside the release package for 1.21.3 in the stable ppa. Can we have some debug logs from the juju machine 0 ? Inside the machine log (at DEBUG level) it will become apparent if the megawatcher's UnitInfo change it sends has the Ports and PortRanges set as per the fixes.

[Ryan Beisner (1chb1n)]

We saw the same breakage throughout OpenStack charm testing CI. Here's the patch i'm rolling out in UOSCI for the time being. It gets deployer working again. http://paste.ubuntu.com/10409429/

# FYI, version info:
jenkins@juju-beis0-machine-3:~/bzr/flat/bundles/other$ apt-cache policy juju-core
juju-core:
  Installed: 1.21.3-0ubuntu1~14.04.1~juju1
  Candidate: 1.21.3-0ubuntu1~14.04.1~juju1
  Version table:
 *** 1.21.3-0ubuntu1~14.04.1~juju1 0
        500 http://ppa.launchpad.net/juju/stable/ubuntu/ trusty/main amd64 Packages
        100 /var/lib/dpkg/status
     1.20.11-0ubuntu0.14.04.1 0
        500 http://nova.clouds.archive.ubuntu.com/ubuntu/ trusty-updates/universe amd64 Packages
     1.18.1-0ubuntu1 0
        500 http://nova.clouds.archive.ubuntu.com/ubuntu/ trusty/universe amd64 Packages

jenkins@juju-beis0-machine-3:~/bzr/flat/bundles/other$ apt-cache policy juju-deployer
juju-deployer:
  Installed: 0.4.3-0ubuntu1~ubuntu14.04.1~ppa1
  Candidate: 0.4.3-0ubuntu1~ubuntu14.04.1~ppa1
  Version table:
 *** 0.4.3-0ubuntu1~ubuntu14.04.1~ppa1 0
        500 http://ppa.launchpad.net/juju/stable/ubuntu/ trusty/main amd64 Packages
        100 /var/lib/dpkg/status
     0.3.6-0ubuntu2 0
        500 http://nova.clouds.archive.ubuntu.com/ubuntu/ trusty/universe amd64 Packages

jenkins@juju-beis0-machine-3:~/bzr/flat/bundles/other$ apt-cache policy python-jujuclient
python-jujuclient:
  Installed: 0.18.4-5
  Candidate: 0.18.4-5
  Version table:
 *** 0.18.4-5 0
        500 http://ppa.launchpad.net/juju/stable/ubuntu/ trusty/main amd64 Packages
        100 /var/lib/dpkg/status
     0.17.5-0ubuntu2 0
        500 http://nova.clouds.archive.ubuntu.com/ubuntu/ trusty/universe amd64 Packages

[Jason Hobbs (jason-hobbs)]

This is affecting OIL - it's causing every single OIL deployment to fail.

[Dimiter Naydenov (dimitern)]

I've confirmed the 1.21.3 amd64 release for trusty in the stable ppa has the expected features (e.g. https://github.com/juju/juju/pull/1616 which is the last PR in 1.21 before the release is included). I wasn't completely sure something was not wrong with the release package.

I did an experiment with that same build, bootstrapping a local environment and adding a few ubuntu units, one juju-gui. Then monitored the logs as I made changes to open/close ports. The results confirm both "Ports" and "PortRanges" fields of the "Deltas" reported by the AllWatcher are either populated or empty, but never "null".

http://paste.ubuntu.com/10411358/

[Edward Hope-Morley (hopem)]

Why is this Incomplete? I am also hitting this issue and the problem and I am running 1.21.3

[Curtis Hovey (sinzui)]

We cannot re-produce this issue. Which versions of juju-deployer, python-jujuclient, and python-websocket-client are you using?

[Jason Hobbs (jason-hobbs)]

Here's the versions:

juju-deployer 0.4.3-0ubuntu1~ubuntu12.04.1~ppa1
python-jujuclient 0.18.4-5 Python API client for juju-core
python-websocket 0.18.0-0ubuntu0.12.04~ppa6

[Edward Hope-Morley (hopem)]

FWIW the linked patch fixes this problem and broken juju debug-log for me.

Versions:

juju-deployer: 0.4.3-0ubuntu1~ubuntu14.04.1~ppa1
python-jujuclient: 0.18.4-3
python-websocket: 0.18.0-0ubuntu0.14.04~ppa5

[Jason Hobbs (jason-hobbs)]

As requested, here is machine-0.log at DEBUG level:

https://pastebin.canonical.com/126350/

[Dimiter Naydenov (dimitern)]

OK, now we're getting somewhere! Thanks for the logs Jason!

I can indeed see there's a Ports: null, PortRanges: null in the Deltas reported by AllWatcher. In one case only - unit nova-compute/0, for which I can previously see an error: 2015-02-25 21:23:01 ERROR juju.state.unit unit.go:648 unit nova-compute/0 cannot get assigned machine: unit "nova-compute/0" is not assigned to a machine

But later it gets assigned to machine 4. So the problem seems a minor issue - race condition between a new unit getting created for a service and assigning it to a machine. Because port ranges are associated with machines, not units anymore, having null Ports and PortRanges makes sense in this case. This is easily fixable by just lying - i.e. returning empty ports and ranges lists when in fact there are none associated yet (i.e. in PR #1595 that fixed the other cases we should in both cases in updated() ensure if info.Ports has length of 0, set it to an empty slice).

However, this issue at first glance, should be happening much less frequently than it is in practise. Either the deployer is too quick to check the status after adding the nova-compute service (and before its first unit is assigned), or juju's watcher is reporting it too early (and not handling the Ports / PortRanges case being nil) - most likely the latter, as juju should be resilient against this.

This affects 1.21, 1.22 and 1.23 equally.

[Kapil Thangavelu (hazmat)]

Deployer functional test should have been in core release Qa tests A long
time ago

On Wednesday, February 25, 2015, Curtis Hovey <email address hidden> wrote:

> ** Changed in: juju-core
> Milestone: 1.21.4 => 1.23
>
> ** Changed in: juju-core
> Importance: Critical => High
>
> ** Changed in: juju-core/1.23
> Status: New => Triaged
>
> ** Changed in: juju-core/1.22
> Status: New => Triaged
>
> ** Changed in: juju-core/1.23
> Importance: Undecided => Medium
>
> ** Changed in: juju-core/1.22
> Importance: Undecided => Medium
>
> --
> You received this bug notification because you are subscribed to juju-
> core.
> https://bugs.launchpad.net/bugs/1425435
>
> Title:
> juju-deployer/jujuclient incompatibility with 1.21.3
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju-core/+bug/1425435/+subscriptions
>

[Curtis Hovey (sinzui)]

As 1.22 is stablising and there is a work around for jujuclient. Engineers will fix the issue in 1.23 and 1.22.

[Darryl Weaver (dweaver)]

We are also seeing the same issue with our Orange Box demos.
We use the juju stable PPA and upgrading to 1.21.3, breaks our openstack deployments.
If I manually downgrade to 1.21.1 again, the deployments work again.

[Darryl Weaver (dweaver)]

I am using the packages upgraded from the Juju stable PPA.
i.e.:
juju-core 1.21.3-0ubuntu1~14.04.1~juju1
juju-deployer 0.4.3-0ubuntu1~ubuntu14.04.1~ppa1
python-jujuclient 0.18.4-5
websocket-client 0.18.0-0ubuntu0.14.04~ppa5

I have the option to downgrade to juju-core version 1.20.11, but we were already using 1.21.
How do I get back to a working state using the package manager?

[Andres Rodriguez (andreserl)]

This has affected OIL and has completely broken our environment. This should be marked as Critical!

[Dimiter Naydenov (dimitern)]

Fix for 1.22 landed with https://github.com/juju/juju/pull/1707; Fix for 1.23 is proposed with https://github.com/juju/juju/pull/1720