network-get fails to find configs on 2.3.0 that worked on 2.2.6

[William Grant]

After upgrading one of my controllers from Juju 2.2.6 to 2.3.0 this morning (model still 2.2.6):

  $ juju run --unit=telegraf/0 'network-get --primary-address prometheus-client'
  ERROR no network config found for binding "prometheus-client"

On the production equivalent, which is still fully Juju 2.2.6:

  $ juju run --unit=telegraf/0 'network-get --primary-address prometheus-client'
  10.50.92.21

[Ian Booth (wallyworld)]

wgrant did some digging into their staging server
[13:59:11] <wgrant> Hmmh mmm
[13:59:19] <wgrant> Are subnets new in 2.3?
[13:59:53] <wgrant> Trying to debug this 2.3.0 upgrade network-get thing. New models are fine.
[14:00:08] <wgrant> But the only subnets in the DB are for the new model
[14:06:36] <wgrant> Looking at the code, it does not seem implausible that this is the problem.
[14:06:51] <wgrant> getOneNetworkConfig requires a corresponding subnet to exist to match the IP to the space.
[14:10:22] <wgrant> Oh interesting.
[14:11:26] <wgrant> I have a second model on the prod controller (now 2.2.6) that was created in 2.2.0, and it has subnets, while the controller model and main model (created in 2.0 or 2.1) don't have subnets.
[14:11:48] <wgrant> So I guess subnets were new in 2.2, and 2.3 breaks if they're not there, but no upgrade added them from 2.1 and earlier?

[Ian Booth (wallyworld)]

Not sure that the above is entirely relevant here since from what I can see, getOneNetworkConfig() is on v4 of the uniter which I think was obsoleted by v5 or later in 2.2.6

[Ian Booth (wallyworld)]

2.2 branch uses uniter v6 which uses NetworkInfo instead of NetworkConfig

[William Grant (wgrant)]

I can't find obvious differences in the DB state, so I wonder if this is just protocol incompatibility. Unfortunately that's difficult to test due to 2.3.0 agents not working on a model upgraded from 2.2.6 (bug #1737107).

[William Grant (wgrant)]

This seems to be a protocol incompatibility. After upgrading the controller to 2.3.1, 2.2.6 agents still couldn't network-get. Once those agents were upgraded to 2.3.1, network-get worked fine.

[Eric Claude Jones (ecjones)]

https://github.com/juju/juju/pull/8235

[Mario Splivalo (mariosplivalo)]

I hit the same issue upgrading my environment from 2.2.5 to 2.3.1.
I have two models, one with openstack on trusty and one with openstack on xenial.
Upgrading the controller model to 2.3.1 rendered all of the units on trusty model to be in error state (xenial model was fine).
Upgrading both models to 2.3.1 resolved all of the issues (xenial was fine before, but trusty one got fixed too).

[John A Meinel (jameinel)]

2.3.2 will fix this so that upgrading 2.2.X => 2.3.2 in the controller
won't break the existing 2.2 models. But yes, the current workaround is to
upgrade all models to 2.3.1. 2.3.2 should be out this week.

On Tue, Jan 16, 2018 at 2:30 PM, Mario Splivalo <
<email address hidden>> wrote:

> I hit the same issue upgrading my environment from 2.2.5 to 2.3.1.
> I have two models, one with openstack on trusty and one with openstack on
> xenial.
> Upgrading the controller model to 2.3.1 rendered all of the units on
> trusty model to be in error state (xenial model was fine).
> Upgrading both models to 2.3.1 resolved all of the issues (xenial was fine
> before, but trusty one got fixed too).
>
> --
> You received this bug notification because you are subscribed to juju.
> Matching subscriptions: juju bugs
> https://bugs.launchpad.net/bugs/1737058
>
> Title:
> network-get fails to find configs on 2.3.0 that worked on 2.2.6
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1737058/+subscriptions
>

[William Grant (wgrant)]

2.3.2, 2.3.3 and 2.3.4 are still broken.

$ juju bootstrap --agent-version=2.2.6 --bootstrap-series=xenial
$ juju deploy cs:mysql
$ juju run --unit=mysql/0 "network-get --primary-address cluster"
10.50.79.120
$ juju upgrade-juju -m controller --agent-version 2.3.2
started upgrade to 2.3.2
$ juju run --unit=mysql/0 "network-get --primary-address cluster"
ERROR no network config found for binding "cluster"

[Eric Claude Jones (ecjones)]

https://github.com/juju/juju/pull/8500

[John A Meinel (jameinel)]

marking Released in 2.4 because there hasn't been a 2.4 release without this fix, and it is fixed in the branch.