vsphere bootstrap: go panic comparing uncomparable type map[string]interface {}

[David Britton]

This happens on bootstrap in vsphere 6.0.0 with juju 2.3.2 from the snap:

Will attach more details as attachments.

2018-02-02 16:52:57 CRITICAL juju.cmd.jujud main.go:204 Unhandled panic:
runtime error: comparing uncomparable type map[string]interface {}
goroutine 1 [running]:
main.Main.func1()
        /tmp/tmpWNw4Dv/juju-core_2.3.2/src/github.com/juju/juju/cmd/jujud/main.go:203 +0xbc
panic(0x2b42e80, 0xc420c27e10)
        /var/lib/jenkins/crossbuild/golang-1.9/src/runtime/panic.go:491 +0x283
github.com/juju/juju/state.(*Settings).settingsUpdateOps(0xc420085a90, 0xc42025dbf0, 0x31229a4, 0x19, 0xc420a62ad8, 0xc420c26001, 0xc420c269d0)
        /tmp/tmpWNw4Dv/juju-core_2.3.2/src/github.com/juju/juju/state/settings.go:175 +0x780
github.com/juju/juju/state.(*Model).UpdateModelConfig(0xc42039aea0, 0xc42006f800, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x18, 0xedcd84)
        /tmp/tmpWNw4Dv/juju-core_2.3.2/src/github.com/juju/juju/state/modelconfig.go:346 +0x67c
github.com/juju/juju/state.(*Model).AutoConfigureContainerNetworking(0xc42039aea0, 0x4d6de40, 0xc420450fa0, 0xc420594b68, 0x8)
        /tmp/tmpWNw4Dv/juju-core_2.3.2/src/github.com/juju/juju/state/containernetworking.go:39 +0x179
github.com/juju/juju/agent/agentbootstrap.InitializeState(0x30e4b59, 0x5, 0x0, 0x0, 0x7fcee8601e50, 0xc4200c4dc0, 0xc420399230, 0x1, 0xc420594b68, 0x8, ...)
        /tmp/tmpWNw4Dv/juju-core_2.3.2/src/github.com/juju/juju/agent/agentbootstrap/bootstrap.go:221 +0x1537
main.(*BootstrapCommand).Run.func2(0x7fcee8601e50, 0xc4200c4dc0, 0xc4200c4dc0, 0x7fcee8601e50)
        /tmp/tmpWNw4Dv/juju-core_2.3.2/src/github.com/juju/juju/cmd/jujud/bootstrap.go:266 +0x436
github.com/juju/juju/cmd/jujud/agent.(*agentConf).ChangeConfig(0xc4204b3aa0, 0xc4205b0900, 0x0, 0x0)
        /tmp/tmpWNw4Dv/juju-core_2.3.2/src/github.com/juju/juju/cmd/jujud/agent/agent.go:103 +0xba
main.(*BootstrapCommand).Run(0xc4204b3ad0, 0xc420261770, 0x0, 0x0)
        /tmp/tmpWNw4Dv/juju-core_2.3.2/src/github.com/juju/juju/cmd/jujud/bootstrap.go:250 +0xc1a
github.com/juju/cmd.(*SuperCommand).Run(0xc42015e5a0, 0xc420261770, 0xc420261770, 0x0)
        /tmp/tmpWNw4Dv/juju-core_2.3.2/src/github.com/juju/cmd/supercommand.go:456 +0x2e7
github.com/juju/cmd.Main(0x4d64100, 0xc42015e5a0, 0xc420261770, 0xc42008a010, 0x7, 0x7, 0x0)
        /tmp/tmpWNw4Dv/juju-core_2.3.2/src/github.com/juju/cmd/cmd.go:317 +0x266
main.jujuDMain(0xc42008a000, 0x8, 0x8, 0xc420261770, 0x0, 0x0, 0x0)
        /tmp/tmpWNw4Dv/juju-core_2.3.2/src/github.com/juju/juju/cmd/jujud/main.go:186 +0x911
main.Main(0xc42008a000, 0x8, 0x8, 0x0)
        /tmp/tmpWNw4Dv/juju-core_2.3.2/src/github.com/juju/juju/cmd/jujud/main.go:219 +0x1d9
main.MainWrapper(0xc42008a000, 0x8, 0x8)
        /tmp/tmpWNw4Dv/juju-core_2.3.2/src/github.com/juju/juju/cmd/jujud/main.go:194 +0x3f
main.main()
        /tmp/tmpWNw4Dv/juju-core_2.3.2/src/github.com/juju/juju/cmd/jujud/main_nix.go:22 +0x45

[Anastasia (anastasia-macmood)]

The panic is coming from comparing new and old settings values when storing setting during bootstrap, https://github.com/juju/juju/blob/2.3/state/settings.go#L176

Interestingly, the code snippet between 2.3 and develop varies in the actual comparison implementation.
The code from 2.3 that is creating this panic:

 if reflect.DeepEqual(new, old) {
   continue
  }

The code from develop, which may not able to do the comparison either:

 if new == old {
   continue
  }

Either code path may not behave as expected...

[Anastasia (anastasia-macmood)]

So the code that sets/writes settings values needs to be more robust (or at least more descriptive when panicing). I will try to reproduce the panic and make code better.

Meanwhile, the actual container networking error is coming from juju/juju/state/containernetworking.go

At this stage, I suspect if-stmt from https://github.com/juju/juju/blob/2.3/state/containernetworking.go#L31...

[Anastasia (anastasia-macmood)]

Is it possible that ubuntu-fan has been unpackaged and installed but not configured as per Juju expectations, as from the log we get controller config with "fan-config:" and " container-networking-method:", i.e. no value...?

[Anastasia (anastasia-macmood)]

@David Britton,

The panic is coming from the comparison of vsphere config values.

2.3.2 implementation has introduced the ability to specify complex values but the comparison code was not capable of always doing the right thing. [Old implementation had == whereas we needed more sophisticated reflect.DeepEquals].

The good news is that it is now fixed in 2.3.3 and will be forward-ported to develop (2.4+) shortly.

Commit that fixes this issue on 2.3 tip is https://github.com/juju/juju/commit/35e2ceaef8ccde530370d20e8f294451e6267486

[David Britton (dpb)]

On Wed, Feb 07, 2018 at 12:00:02AM -0000, Anastasia wrote:
> The good news is that it is now fixed in 2.3.3 and will be forward-
> ported to develop (2.4+) shortly.

Thanks Anastasia -- I guess 2.3.3 will show up in the snap at some
point in the future? Just curious if there is an expected release
date.

Also, can I test 2.4 tip (develop) with a snap somehow?

--
David Britton <email address hidden>

[Nicholas Skaggs (nskaggs)]

David, yes snap install juju --edge --classic. Once this is in 2.4, the snap will be built and rolled out shortly after.

[Anastasia (anastasia-macmood)]

This fix has been in 2.4 branch since 2.4-beta1, so I am pretty sure that we have released it in 2.4.0.