vsphere bootstrap: go panic comparing uncomparable type map[string]interface {}

Bug #1747048 reported by David Britton on 2018-02-02
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
juju
High
Christian Muirhead
2.3
Critical
Christian Muirhead

Bug Description

This happens on bootstrap in vsphere 6.0.0 with juju 2.3.2 from the snap:

Will attach more details as attachments.

2018-02-02 16:52:57 CRITICAL juju.cmd.jujud main.go:204 Unhandled panic:
runtime error: comparing uncomparable type map[string]interface {}
goroutine 1 [running]:
main.Main.func1()
        /tmp/tmpWNw4Dv/juju-core_2.3.2/src/github.com/juju/juju/cmd/jujud/main.go:203 +0xbc
panic(0x2b42e80, 0xc420c27e10)
        /var/lib/jenkins/crossbuild/golang-1.9/src/runtime/panic.go:491 +0x283
github.com/juju/juju/state.(*Settings).settingsUpdateOps(0xc420085a90, 0xc42025dbf0, 0x31229a4, 0x19, 0xc420a62ad8, 0xc420c26001, 0xc420c269d0)
        /tmp/tmpWNw4Dv/juju-core_2.3.2/src/github.com/juju/juju/state/settings.go:175 +0x780
github.com/juju/juju/state.(*Model).UpdateModelConfig(0xc42039aea0, 0xc42006f800, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x18, 0xedcd84)
        /tmp/tmpWNw4Dv/juju-core_2.3.2/src/github.com/juju/juju/state/modelconfig.go:346 +0x67c
github.com/juju/juju/state.(*Model).AutoConfigureContainerNetworking(0xc42039aea0, 0x4d6de40, 0xc420450fa0, 0xc420594b68, 0x8)
        /tmp/tmpWNw4Dv/juju-core_2.3.2/src/github.com/juju/juju/state/containernetworking.go:39 +0x179
github.com/juju/juju/agent/agentbootstrap.InitializeState(0x30e4b59, 0x5, 0x0, 0x0, 0x7fcee8601e50, 0xc4200c4dc0, 0xc420399230, 0x1, 0xc420594b68, 0x8, ...)
        /tmp/tmpWNw4Dv/juju-core_2.3.2/src/github.com/juju/juju/agent/agentbootstrap/bootstrap.go:221 +0x1537
main.(*BootstrapCommand).Run.func2(0x7fcee8601e50, 0xc4200c4dc0, 0xc4200c4dc0, 0x7fcee8601e50)
        /tmp/tmpWNw4Dv/juju-core_2.3.2/src/github.com/juju/juju/cmd/jujud/bootstrap.go:266 +0x436
github.com/juju/juju/cmd/jujud/agent.(*agentConf).ChangeConfig(0xc4204b3aa0, 0xc4205b0900, 0x0, 0x0)
        /tmp/tmpWNw4Dv/juju-core_2.3.2/src/github.com/juju/juju/cmd/jujud/agent/agent.go:103 +0xba
main.(*BootstrapCommand).Run(0xc4204b3ad0, 0xc420261770, 0x0, 0x0)
        /tmp/tmpWNw4Dv/juju-core_2.3.2/src/github.com/juju/juju/cmd/jujud/bootstrap.go:250 +0xc1a
github.com/juju/cmd.(*SuperCommand).Run(0xc42015e5a0, 0xc420261770, 0xc420261770, 0x0)
        /tmp/tmpWNw4Dv/juju-core_2.3.2/src/github.com/juju/cmd/supercommand.go:456 +0x2e7
github.com/juju/cmd.Main(0x4d64100, 0xc42015e5a0, 0xc420261770, 0xc42008a010, 0x7, 0x7, 0x0)
        /tmp/tmpWNw4Dv/juju-core_2.3.2/src/github.com/juju/cmd/cmd.go:317 +0x266
main.jujuDMain(0xc42008a000, 0x8, 0x8, 0xc420261770, 0x0, 0x0, 0x0)
        /tmp/tmpWNw4Dv/juju-core_2.3.2/src/github.com/juju/juju/cmd/jujud/main.go:186 +0x911
main.Main(0xc42008a000, 0x8, 0x8, 0x0)
        /tmp/tmpWNw4Dv/juju-core_2.3.2/src/github.com/juju/juju/cmd/jujud/main.go:219 +0x1d9
main.MainWrapper(0xc42008a000, 0x8, 0x8)
        /tmp/tmpWNw4Dv/juju-core_2.3.2/src/github.com/juju/juju/cmd/jujud/main.go:194 +0x3f
main.main()
        /tmp/tmpWNw4Dv/juju-core_2.3.2/src/github.com/juju/juju/cmd/jujud/main_nix.go:22 +0x45

David Britton (davidpbritton) wrote :
David Britton (davidpbritton) wrote :
David Britton (davidpbritton) wrote :
Ian Booth (wallyworld) on 2018-02-02
Changed in juju:
milestone: none → 2.3.3
status: New → Triaged
importance: Undecided → Critical
Changed in juju:
assignee: nobody → Anastasia (anastasia-macmood)
status: Triaged → In Progress
Anastasia (anastasia-macmood) wrote :

The panic is coming from comparing new and old settings values when storing setting during bootstrap, https://github.com/juju/juju/blob/2.3/state/settings.go#L176

Interestingly, the code snippet between 2.3 and develop varies in the actual comparison implementation.
The code from 2.3 that is creating this panic:

 if reflect.DeepEqual(new, old) {
   continue
  }

The code from develop, which may not able to do the comparison either:

 if new == old {
   continue
  }

Either code path may not behave as expected...

Anastasia (anastasia-macmood) wrote :

So the code that sets/writes settings values needs to be more robust (or at least more descriptive when panicing). I will try to reproduce the panic and make code better.

Meanwhile, the actual container networking error is coming from juju/juju/state/containernetworking.go

At this stage, I suspect if-stmt from https://github.com/juju/juju/blob/2.3/state/containernetworking.go#L31...

Anastasia (anastasia-macmood) wrote :

Is it possible that ubuntu-fan has been unpackaged and installed but not configured as per Juju expectations, as from the log we get controller config with "fan-config:" and " container-networking-method:", i.e. no value...?

Anastasia (anastasia-macmood) wrote :

@David Britton,

The panic is coming from the comparison of vsphere config values.

2.3.2 implementation has introduced the ability to specify complex values but the comparison code was not capable of always doing the right thing. [Old implementation had == whereas we needed more sophisticated reflect.DeepEquals].

The good news is that it is now fixed in 2.3.3 and will be forward-ported to develop (2.4+) shortly.

Commit that fixes this issue on 2.3 tip is https://github.com/juju/juju/commit/35e2ceaef8ccde530370d20e8f294451e6267486

Changed in juju:
assignee: Anastasia (anastasia-macmood) → nobody
assignee: nobody → Christian Muirhead (2-xtian)
status: In Progress → Fix Committed
milestone: 2.3.3 → 2.4-beta1
importance: Critical → High
status: Fix Committed → In Progress

On Wed, Feb 07, 2018 at 12:00:02AM -0000, Anastasia wrote:
> The good news is that it is now fixed in 2.3.3 and will be forward-
> ported to develop (2.4+) shortly.

Thanks Anastasia -- I guess 2.3.3 will show up in the snap at some
point in the future? Just curious if there is an expected release
date.

Also, can I test 2.4 tip (develop) with a snap somehow?

--
David Britton <email address hidden>

Nicholas Skaggs (nskaggs) wrote :

David, yes snap install juju --edge --classic. Once this is in 2.4, the snap will be built and rolled out shortly after.

Changed in juju:
milestone: 2.4-beta1 → none
Anastasia (anastasia-macmood) wrote :

This fix has been in 2.4 branch since 2.4-beta1, so I am pretty sure that we have released it in 2.4.0.

Changed in juju:
milestone: none → 2.4.0
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers