panic: empty value for "max-txn-log-size" found in configuration

[John A Meinel]

Upgrading a controller from 2.2-beta4 to 2.2-rc1 (development preview) gives this failure in the logs:
panic: empty value for "max-txn-log-size" found in configuration (type <nil>, val <nil>)

goroutine 315 [running]:
github.com/juju/juju/controller.Config.mustString(0xc4206cd7a0, 0x263dae6, 0x10, 0xc420699220, 0xd)
        /home/jameinel/dev/go/src/github.com/juju/juju/controller/config.go:199 +0x1f0
github.com/juju/juju/controller.Config.MaxTxnLogSizeMB(0xc4206cd7a0, 0x8)
        /home/jameinel/dev/go/src/github.com/juju/juju/controller/config.go:313 +0x40
github.com/juju/juju/state.collectionSchema.Create(0xc4206cd7d0, 0xc42067c720, 0xc4206cd7a0, 0x12, 0xc4206d82a0)
        /home/jameinel/dev/go/src/github.com/juju/juju/state/database.go:186 +0x561
github.com/juju/juju/state.InitDatabase(0xc4203804e0, 0xc4206cd7a0, 0x0, 0x0)
        /home/jameinel/dev/go/src/github.com/juju/juju/state/open.go:280 +0xa1
github.com/juju/juju/state.open(0xc4201ee786, 0x24, 0xc42069be00, 0x6fc23ac00, 0xdf8475800, 0x0, 0x0, 0x26f2c78, 0x0, 0xc420103920, ...)
        /home/jameinel/dev/go/src/github.com/juju/juju/state/open.go:160 +0x35b
github.com/juju/juju/state.Open(0x3a33020, 0x3a9adb8, 0xc4201ee63b, 0x24, 0xc4201ee786, 0x24, 0xc42069be00, 0x6fc23ac00, 0xdf8475800, 0x0, ...)
        /home/jameinel/dev/go/src/github.com/juju/juju/state/open.go:101 +0x10d
github.com/juju/juju/cmd/jujud/agent.(*MachineAgent).openStateForUpgrade(0xc42000b300, 0xc4202cc5e4, 0x2, 0xaea401)
        /home/jameinel/dev/go/src/github.com/juju/juju/cmd/jujud/agent/machine.go:861 +0x314
github.com/juju/juju/cmd/jujud/agent.(*MachineAgent).(github.com/juju/juju/cmd/jujud/agent.openStateForUpgrade)-fm(0xc4201323c0, 0xe, 0x26369cf)
        /home/jameinel/dev/go/src/github.com/juju/juju/cmd/jujud/agent/machine.go:537 +0x2a
github.com/juju/juju/worker/upgradesteps.(*upgradesteps).run(0xc420694360, 0x0, 0x0)
        /home/jameinel/dev/go/src/github.com/juju/juju/worker/upgradesteps/worker.go:194 +0x399
github.com/juju/juju/worker/upgradesteps.NewWorker.func1(0xc420694360)
        /home/jameinel/dev/go/src/github.com/juju/juju/worker/upgradesteps/worker.go:115 +0x57
created by github.com/juju/juju/worker/upgradesteps.NewWorker
        /home/jameinel/dev/go/src/github.com/juju/juju/worker/upgradesteps/worker.go:116 +0x218

At this point, the controller is no longer usable.

Even though this is config that has an upgrade step, we should *not* panic because a value isn't set in the database. We *have* a sane default value for it.

[John A Meinel (jameinel)]

After this happens, the controller doesn't appear to restart, (no ongoing 'will restart' messages in the log files, and it appears to accept connections), but 'juju status' hangs indefinitely.

Restarting the agent seems to make things work again, until I do "juju upgrade-juju" on one of the models, and then it dies with the same panic.

[Ian Booth (wallyworld)]

For methods to access config attributes on a config.Config struct, where those config attributes have a default value defined in the schema, or are mandatory, we tend to use mustString() which panics if the value is not there. eg see FirewallMode), Name(), etc. For new controllers, the schema default value is added to the config map; otherwise the upgrade step should have added the value. Investigation is needed as to why the upgrade step didn't work in this case.

[Ian Booth (wallyworld)]

We are opening state to use for running the upgrade steps but doing so requires the max-txn-log-size because we are also initialising the db collections and indices. So it's a catch 22. The way state is opened for upgrades will need to be tweaked.

[John A Meinel (jameinel)]

Thanks for jumping on this quickly.

John
=:->

On May 29, 2017 04:20, "Ian Booth" <email address hidden> wrote:

> ** Changed in: juju
> Status: In Progress => Fix Committed
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1694116
>
> Title:
> panic: empty value for "max-txn-log-size" found in configuration
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1694116/+subscriptions
>