juju status command returns runtime error: invalid memory address or nil pointer dereference

[Daniel Bidwell]

After upgrading to 2.6.1-bionic-amd64 juju status returns
runtime error: invalid memory address or nil pointer dereference

/var/log/juju/machine-0.log on the controller contains:

2019-05-14 13:15:50 CRITICAL juju.rpc server.go:558 panic running request {MethodCaller:0xc00545ed80 transformErrors:0x20c4250 hdr:{RequestId:5 Request:{Type:Client Version:2 Id: Action:FullStatus} Error: ErrorCode: ErrorInfo:map[] Version:1}} with arg {Patterns:[]}: runtime error: invalid memory address or nil pointer dereference
goroutine 912508 [running]:
runtime/debug.Stack(0xc0052a8048, 0x328ece0, 0x69dd6b0)
 /snap/go/3730/src/runtime/debug/stack.go:24 +0xa7
github.com/juju/juju/rpc.(*Conn).runRequest.func1(0xc0052a9f28, 0x3353980, 0xc0048b07e0, 0x199, 0xc0036d0820, 0x3f864a0, 0xc0048b07c0)
 /workspace/_build/src/github.com/juju/juju/rpc/server.go:559 +0xcd
panic(0x328ece0, 0x69dd6b0)
 /snap/go/3730/src/runtime/panic.go:513 +0x1b9
github.com/juju/juju/vendor/gopkg.in/juju/charm%2ev6.(*LXDProfile).Empty(0x0, 0x3f8ae60)
 /workspace/_build/src/github.com/juju/juju/vendor/gopkg.in/juju/charm.v6/lxdprofile.go:73 +0x5
github.com/juju/juju/core/lxdprofile.NotEmpty(0x3f7fc40, 0xc005644580, 0xc0052fc400)
 /workspace/_build/src/github.com/juju/juju/core/lxdprofile/validate.go:62 +0x4a
github.com/juju/juju/apiserver/facades/client/client.(*statusContext).processApplication(0xc00001fbc0, 0xc005e21d10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
 /workspace/_build/src/github.com/juju/juju/apiserver/facades/client/client/status.go:1018 +0xfe
github.com/juju/juju/apiserver/facades/client/client.(*statusContext).processApplications(0xc00001fbc0, 0xc0055d8ed0)
 /workspace/_build/src/github.com/juju/juju/apiserver/facades/client/client/status.go:1006 +0xee
github.com/juju/juju/apiserver/facades/client/client.(*Client).FullStatus(0xc005b1d500, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
 /workspace/_build/src/github.com/juju/juju/apiserver/facades/client/client/status.go:368 +0x18ef
reflect.Value.call(0x384c780, 0xc005b1d500, 0x2a13, 0x396ed84, 0x4, 0xc005a7cca0, 0x1, 0x1, 0xa263f1, 0x31154c0, ...)
 /snap/go/3730/src/reflect/value.go:447 +0x454
reflect.Value.Call(0x384c780, 0xc005b1d500, 0x2a13, 0xc005a7cca0, 0x1, 0x1, 0x1, 0x1, 0xc000ff1500)
 /snap/go/3730/src/reflect/value.go:308 +0xa4
github.com/juju/juju/rpc/rpcreflect.newMethod.func8(0x3faf320, 0xc005454300, 0x384c780, 0xc005b1d500, 0x16, 0x3353980, 0xc0048b07e0, 0x199, 0x42e592, 0xc000000000, ...)
 /workspace/_build/src/github.com/juju/juju/rpc/rpcreflect/type.go:344 +0x109
github.com/juju/juju/apiserver.(*srvCaller).Call(0xc00545ed80, 0x3faf320, 0xc005454300, 0x0, 0x0, 0x3353980, 0xc0048b07e0, 0x199, 0xc0048b07c0, 0x342fdeee2fc3, ...)
 /workspace/_build/src/github.com/juju/juju/apiserver/root.go:170 +0xd6
github.com/juju/juju/rpc.(*Conn).runRequest(0xc0036d0820, 0x3f9a820, 0xc00545ed80, 0x3adb6d0, 0x5, 0xc005261340, 0x6, 0x2, 0x0, 0x0, ...)
 /workspace/_build/src/github.com/juju/juju/rpc/server.go:572 +0x1ab
created by github.com/juju/juju/rpc.(*Conn).handleRequest
 /workspace/_build/src/github.com/juju/juju/rpc/server.go:476 +0x93f

juju show-controller returns:

alumnicontroller:
  details:
    uuid: 0b661548-7333-432e-8c46-e9856b689101
    controller-uuid: 0b661548-7333-432e-8c46-e9856b689101
    api-endpoints: ['10.2.253.248:17070']
    cloud: myvscloud
    region: New Datacenter
    agent-version: 2.6.1
    mongo-version: 3.6.3
    ca-fingerprint: 0D:AE:EA:33:6A:EA:35:0F:34:4C:F0:8E:65:D3:49:42:A4:18:3E:63:29:5F:9B:4D:44:4F:20:34:E3:80:1D:98
    ca-cert: |
      -----BEGIN CERTIFICATE-----
      MIIDrDCCApSgAwIBAgIUOYST5mq1z6P9y3Zc3nT8/DxkwP0wDQYJKoZIhvcNAQEL
      BQAwbjENMAsGA1UEChMEanVqdTEuMCwGA1UEAwwlanVqdS1nZW5lcmF0ZWQgQ0Eg
      Zm9yIG1vZGVsICJqdWp1LWNhIjEtMCsGA1UEBRMkMGI5OGZmNGYtNzA3Ni00N2Yw
      LThkNGMtYWM5NTcyZjU4M2I1MB4XDTE4MTExOTE5NTE0N1oXDTI4MTEyNjE5NTE0
      N1owbjENMAsGA1UEChMEanVqdTEuMCwGA1UEAwwlanVqdS1nZW5lcmF0ZWQgQ0Eg
      Zm9yIG1vZGVsICJqdWp1LWNhIjEtMCsGA1UEBRMkMGI5OGZmNGYtNzA3Ni00N2Yw
      LThkNGMtYWM5NTcyZjU4M2I1MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKC
      AQEAvFsuZttFDYQpCeXXyg03Y+jql4Po01Vc2IiH5qKiulZn5bSMV66USiBSwtaz
      rmeTLnBDVzUKZQ2EpZUXolxsDeBk21DUc/7aC1E8UzaDa2cBDB+/BYiUEv53Jpyn
      jTQXAtV5jd/32cpLdRUSNuCNLm65CK4Qp+2fKTnTjgLBzvySXaPUUPauToEgMQhG
      A1K4gdkS33q7xrDruTzZwQVk0Ah4ukZOt7Z8wCwXcNTgPgIIR1chN6YE8m85+QtF
      zVFixPSSq3Luq8eS/VvTQP9vsVv3Xq2cybXHZSyuEeEcKeYb6NK8TmoYXaVScCSM
      4zfFe2C21K8DJP5BcY3NsxBEWQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAqQwDwYD
      VR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQUDNklHB5aPZaCWwIbpvygCrQNnlowDQYJ
      KoZIhvcNAQELBQADggEBAFfJEDZx5KkqHFHDOfvgZ4q+mfr/7Hc6Z9m+znU1jdhD
      8PE1iHNjC+XVK1Bj63FJpeqR9PkyupEinf3ZUAGDULo41sai/qupNZPj2RY8VKQk
      dscG27UBmBlIWBTdbVwdpEhrXneIgCHLyKwOLvxL6lR7w8nNWV/QsvLdHqKb2jlP
      nKwzHb4K5n3sKLAmyfLtEHV6CiQ2wQmp7rHaomEX8EVcqxcGMbKOnKiVIxOf1PGJ
      7OTn0s178H+kgucSGEM1xOyftQKvQE+87n40PPvq31qD2kH/+axJqb/rLECSkkhA
      MXgEJGqhbh42aRnA/3hs9ebMhH8HBY+blNzUNz3YDYM=
      -----END CERTIFICATE-----
  controller-machines:
    "0":
      instance-id: juju-2f90ad-0
  models:
    controller:
      uuid: e6dadd1e-52e9-4284-8f8b-7d2b4e2f90ad
      model-uuid: e6dadd1e-52e9-4284-8f8b-7d2b4e2f90ad
      machine-count: 1
    default:
      uuid: 4d34a4e3-70db-413a-83c8-0ff21fb8f62f
      model-uuid: 4d34a4e3-70db-413a-83c8-0ff21fb8f62f
      machine-count: 5
      core-count: 12
  current-model: admin/default
  account:
    user: admin
    access: superuser

[Daniel Bidwell (bidwell)]

Of my 13 juju controllers, only 11 upgraded to 2.6.1 and 6 of the 11 upgraded controllers exhibit this problem. 5 seem to work correctly.

[John A Meinel (jameinel)]

I feel like this is likely one of those "Go interface != nil" not
evaluating to true when you have an interface with a type but a pointer
that is nil.

On Tue, May 14, 2019 at 4:35 PM Daniel Bidwell <email address hidden> wrote:

> Of my 13 juju controllers, only 11 upgraded to 2.6.1 and 6 of the 11
> upgraded controllers exhibit this problem. 5 seem to work correctly.
>
> --
> You received this bug notification because you are subscribed to juju.
> Matching subscriptions: juju bugs
> https://bugs.launchpad.net/bugs/1829014
>
> Title:
> juju status command returns runtime error: invalid memory address or
> nil pointer dereference
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1829014/+subscriptions
>

[Heather Lanigan (hmlanigan)]

@bidwell, I'm unable to reproduce so far with the charms I've chosen.

What charms, including revision numbers, are deployed into the model exhibiting this panic?

What cloud is the controller bootstrapped to?

[John A Meinel (jameinel)]

As an example:
https://play.golang.org/p/TSI50UbhVnR

In that you can see that "f" is not nil because it has a type (golang typed
nil)
But the actual underlying pointer in the type *is* nil.

If you do it instead as:
https://play.golang.org/p/Ak3qLS98WDC

You can see that the outer type matches 'nil' as you would expect.

On Tue, May 14, 2019 at 4:58 PM John Meinel <email address hidden> wrote:

> I feel like this is likely one of those "Go interface != nil" not
> evaluating to true when you have an interface with a type but a pointer
> that is nil.
>
>
> On Tue, May 14, 2019 at 4:35 PM Daniel Bidwell <email address hidden>
> wrote:
>
>> Of my 13 juju controllers, only 11 upgraded to 2.6.1 and 6 of the 11
>> upgraded controllers exhibit this problem. 5 seem to work correctly.
>>
>> --
>> You received this bug notification because you are subscribed to juju.
>> Matching subscriptions: juju bugs
>> https://bugs.launchpad.net/bugs/1829014
>>
>> Title:
>> juju status command returns runtime error: invalid memory address or
>> nil pointer dereference
>>
>> To manage notifications about this bug go to:
>> https://bugs.launchpad.net/juju/+bug/1829014/+subscriptions
>>
>

[John A Meinel (jameinel)]

If it is a typed nil issue, then I think the fix would be something like:
diff --git a/apiserver/facades/client/client/status.go
b/apiserver/facades/client/client/status.go
index 2ace0ef662..226634b321 100644
--- a/apiserver/facades/client/client/status.go
+++ b/apiserver/facades/client/client/status.go
@@ -1479,5 +1479,9 @@ func (p lxdStateCharmProfiler) LXDProfile()
lxdprofile.LXDProfile {
        if p.Charm == nil {
                return nil
        }
- return p.Charm.LXDProfile()
+ profile := p.Charm.LXDProfile()
+ if profile == nil {
+ return nil
+ }
+ return profile
 }

On Tue, May 14, 2019 at 5:10 PM Heather Lanigan <email address hidden>
wrote:

> @bidwell, I'm unable to reproduce so far with the charms I've chosen.
>
> What charms, including revision numbers, are deployed into the model
> exhibiting this panic?
>
> What cloud is the controller bootstrapped to?
>
> --
> You received this bug notification because you are subscribed to juju.
> Matching subscriptions: juju bugs
> https://bugs.launchpad.net/bugs/1829014
>
> Title:
> juju status command returns runtime error: invalid memory address or
> nil pointer dereference
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1829014/+subscriptions
>

[Joel Sing (jsing)]

I'd almost be certain it is a nil vs nil interface issue:

https://github.com/juju/juju/blob/2.6/core/lxdprofile/validate.go#L62

Most of this code checks against nil, however LXDProfiler.LXDProfile returns an LXDProfile which is an interface. Another option is to handle the nil checks in the actual functions that implement Empty and ValidateDeviceConfig - that way they're always safe to call.

[Heather Lanigan (hmlanigan)]

Steps to reproduce:

$ sudo snap refresh juju --channel 2.4/stable
$ /snap/bin/juju bootstrap localhost
$ /snap/bin/juju deploy ubuntu

wait for deploy

$ sudo snap refresh juju --channel 2.5/stable
$ /snap/bin/juju upgrade-juju -m controller
$ /snap/bin/juju upgrade-juju

$ sudo snap refresh juju --channel stable
$ /snap/bin/juju upgrade-juju -m controller
$ /snap/bin/juju status
ERROR runtime error: invalid memory address or nil pointer dereference

[Daniel Bidwell (bidwell)]

These controllers are all connected to a VMWare Vsphere "cloud" on
premises. These controllers are on Vsphere version 6.5.0 and 6.7.0.
I have been using it for a couple of years now. Someof the controllers
are running ubuntu 18.04 and some are running 16.04.

The charms under these controllers are are a variety ranging from
mysql, wordpress, to tomcat and java. Most of these charms are local
variations of the ones from the charmstore, modified for our specific
needs and settings.

I am able to successfully run "juju status intstance" on some of the
controllers that fail on running "juju stauts". Other "juju status
instance" vms get the same error as "juju status".

On Tue, 2019-05-14 at 14:58 +0000, Heather Lanigan wrote:
> @bidwell, I'm unable to reproduce so far with the charms I've
> chosen.
>
> What charms, including revision numbers, are deployed into the model
> exhibiting this panic?
>
> What cloud is the controller bootstrapped to?
>
--
Daniel R. Bidwell | <email address hidden>
Sr. Systems Architect | chief Information Security Officer
Andrews University | Information Technology Services
If two always agree, one of them is unnecessary.
Karma is getting what you deserve,
mercy is not getting what you deserve
grace is getting what you do not deserve.
In theory, theory and practice are the same.
In practice, they are not.

[Heather Lanigan (hmlanigan)]

https://github.com/juju/juju/pull/10189 for 2.6 branch

[Heather Lanigan (hmlanigan)]

PR 10189 also fixes a nil pointer panic with a similare trace in 'juju upgrade-charm'