panic when getting juju full status

[Chengen Du]

Version: Juju 2.9.33

A panic occurred when getting juju full status.

2022-11-17 04:38:13 CRITICAL juju.rpc server.go:557 panic running request {MethodCaller:0xc016b24390 transformErrors:0x2b376a0 hdr:{RequestId:2 Request:{Type:Client Version:5 Id: Action:FullStatus} Error: ErrorCode: ErrorInfo:map[] Version:1}} with arg {Patterns:[]}: runtime error: invalid memory address or nil pointer dereference
goroutine 93192940 [running]:
runtime/debug.Stack()
  /usr/local/go/src/runtime/debug/stack.go:24 +0x65
github.com/juju/juju/rpc.(*Conn).runRequest.func1()
  /home/jenkins/workspace/build-juju/build/src/github.com/juju/juju/rpc/server.go:558 +0x105
panic({0x4d8f4e0, 0x906ea40})
  /usr/local/go/src/runtime/panic.go:838 +0x207
github.com/juju/juju/state.(*Application).Charm(0x0)
  /home/jenkins/workspace/build-juju/build/src/github.com/juju/juju/state/application.go:902 +0x1d
github.com/juju/juju/apiserver/facades/client/client.(*statusContext).processUnit(_, _, {_, _}, _)
  /home/jenkins/workspace/build-juju/build/src/github.com/juju/juju/apiserver/facades/client/client/status.go:1558 +0x76b
github.com/juju/juju/apiserver/facades/client/client.(*statusContext).processUnits(0xc0117e8780?, 0xc029a662f0?, {0xc03b12fe90, 0x13}, 0x0?)
  /home/jenkins/workspace/build-juju/build/src/github.com/juju/juju/apiserver/facades/client/client/status.go:1474 +0xff
github.com/juju/juju/apiserver/facades/client/client.(*statusContext).processApplication(_, _)
  /home/jenkins/workspace/build-juju/build/src/github.com/juju/juju/apiserver/facades/client/client/status.go:1279 +0x719
github.com/juju/juju/apiserver/facades/client/client.(*statusContext).processApplications(0xc007696100)
  /home/jenkins/workspace/build-juju/build/src/github.com/juju/juju/apiserver/facades/client/client/status.go:1210 +0xd6
github.com/juju/juju/apiserver/facades/client/client.(*Client).FullStatus(_, {{_, _, _}})
  /home/jenkins/workspace/build-juju/build/src/github.com/juju/juju/apiserver/facades/client/client/status.go:442 +0x109e
reflect.Value.call({0x567e8c0?, 0xc041d2d300?, 0x40d867?}, {0x582d84d, 0x4}, {0xc05666d1e8, 0x1, 0x1621dbd?})
  /usr/local/go/src/reflect/value.go:556 +0x845
reflect.Value.Call({0x567e8c0?, 0xc041d2d300?, 0x4f1ad20?}, {0xc05666d1e8, 0x1, 0x1})
  /usr/local/go/src/reflect/value.go:339 +0xbf
github.com/juju/rpcreflect.newMethod.func8({0x61e2428, 0xc041d2d100}, {0x567e8c0?, 0xc041d2d300?, 0xc00928b580?}, {0x4f1ad20?, 0xc05666ced0?, 0xc041d2cf40?})
  /home/jenkins/workspace/build-juju/build/src/github.com/juju/juju/vendor/github.com/juju/rpcreflect/type.go:344 +0xce
github.com/juju/juju/apiserver.(*srvCaller).Call(0xc016b24390, {0x61e2428, 0xc041d2d100}, {0x0?, 0x7fb162a8af01?}, {0x4f1ad20?, 0xc05666ced0?, 0xc035e93140?})
  /home/jenkins/workspace/build-juju/build/src/github.com/juju/juju/apiserver/root.go:188 +0xa6
github.com/juju/juju/rpc.(*Conn).runRequest(0xc05d47e780, {{0x61cbab0, 0xc016b24390}, 0x5a77430, {0x2, {{0xc008f4bbf0, 0x6}, 0x5, {0x0, 0x0}, ...}, ...}}, ...)
  /home/jenkins/workspace/build-juju/build/src/github.com/juju/juju/rpc/server.go:571 +0x1b6
created by github.com/juju/juju/rpc.(*Conn).handleRequest
  /home/jenkins/workspace/build-juju/build/src/github.com/juju/juju/rpc/server.go:475 +0x651

The call trace has been discussed with juju team member.
it looks like the document in the db doesn't have a charm url and panic happened when trying to access it.

[Juan M. Tirado (tiradojm)]

2.9.33 is the Juju CLI version or the controller version? If they are not the same, do you get the same issue when issues the same CLI and controller versions?

[Chengen Du (chengendu)]

2.9.33 is the controller version.
We are still waiting for the customer to reply with the juju CLI version.

[Heather Lanigan (hmlanigan)]

A straightforward panic to resolve. Investigating how we have an application without a charmurl in the db.

[Heather Lanigan (hmlanigan)]

@chengendu, do you know of any issues with applications in the model where this panic occurred? I'm trying to understand how we got to this point in the database. The places I've seen in the code, all check for the CharmURL before adding an application, or refreshing it.

[Chengen Du (chengendu)]

This issue is related to #2002115.
I uploaded juju logs and system journal logs on that bug,
not sure if the panic related to that snapd refreshed juju-db.
Please refer to the logs for analysis and inform me if other logs are required.

[Heather Lanigan (hmlanigan)]

@chengendu,

Is the panic running juju status seen on the controller model or a different one?

Please provide a juju dumpdb of the model with the panic. (JUJU_DEV_FEATURE_FLAGS=developer-mode juju dump-db) You can place it on mombin or another private location.

[Chengen Du (chengendu)]

The panic happened on controller-2 (the controller model).
https://pastebin.canonical.com/p/3HJgFVTs9Z/

I have uploaded the juju dumpdb.
Please help to check it.
https://drive.google.com/file/d/1SlhJmBkVVkP_0_WK497HCnUZXRJHs_5I/view?usp=sharing

[Heather Lanigan (hmlanigan)]

PR to fix the panic:

https://github.com/juju/juju/pull/15063

Will open a new bug to track the root cause.

[Heather Lanigan (hmlanigan)]

The data in the dumpdb output looks correct. Is this panic still occurring?

If so, please provide the output of `juju_engine_report` run on the controller which shows the panic.

[Seyeong Kim (seyeongkim)]

@hmlanigan

AFAIK, this symptom happened once and After restarting unit agent, it is ok now.