err: getting OCI image resources ... empty id not valid ... nil pointer dereference

[Leon]

I'm trying to deploy some charms locally as part of integration tests and I'm getting the following errors in a loop that does not resolve:

INFO juju.worker.caasapplicationprovisioner.runner stopped "loki-k8s", err: getting OCI image resources: unable to fetch OCI image resources for loki-k8s: empty id not valid

INFO juju.worker.caasapplicationprovisioner.runner stopped "loki-k8s", err: getting OCI image resources: runtime error: invalid memory address or nil pointer dereference

Maybe the two are related: "empty id" -> "nil pointer dereference"

Removing and reinstalling microk8s and juju did not help.

Observed with juju 2.9.22 and 2.9.25.

[Ian Booth (wallyworld)]

How are you deploying the local charms? Can you provide the "juju deploy" command args you used?
Local charms with OCIs need to have those images provided using "--resource". Juju does check that but maybe something else is happening and the supplied image paths are incorrect?
I just tested with no issues with a local mariadb charm on microk8s on tip of juju 2.9

juju deploy /path/to/local/charm --resource mysql_image=mariadb

[Leon (sed-i)]

It happens every time I run integration tests (using pytest-operator).
This happens both locally and on github workflows.
I noticed that external OCI images (non-charmhub) are deployed successfully, but charmhub resources do not:

https://github.com/canonical/loki-k8s-operator/runs/5253135150?check_suite_focus=true
https://github.com/canonical/cos-configuration-k8s-operator/runs/5257034664?check_suite_focus=true

Another thing I noticed but not sure is related:

> microk8s.kubectl logs -n test-loki-push-api-zvxt modeloperator-6659fdfdd4-w8j7j
2022/02/22 16:02:35 http: TLS handshake error from 172.105.6.202:31366: EOF
2022/02/22 16:02:35 http: TLS handshake error from 172.105.6.202:6332: EOF
2022/02/22 16:02:37 http: TLS handshake error from 172.105.6.202:44071: EOF
2022/02/22 16:02:37 http: TLS handshake error from 172.105.6.202:61353: EOF
2022/02/22 16:02:37 http: TLS handshake error from 172.105.6.202:19469: EOF
2022/02/22 16:02:37 http: TLS handshake error from 172.105.6.202:65500: EOF
2022/02/22 16:02:37 http: TLS handshake error from 172.105.6.202:61588: EOF
2022/02/22 16:02:37 http: TLS handshake error from 172.105.6.202:60683: EOF
2022/02/22 16:02:41 http: TLS handshake error from 172.105.6.202:8206: EOF
2022/02/22 16:02:41 http: TLS handshake error from 172.105.6.202:30439: EOF
2022/02/22 16:02:41 http: TLS handshake error from 172.105.6.202:33866: EOF

[Ian Booth (wallyworld)]

We'll need more info to debug this. One possible cause is that the application entity is stored in the juju database without the charm origin id set. This would be a possible reason for the "empty id" message. Were there any other errors in the logs? Can we get a dump of the "applications" collection to see what's in there?

[Joseph Phillips (manadart)]

This is manifest in integration tests that are using pylibjuju as the driver.

The tests were depending on HEAD, where the bug was introduced by this patch:
https://github.com/juju/python-libjuju/pull/633

It works when using the latest (earlier) release from PyPi.

[Caner Derici (cderici)]

I found a small bug on libjuju that might be the cause of this. The patch is here: https://github.com/juju/python-libjuju/pull/647

But I can't reproduce this problem, not sure why, (all the tests seem to be passing for me -- tried old&new libjuju versions, and tried older HEADs of loki-k8s-operator as well), so I'm unable to QA.

@Joe if you can reproduce it then it would be great if you can either let me know how or QA this with that little patch real quick yourself, that'd be awesome, thanks!

[Caner Derici (cderici)]

Just to complete the discussion here, while working on the little patch from my last comment I was able to reproduce the problem and found the bug introduced in #633 and fixed it. The details can be found in https://github.com/juju/python-libjuju/pull/647