inconsistent state in EnterScope

[Judit Novak]

I'm facing a Juju3 issue when aiming to introduce Juju3 testing on data-platform-libs [1].
One particular test fails [2] (postgres i.e. integration-database) with the following error:

unit-application-0: 17:27:56 ERROR juju.worker.uniter resolver loop error: cannot enter scope for unit "application/0" in relation "application:aliased-multiple-database-clusters another-database:database": inconsistent state in EnterScope
unit-application-0: 17:27:56 INFO juju.worker.uniter unit "application/0" shutting down: cannot enter scope for unit "application/0" in relation "application:aliased-multiple-database-clusters another-database:database": inconsistent state in EnterScope

In the meanwhile indeed showing irregularities in the corresponding relation (relation-5):

  - relation-id: 4
    endpoint: aliased-multiple-database-clusters
    related-endpoint: database
    application-data:
      data: '{"database": "application_aliased_multiple_database_clusters", "extra-user-roles":
        "SUPERUSER"}'
      endpoints: 10.1.190.145:5432
      password: Wtr7E9dni1qADMUI
      tls: "False"
      username: juju_application_aliased_multiple_database_clusters
      version: PostgreSQL 14.3 (Ubuntu 14.3-0ubuntu0.22.04.1) on x86_64-pc-linux-gnu,
        compiled by gcc (Ubuntu 11.2.0-19ubuntu1) 11.2.0, 64-bit
    related-units:
      database/0:
        in-scope: true
[..]
  - relation-id: 5
    endpoint: aliased-multiple-database-clusters
    related-endpoint: database
    application-data:
      data: '{}'
    related-units:
      another-database/0:
        in-scope: true
        data:
          egress-subnets: 10.152.183.208/32
          ingress-address: 10.152.183.208
          private-address: 10.152.183.208
  provider-id: application-0
  address: 10.1.190.156

See attached screenshot as well.

The issue seem to appear on different units throughout the test run. Locally sometimes even all tests work. I couldn't spot a pattern yet.

The issue is the same also referred by Dimitry [3].

Thanks very much for your help,
Judit

[1]: https://github.com/canonical/data-platform-libs/pull/84
[2]: https://github.com/canonical/data-platform-libs/actions/runs/5834178511/job/15823259181
[3]: https://chat.charmhub.io/charmhub/pl/orjooyti9pbzpbatgrwz9knm3a

[Alex Lutay (taurus)]

I am facing this issue constantly with mysql-k8s+mysql-test-app charm.
Steps to reproduce:

juju deploy mysql-k8s --trust --channel 8.0/beta -n 3
juju wait-for application mysql-k8s
juju deploy mysql-test-app --channel edge
juju relate mysql-k8s mysql-test-app:database
sleep 120 # wait some time
juju remove-relation mysql-k8s mysql-test-app
sleep 60 # wait some time
juju relate mysql-k8s mysql-test-app:database

Agent went into the failed at least on some unit (could be many):
> Unit Workload Agent Address Ports Message
> mysql-k8s/2 active failed 10.1.84.121
> mysql-test-app/0 waiting failed 10.1.84.98

Debug-log:
unit-mysql-test-app-0: 12:57:30 ERROR juju.worker.uniter resolver loop error: cannot enter scope for unit "mysql-test-app/0" in relation "mysql-test-app:database mysql-k8s:database": inconsistent state in EnterScope

Juju 3.1.6-9a5f4ee from 3.1/edge (rev 24354).

[Nicolas Vinuesa (nvinuesa)]

PR (merged): https://github.com/juju/juju/pull/16257