[2.9.16 & 2.9.17] juju trust gets lost if juju config is run on application

[Alexander Balderson]

We are seeing instances of juju trust calls getting lost when called right after deploying a bundle.

The bundle deploy happens as normal, juju trust is called, but then the application (openstack-integrator in this case) gets blocked further down in the deployment requesting that "juju trust" gets run. I can confirm in the exported bundle that juju trust was not set, as the application doesnt show "trusted=true" in the config.

If it matters, we're doing this programmatically, so there is almost no delay between juju deploy finishing, and juju trust getting called.

Additionally, we have only started seeing this error in the last 24 hours.

I'm working on collecting logs from a 2.9.17 run that is exhibiting this problem, and will add them as soon as they are available.

from the crashdump, you should be able to see the current status of the openstack-integrator is waiting for "juju trust" but juju status doesn't show if things are trusted, and I don't have the exported bundle, unfortunately.

In the foundation log, if you look for `juju trust` you can see trust was called just before juju wait was called, and after the deploy has finished

crashdump:
https://oil-jenkins.canonical.com/artifacts/ade75552-3afd-4eb3-8e7f-6b1474265bf2/generated/generated/kubernetes/juju-crashdump-kubernetes-2021-10-22-18.34.43.tar.gz

foundation.log:
https://oil-jenkins.canonical.com/artifacts/ade75552-3afd-4eb3-8e7f-6b1474265bf2/generated/generated/foundation.log

Controller logs:
https://oil-jenkins.canonical.com/artifacts/ade75552-3afd-4eb3-8e7f-6b1474265bf2/generated/generated/juju_openstack_controller/juju-crashdump-controller-2021-10-22-18.43.51.tar.gz

full test run:
https://solutions.qa.canonical.com/testruns/testRun/ade75552-3afd-4eb3-8e7f-6b1474265bf2

[Alexander Balderson (asbalderson)]

It's probably worth mentioning:

I havent been able to reproduce this on aws, where we run `juju trust` in the same way

[John A Meinel (jameinel)]

If there is an application that needs to be trusted, you can deploy it with trust with "juju deploy --trust app".
If you are deploying a bundle, then the bundle can label individual items with "trust: true", and only applications the bundle has highlighted to be trusted will be trusted by specifying --trust.

You can find out whether a given application is trusted with 'juju config app':
```
application: dummy-source
application-config:
  trust:
    default: false
    description: Does this application have access to trusted credentials
    source: default
    type: bool
    value: false
```

[Marian Gasparovic (marosg)]

We are aware of those options. What we report here is we saw juju trust call to be "lost". It is something we have been using for a very long time without any issue, it started to happen last week and not consistently.

[Alexander Balderson (asbalderson)]

I updated the description since i have a simple reproducer now
We have found that if you run juju config on an application after deploy, juju trust is lost

a simple reproducer i ran on my local machine:

$ juju deploy cs:jenkins
$ juju trust jenkins
$ juju config jenkins trust
true
$ juju config jenkins password=badbadbad
$ juju config jenkins trust
false

so after changing config trust is lost, but i think it should be preserved

[John A Meinel (jameinel)]

that, indeed, sounds very disconcerting. setting a config value should not
impact any other config values

On Tue, Oct 26, 2021 at 9:25 AM Alexander Balderson <
<email address hidden>> wrote:

> I updated the description since i have a simple reproducer now
> We have found that if you run juju config on an application after deploy,
> juju trust is lost
>
> a simple reproducer i ran on my local machine:
>
> $ juju deploy cs:jenkins
> $ juju config jenkins trust
> true
> $ juju config jenkins password=badbadbad
> $ juju config jenkins trust
> false
>
> so after changing config trust is lost, but i think it should be
> preserved
>
> ** Summary changed:
>
> - [2.9.16 & 2.9.17] juju trust call gets lost if called right after deploy
> + [2.9.16 & 2.9.17] juju trust gets lost if juju config is run on
> application
>
> --
> You received this bug notification because you are subscribed to juju.
> Matching subscriptions: juju bugs
> https://bugs.launchpad.net/bugs/1948496
>
> Title:
> [2.9.16 & 2.9.17] juju trust gets lost if juju config is run on
> application
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1948496/+subscriptions
>
>

[John A Meinel (jameinel)]

Confirmed here with a trivial charm:
$ juju deploy ./dummy-sink
$ juju trust dummysink
$ juju config dummy-sink
application: dummy-sink
application-config:
  trust:
    default: false
    description: Does this application have access to trusted credentials
    source: user
    type: bool
    value: true
charm: dummy-sink
settings:
  token:
    default: ""
    description: Token used to prove that the relation is working
    source: default
    type: string
    value: ""

$ juju config dummy-sink token=a
$ juju config dummy-sink
application: dummy-sink
application-config:
  trust:
    default: false
    description: Does this application have access to trusted credentials
    source: default
    type: bool
    value: false
charm: dummy-sink
settings:
  token:
    default: ""
    description: Token used to prove that the relation is working
    source: user
    type: string
    value: a

On Tue, Oct 26, 2021 at 12:45 PM John Meinel <email address hidden> wrote:

> that, indeed, sounds very disconcerting. setting a config value should not
> impact any other config values
>
>
> On Tue, Oct 26, 2021 at 9:25 AM Alexander Balderson <
> <email address hidden>> wrote:
>
>> I updated the description since i have a simple reproducer now
>> We have found that if you run juju config on an application after deploy,
>> juju trust is lost
>>
>> a simple reproducer i ran on my local machine:
>>
>> $ juju deploy cs:jenkins
>> $ juju config jenkins trust
>> true
>> $ juju config jenkins password=badbadbad
>> $ juju config jenkins trust
>> false
>>
>> so after changing config trust is lost, but i think it should be
>> preserved
>>
>> ** Summary changed:
>>
>> - [2.9.16 & 2.9.17] juju trust call gets lost if called right after deploy
>> + [2.9.16 & 2.9.17] juju trust gets lost if juju config is run on
>> application
>>
>> --
>> You received this bug notification because you are subscribed to juju.
>> Matching subscriptions: juju bugs
>> https://bugs.launchpad.net/bugs/1948496
>>
>> Title:
>> [2.9.16 & 2.9.17] juju trust gets lost if juju config is run on
>> application
>>
>> To manage notifications about this bug go to:
>> https://bugs.launchpad.net/juju/+bug/1948496/+subscriptions
>>
>>

[John A Meinel (jameinel)]

Definitely a regression

[John A Meinel (jameinel)]

Downgrading from Critical because the behavior has existed for a long time and isn't just a regression in 2.9.17.

[John A Meinel (jameinel)]

My reproducer for this behavior does not reproduce with Juju 2.8. However, it did reproduce after upgrading to 2.9.5 so likely this was broken since 2.9.0.

[Harry Pidcock (hpidcock)]

https://github.com/juju/juju/pull/13488