Juju 2.7 bundles hates relative paths?

[Liam Young]

We just switched over to using juju 2.7 candidate and our charm testing broke. The problem is that when we test a charm we use a bundle which uses all supporting charms from the charm store and uses a local copy of the charm under test.

We reference the charm being tests in the bundle like this:

charm: ../../../nova-cell-controller

because the overlay with that in lives inside the charm here:

nova-cell-controller/tests/bundles/overlays

and the local paths seems to be relative to the overlay location. This worked in 2.6 but in 2.7 it breaks with:

ERROR cannot deploy bundle: the provided bundle has the following errors:
charm path in application "nova-cell-controller-cell2" does not exist: /nova-cell-controller

Does anyone know of a 2.7 change that might account for this ?

[Richard Harding (rharding)]

Yes, bundles were accidentally not using the same code that charms do that
were meant to be an explicit full path. 2.7 brought bundles in line in prep
for making sure we have room and lack of ambiguity for charm urls which was
why that was done with charms in the past. We had meant to make this a big
highlight as a "watch out for" in the 2.7 release docs but missed having
that there. We'll make sure to correct the docs around this.

On Fri, Nov 8, 2019 at 5:45 AM Liam Young <email address hidden> wrote:

> Public bug reported:
>
> We just switched over to using juju 2.7 candidate and our charm testing
> broke. The problem is that when we test a charm we use a bundle which
> uses all supporting charms from the charm store and uses a local copy
> of the charm under test.
>
> We reference the charm being tests in the bundle like this:
>
> charm: ../../../nova-cell-controller
>
> because the overlay with that in lives inside the charm here:
>
> nova-cell-controller/tests/bundles/overlays
>
> and the local paths seems to be relative to the overlay location. This
> worked in 2.6 but in 2.7 it breaks with:
>
> ERROR cannot deploy bundle: the provided bundle has the following errors:
> charm path in application "nova-cell-controller-cell2" does not exist:
> /nova-cell-controller
>
> Does anyone know of a 2.7 change that might account for this ?
>
> ** Affects: juju
> Importance: Undecided
> Status: New
>
> --
> You received this bug notification because you are subscribed to juju.
> https://bugs.launchpad.net/bugs/1851817
>
> Title:
> Juju 2.7 bundles hates relative paths?
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1851817/+subscriptions
>

[Richard Harding (rharding)]

Actually, I'm reading this again and wonder if that's right. I know we were more strict on the CLI side but having the relative paths in the bundle do you could shop a bundle with the charms needed together I believe is supposed to work.

I'll see if Achilleas can peek at this next week.

[Achilleas Anagnostopoulos (achilleasa)]

Does this mean that you have a base bundle somewhere with N charms and you use an overlay (from a different folder) to override one of them? In other words, is the "charm: ../../../nova-cell-controller" line in an overlay?

[Richard Harding (rharding)]

... had a whole reply and reread the bug and actually got more confused.
Let's see if we can get a sync with Liam next week.

On Tue, Nov 12, 2019 at 9:43 AM Achilleas Anagnostopoulos <
<email address hidden>> wrote:

> Does this mean that you have a base bundle somewhere with N charms and
> you use an overlay (from a different folder) to override one of them? In
> other words, is the "charm: ../../../nova-cell-controller" line in an
> overlay?
>
> --
> You received this bug notification because you are subscribed to juju.
> https://bugs.launchpad.net/bugs/1851817
>
> Title:
> Juju 2.7 bundles hates relative paths?
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1851817/+subscriptions
>

[Ramon Grullon (rgrullon)]

Hit this issue yesterday as the snap auto updated

juju deploy ./config/bundle.yaml --overlay config/overlays/bluestore.yaml --overlay config/overlays/ldap.yaml --overlay config/overlays/hostnames.yaml --overlay config/overlays/ssl.yaml --overlay config/overlays/sriov.yaml --dry-run

ERROR cannot deploy bundle: processing option "ldap-user" for application "int-ldap": resolving include "../secrets/ldap-user.txt": include file "../secrets/ldap-user.txt" not found

overlay file -

ldap-password: include-file://../secrets/ldap-password.txt

{{Workaround}}
juju revert juju

Trick workaround

fool the snap tool from auto updating. Simply run the following commands on ALL your LXD servers:

snap set system proxy.http="http://127.0.0.1:1111"
snap set system proxy.https="http://127.0.0.1:1111"

[Achilleas Anagnostopoulos (achilleasa)]

Please note that from juju 2.7, relative paths (charms, included files etc) should be resolved relative to the path of the entity (base bundle or overlay) that contains them. In previous juju versions, these includes would *always* be resolved relative to the base bundle or the CWD if the bundle was downloaded from the charm store. The latter behavior could cause issues if you weren't in the right directory.

We have updated our docs to reflect this change.

Ramon, can you please try to update your overlays so that the include paths are relative to their location and let me know if the above deploy command works for you then?

[Felipe Reyes (freyes)]

> Please note that from juju 2.7, relative paths (charms, included files etc) should be resolved relative to the path of the entity (base bundle or overlay) that contains them.

if I have a bundle that contains something like "include-file: ../../../../../../foo.txt" then this should fail:

charm pull cs:some-bundle
cd some-bundle
juju deploy ./bundle.yaml # <- this should fail

Is it possible to make juju fail with a informative message of the failure?, basically inform that the calculated absolute path from "../../../../../../foo.txt" was "/a/b/foo.txt" which it's outside the the directory where bundle.yaml is.

[Ian Johnson (ijoh)]

HI Achilleas , changing the include paths relative to their location has resolved the issue.

-----Original Message-----
From: <email address hidden> <email address hidden> On Behalf Of Achilleas Anagnostopoulos
Sent: 17 December 2019 11:39
To: Johnson,I,Ian,TNA6 R <email address hidden>
Subject: [Bug 1851817] Re: Juju 2.7 bundles hates relative paths?

Please note that from juju 2.7, relative paths (charms, included files
etc) should be resolved relative to the path of the entity (base bundle or overlay) that contains them. In previous juju versions, these includes would *always* be resolved relative to the base bundle or the CWD if the bundle was downloaded from the charm store. The latter behavior could cause issues if you weren't in the right directory.

We have updated our docs to reflect this change.

Ramon, can you please try to update your overlays so that the include paths are relative to their location and let me know if the above deploy command works for you then?

--
You received this bug notification because you are subscribed to the bug report.
https://bugs.launchpad.net/bugs/1851817

Title:
  Juju 2.7 bundles hates relative paths?

Status in juju:
  Triaged

Bug description:
  We just switched over to using juju 2.7 candidate and our charm
  testing broke. The problem is that when we test a charm we use a
  bundle which uses all supporting charms from the charm store and uses
  a local copy of the charm under test.

  We reference the charm being tests in the bundle like this:

  charm: ../../../nova-cell-controller

  because the overlay with that in lives inside the charm here:

  nova-cell-controller/tests/bundles/overlays

  and the local paths seems to be relative to the overlay location. This
  worked in 2.6 but in 2.7 it breaks with:

  ERROR cannot deploy bundle: the provided bundle has the following errors:
  charm path in application "nova-cell-controller-cell2" does not exist: /nova-cell-controller

  Does anyone know of a 2.7 change that might account for this ?

To manage notifications about this bug go to:
https://bugs.launchpad.net/juju/+bug/1851817/+subscriptions

[Richard Harding (rharding)]

Thanks, and we agree improved error messaging is a great thing to make sure
we add in. We'll make sure it's as clear as possible to assist.

On Tue, Dec 17, 2019 at 1:30 PM Ian Johnson <email address hidden>
wrote:

> HI Achilleas , changing the include paths relative to their location has
> resolved the issue.
>
> -----Original Message-----
> From: <email address hidden> <email address hidden> On Behalf Of
> Achilleas Anagnostopoulos
> Sent: 17 December 2019 11:39
> To: Johnson,I,Ian,TNA6 R <email address hidden>
> Subject: [Bug 1851817] Re: Juju 2.7 bundles hates relative paths?
>
> Please note that from juju 2.7, relative paths (charms, included files
> etc) should be resolved relative to the path of the entity (base bundle or
> overlay) that contains them. In previous juju versions, these includes
> would *always* be resolved relative to the base bundle or the CWD if the
> bundle was downloaded from the charm store. The latter behavior could cause
> issues if you weren't in the right directory.
>
> We have updated our docs to reflect this change.
>
> Ramon, can you please try to update your overlays so that the include
> paths are relative to their location and let me know if the above deploy
> command works for you then?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1851817
>
> Title:
> Juju 2.7 bundles hates relative paths?
>
> Status in juju:
> Triaged
>
> Bug description:
> We just switched over to using juju 2.7 candidate and our charm
> testing broke. The problem is that when we test a charm we use a
> bundle which uses all supporting charms from the charm store and uses
> a local copy of the charm under test.
>
> We reference the charm being tests in the bundle like this:
>
> charm: ../../../nova-cell-controller
>
> because the overlay with that in lives inside the charm here:
>
> nova-cell-controller/tests/bundles/overlays
>
> and the local paths seems to be relative to the overlay location. This
> worked in 2.6 but in 2.7 it breaks with:
>
> ERROR cannot deploy bundle: the provided bundle has the following errors:
> charm path in application "nova-cell-controller-cell2" does not exist:
> /nova-cell-controller
>
> Does anyone know of a 2.7 change that might account for this ?
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1851817/+subscriptions
>
> --
> You received this bug notification because you are subscribed to juju.
> https://bugs.launchpad.net/bugs/1851817
>
> Title:
> Juju 2.7 bundles hates relative paths?
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1851817/+subscriptions
>

[Achilleas Anagnostopoulos (achilleasa)]

PR https://github.com/juju/juju/pull/11080 updates the CLI error messages for missing includes so they are more informative (see QA steps in linked PR for example error output).

This change is included in the SHA for the upcoming juju 2.7.1 and has also been forward-ported to the develop branch.

Is there anything else we need to do on our end to mark this issue as closed?

[Ian Booth (wallyworld)]

Marking as Fix Released since it appears that we added better error messages to inform what the issue with the bundle paths was. We can reopen if needed.