[cosmic] juju panic: charm or bundle URL has invalid form: "../../../foo"

[Ryan Beisner]

In adding Nova-cell-controller Cosmic functional tests, I'm hitting a condition which appears to be an issue in Juju core. The same test passes on series: Bionic. Identical everything, other than series afaict. Raising against the charm first, for peer validation.

panic: charm or bundle URL has invalid form: "../../../nova-cell-controller"

To re-iterate: this works fine with Bionic series.

http://paste.ubuntu.com/p/34KfWpWsXj/

http://paste.ubuntu.com/p/nqVPgkffM9/

func create: /home/ubuntu/git/temp/nova-cell-controller/build/builds/nova-cell-controller/.tox/func
func installdeps: -r/home/ubuntu/git/temp/nova-cell-controller/build/builds/nova-cell-controller/test-requirements.txt
func installed: appdirs==1.4.3,asn1crypto==0.24.0,async-generator==1.10,attrs==18.2.0,Babel==2.6.0,bcrypt==3.1.4,certifi==2018.10.15,cffi==1.11.5,chardet==3.0.4,cliff==2.14.0,cmd2==0.9.6,colorama==0.4.0,cryptography==2.4.1,debtcollector==1.20.0,decorator==4.3.0,dogpile.cache==0.6.7,hvac==0.6.4,idna==2.7,iso8601==0.1.12,Jinja2==2.10,jmespath==0.9.3,jsonpatch==1.23,jsonpointer==2.0,jsonschema==2.6.0,juju==0.10.2,juju-wait==2.6.4,jujubundlelib==0.5.6,keystoneauth1==3.11.1,macaroonbakery==1.2.1,MarkupSafe==1.1.0,monotonic==1.5,msgpack==0.5.6,munch==2.3.2,netaddr==0.7.19,netifaces==0.10.7,openstacksdk==0.19.0,os-client-config==1.31.2,os-service-types==1.3.0,osc-lib==1.11.1,oslo.config==6.7.0,oslo.context==2.21.0,oslo.i18n==3.22.1,oslo.log==3.40.1,oslo.serialization==2.28.1,oslo.utils==3.37.1,paramiko==2.4.2,pbr==5.1.1,pkg-resources==0.0.0,prettytable==0.7.2,protobuf==3.6.1,pyasn1==0.4.4,pycparser==2.19,pyinotify==0.9.6,pymacaroons==0.13.0,PyNaCl==1.3.0,pyOpenSSL==18.0.0,pyparsing==2.3.0,pyperclip==1.7.0,pyRFC3339==1.1,python-cinderclient==4.1.0,python-dateutil==2.7.5,python-glanceclient==2.14.0,python-keystoneclient==3.18.0,python-neutronclient==6.11.0,python-novaclient==11.1.0,pytz==2018.7,PyYAML==3.13,requests==2.20.1,requestsexceptions==1.4.0,rfc3986==1.1.0,simplejson==3.16.0,six==1.11.0,stevedore==1.30.0,tenacity==5.0.2,theblues==0.5.0,urllib3==1.24.1,warlock==1.3.0,wcwidth==0.1.7,websockets==6.0,wrapt==1.10.11,zaza==0.0.2.dev1
func runtests: PYTHONHASHSEED='0'
func runtests: commands[0] | functest-run-suite --keep-model
Controller: beisner-serverstack

Model Cloud/Region Status Machines Cores Access Last connection
beisner serverstack/serverstack available 0 - admin 32 minutes ago
controller serverstack/serverstack available 1 4 admin just now
default serverstack/serverstack available 0 - admin 2018-11-12
zaza-71c8b0ef9d0e serverstack/serverstack available 0 - admin just now
zaza-ad21fe0cd6f9 serverstack/serverstack available 0 - admin 4 minutes ago
zaza-cdf537ce277c* serverstack/serverstack available 0 - admin 3 seconds ago

INFO:root:Deploying bundle ./tests/bundles/cosmic-rocky.yaml on to zaza-71c8b0ef9d0e model
Resolving charm: cs:~openstack-charmers-next/glance
Resolving charm: cs:~openstack-charmers-next/keystone
Resolving charm: cs:~openstack-charmers-next/percona-cluster
Resolving charm: cs:~openstack-charmers-next/percona-cluster
Resolving charm: cs:~openstack-charmers-next/neutron-api
Resolving charm: cs:~openstack-charmers-next/neutron-gateway
Resolving charm: cs:~openstack-charmers-next/neutron-openvswitch
Resolving charm: cs:~openstack-charmers-next/nova-cloud-controller
Resolving charm: cs:~openstack-charmers-next/nova-compute
Resolving charm: cs:~openstack-charmers-next/rabbitmq-server
Resolving charm: cs:~openstack-charmers-next/rabbitmq-server
Resolving charm: cs:~openstack-charmers-next/rabbitmq-server
panic: charm or bundle URL has invalid form: "../../../nova-cell-controller"

goroutine 1 [running]:
github.com/juju/juju/vendor/gopkg.in/juju/charm%2ev6.MustParseURL(0xc4207d3120, 0x1d, 0xc4201ee260)
 /build/juju/parts/juju/go/src/github.com/juju/juju/vendor/gopkg.in/juju/charm.v6/url.go:99 +0x6f
github.com/juju/juju/vendor/github.com/juju/bundlechanges.getSeries(0xc420489760, 0xc4207c42b0, 0x6, 0x1a, 0x48378a0)
 /build/juju/parts/juju/go/src/github.com/juju/juju/vendor/github.com/juju/bundlechanges/handlers.go:985 +0xb8
github.com/juju/juju/vendor/github.com/juju/bundlechanges.(*resolver).handleApplications(0xc4209a3750, 0xc420ba8f20)
 /build/juju/parts/juju/go/src/github.com/juju/juju/vendor/github.com/juju/bundlechanges/handlers.go:49 +0x982
github.com/juju/juju/vendor/github.com/juju/bundlechanges.FromData(0xc4207ce000, 0xc4204e3ec0, 0x306bd60, 0xc4205f8c00, 0x0, 0x0, 0x2650440, 0xc420bac710, 0xc42001d530, 0xc4200d76c0, ...)
 /build/juju/parts/juju/go/src/github.com/juju/juju/vendor/github.com/juju/bundlechanges/changes.go:61 +0x130
github.com/juju/juju/cmd/juju/application.(*bundleHandler).getChanges(0xc420422b60, 0x0, 0x0)
 /build/juju/parts/juju/go/src/github.com/juju/juju/cmd/juju/application/bundle.go:355 +0xb7
github.com/juju/juju/cmd/juju/application.deployBundle(0xc420058540, 0x5a, 0xc4207ce000, 0x0, 0xc4209bf180, 0x1, 0x1, 0x0, 0x0, 0x30ee4c0, ...)
 /build/juju/parts/juju/go/src/github.com/juju/juju/cmd/juju/application/bundle.go:138 +0x1de
github.com/juju/juju/cmd/juju/application.(*DeployCommand).deployBundle(0xc4202c4c80, 0xc420613130, 0xc420058540, 0x5a, 0xc4207ce000, 0x0, 0x0, 0x0, 0x30ee4c0, 0xc42002fbd0, ...)
 /build/juju/parts/juju/go/src/github.com/juju/juju/cmd/juju/application/deploy.go:905 +0x707
github.com/juju/juju/cmd/juju/application.(*DeployCommand).maybeReadLocalBundle.func1(0xc420613130, 0x30ee4c0, 0xc42002fbd0, 0xc42001da70, 0x0)
 /build/juju/parts/juju/go/src/github.com/juju/juju/cmd/juju/application/deploy.go:1352 +0xa9
github.com/juju/juju/cmd/juju/application.(*DeployCommand).Run(0xc4202c4c80, 0xc420613130, 0x0, 0x0)
 /build/juju/parts/juju/go/src/github.com/juju/juju/cmd/juju/application/deploy.go:1182 +0x491
github.com/juju/juju/cmd/modelcmd.(*modelCommandWrapper).Run(0xc42085d530, 0xc420613130, 0xc42085d530, 0xc42002a800)
 /build/juju/parts/juju/go/src/github.com/juju/juju/cmd/modelcmd/modelcommand.go:505 +0x128
github.com/juju/juju/cmd/modelcmd.(*baseCommandWrapper).Run(0xc4208da690, 0xc420613130, 0x0, 0x0)
 /build/juju/parts/juju/go/src/github.com/juju/juju/cmd/modelcmd/base.go:407 +0xab
github.com/juju/juju/vendor/github.com/juju/cmd.(*SuperCommand).Run(0xc4208426c0, 0xc420613130, 0xc420613130, 0x0)
 /build/juju/parts/juju/go/src/github.com/juju/juju/vendor/github.com/juju/cmd/supercommand.go:456 +0x2c0
github.com/juju/juju/vendor/github.com/juju/cmd.Main(0x309e3c0, 0xc4208426c0, 0xc420613130, 0xc42004c080, 0x6, 0x6, 0x2980980)
 /build/juju/parts/juju/go/src/github.com/juju/juju/vendor/github.com/juju/cmd/cmd.go:317 +0x266
github.com/juju/juju/cmd/juju/commands.main.Run(0x2e69168, 0xc42004c070, 0x7, 0x7, 0x0)
 /build/juju/parts/juju/go/src/github.com/juju/juju/cmd/juju/commands/main.go:179 +0x1af
github.com/juju/juju/cmd/juju/commands.Main(0xc42004c070, 0x7, 0x7, 0xc42000c160)
 /build/juju/parts/juju/go/src/github.com/juju/juju/cmd/juju/commands/main.go:125 +0x4b
main.main()
 /build/juju/parts/juju/go/src/github.com/juju/juju/cmd/juju/main.go:38 +0x8c
Traceback (most recent call last):
  File "/home/ubuntu/git/temp/nova-cell-controller/build/builds/nova-cell-controller/.tox/func/bin/functest-run-suite", line 11, in <module>
    sys.exit(main())
  File "/home/ubuntu/git/temp/nova-cell-controller/build/builds/nova-cell-controller/.tox/func/lib/python3.6/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 129, in main
    bundle=args.bundle)
  File "/home/ubuntu/git/temp/nova-cell-controller/build/builds/nova-cell-controller/.tox/func/lib/python3.6/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 59, in func_test_runner
    model_name)
  File "/home/ubuntu/git/temp/nova-cell-controller/build/builds/nova-cell-controller/.tox/func/lib/python3.6/site-packages/zaza/charm_lifecycle/deploy.py", line 253, in deploy
    deploy_bundle(bundle, model)
  File "/home/ubuntu/git/temp/nova-cell-controller/build/builds/nova-cell-controller/.tox/func/lib/python3.6/site-packages/zaza/charm_lifecycle/deploy.py", line 240, in deploy_bundle
    subprocess.check_call(cmd)
  File "/usr/lib/python3.6/subprocess.py", line 291, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['juju', 'deploy', '-m', 'zaza-71c8b0ef9d0e', './tests/bundles/cosmic-rocky.yaml', '--overlay', '/tmp/tmpx09c2nia/local-charm-overlay.yaml']' returned non-zero exit status 2.
ERROR: InvocationError: '/home/ubuntu/git/temp/nova-cell-controller/build/builds/nova-cell-controller/.tox/func/bin/functest-run-suite --keep-model'
___________________________________ summary ____________________________________
ERROR: func: commands failed

[Ryan Beisner (1chb1n)]

Confirmed by another team member, concur this is a Juju core 'series: cosmic' issue.

[Ryan Beisner (1chb1n)]

IMPACT: OpenStack CI cannot enable series: cosmic at this time.

[Ryan Beisner (1chb1n)]

Bundle:

https://git.openstack.org/cgit/openstack/charm-nova-cell-controller/tree/src/tests/bundles/cosmic-rocky.yaml?id=36189ac193af399d77578fca13dfcbaf30098e2e

Overlay:

https://git.openstack.org/cgit/openstack/charm-nova-cell-controller/tree/src/tests/bundles/overlays/local-charm-overlay.yaml.j2?id=36189ac193af399d77578fca13dfcbaf30098e2e

FYI The overlay gets rendered to a .yaml in that dir, before juju deploy is called. In this case, there is nothing to render, but it is just part of the test prep automation.

[Ryan Beisner (1chb1n)]

FWIW, the same procedure with series: bionic:

http://paste.ubuntu.com/p/xbGgKgV4gs/

[Richard Harding (rharding)]

Sorry but more info then. Can we verify that the charm in ../../../nova-cell-controller declares cosmic and bionic as valid series and to verify this works if you just change the top of the bundle from cosmic to bionic?

[Ryan Beisner (1chb1n)]

Indeed, confirmed. Metadata contains series: cosmic.

ubuntu@beisner-bastion:~/git/temp/nova-cell-controller/build/builds/nova-cell-controller$ cat metadata.yaml
"name": "nova-cell-controller"
"summary": "OpenStack Compute - Nova cloud controller for a cell."
"maintainer": "OpenStack Charmers <email address hidden>"
"description": |
  OpenStack is a reliable cloud infrastructure. Its mission is to produce
  the ubiquitous cloud computing platform that will meet the needs of public
  and private cloud providers regardless of size, by being simple to implement
  and massively scalable.
  .
  OpenStack Compute, codenamed Nova, is a cloud computing fabric controller. In
  addition to its "native" API (the OpenStack API), it also supports the Amazon
  EC2 API.
  .
  This charm provides the cloud controller service for the OpenStack Nova cell
  and includes the nova-conductor service.
  .
  OpenStack Rocky or later is required for the Nova cells feature.
  .
  See usage guidance as published in the charm deployment guide:
    * https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/latest/app-nova-cells.html
"tags":
- "openstack"
- "misc"
"series":
- "bionic"
- "cosmic"
"requires":
  "certificates":
    "interface": "tls-certificates"
  "shared-db":
    "interface": "mysql-shared"
  "amqp":
    "interface": "rabbitmq"
  "identity-service":
    "interface": "keystone"
  "ha":
    "interface": "hacluster"
    "scope": "container"
  "cloud-compute":
    "interface": "nova-compute"
  "nova-cell-compute":
    "interface": "nova-cell"
"provides":
  "nrpe-external-master":
    "interface": "nrpe-external-master"
    "scope": "container"
  "cloud-controller":
    "interface": "nova"
"peers":
  "cluster":
    "interface": "nova-ha"
"extra-bindings":
  "public": !!null ""
  "admin": !!null ""
  "internal": !!null ""
"subordinate": !!bool "false"

[Joseph Phillips (manadart)]

Reproduced. Will address as soon as I am able.

[Joseph Phillips (manadart)]

The pertinent difference between these bundles is that nova-cell-controller-cell2 specifies a series here:

https://git.openstack.org/cgit/openstack/charm-nova-cell-controller/tree/src/tests/bundles/bionic-rocky.yaml?id=36189ac193af399d77578fca13dfcbaf30098e2e#n117

If I add "series: cosmic" here:

https://git.openstack.org/cgit/openstack/charm-nova-cell-controller/tree/src/tests/bundles/cosmic-rocky.yaml?id=36189ac193af399d77578fca13dfcbaf30098e2e#n110

Then the panic does not occur. However https://jujucharms.com/ntp/24 does not support cosmic, so deploy still fails (ntp #31 supports it).

How this is working is that declaring the series in the bundle returns early instead of falling through to interrogating the charm. The relative path in the overlay is what appears to be messing things up here. When I use an absolute path in the overlay, it works.

[Joseph Phillips (manadart)]

I have marked this as incomplete, as I am peeling off to some other tickets.

If we still think there is something that should be better handled by Juju, I'm ready to take it back up.

[Launchpad Janitor (janitor)]

[Expired for juju because there has been no activity for 60 days.]

[Tim McNamara (tim-clicks)]

Resurrecting this ticket. Because its severity, I think that it should be actively closed rather than cleaned up by the janitor.

[Anastasia (anastasia-macmood)]

@Tim McNamara (tim-clicks),

The report was marked as Incomplete as there was not enough information around what needs to be done on Juju side.

Re-reading the comments, we still do not know what to do on Juju side and according to comment # 8, there is a clear reason why the failure happened with a correct bundle specification that does not produce a failure. In other words, the initial report could have been caused by a user error, i.e. nothing to fix on Juju side.

I am marking this back as Incomplete, allowing for another 60 days of feedback. If we receive nothing further, it will expire again and should not be resurrected. (severity is not always the determining factor for whether the ticket is valid to work on).

[Chris MacNaughton (chris.macnaughton)]

As a suggestion, when a bundle declares a series and a local charm does not, the deploy will fail with this error. If you bundle does not declare a series, I can understand requiring a series but, for a multiseries local charm, it seems reasonable that Juju could use the bundle's series rather than throwing this error.

[Anastasia (anastasia-macmood)]

Sounds good, I'll flick back to Triaged and it will be addressed as part of the queue.