bootstrap fails in airgapped vshpere environment

Bug #2056550 reported by Bayani Carbone
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Canonical Juju
New
Undecided
Unassigned

Bug Description

Juju client version 3.3.3 (also tested 3.1.7)
Juju agent version 3.3.3 (also tested 3.1.5)

I'm trying to bootstrap a juju controller on an airgapped vmware environment.

I pass the following to the bootstrap command via --config:

container-image-metadata-url: "https://repo.example.com/simple-streams-lxd-ova/"
container-image-stream: released

image-metadata-url: "https://repo.example.com/simple-streams-lxd-ova/"
image-stream: released

agent-metadata-url: "https://repo.example.com/simple-streams-juju/"
agent-stream: released

Here are the client debug logs:
terraform_data.juju_bootstrap (local-exec): 10:38:51 INFO cmd bootstrap.go:409 Loading image metadata
terraform_data.juju_bootstrap (local-exec): 10:38:51 DEBUG juju.environs imagemetadata.go:124 obtained image datasource "image-metadata-url"
terraform_data.juju_bootstrap (local-exec): 10:38:51 DEBUG juju.environs imagemetadata.go:124 obtained image datasource "default ubuntu cloud images"
terraform_data.juju_bootstrap (local-exec): 10:38:51 DEBUG juju.environs.bootstrap bootstrap.go:984 constraints for image metadata lookup &{{{TestRegion 10.246.152.100} [12.04 12.10 13.04 13.10 14.04 14.10 15.04 15.10 16.04 16.10 17.0
4 17.10 18.04 18.10 19.04 19.10 20.04 20.10 21.04 21.10 22.04 22.10 23.04 23.10 24.04 LTS 7 9 genericlinux kubernetes] [amd64 arm64 ppc64el s390x riscv64] released}}
terraform_data.juju_bootstrap (local-exec): 10:38:51 DEBUG juju.environs.simplestreams simplestreams.go:417 searching for signed metadata in datasource "image-metadata-url"
terraform_data.juju_bootstrap (local-exec): 10:38:51 DEBUG juju.environs.simplestreams simplestreams.go:451 looking for data index using path streams/v1/index2.sjson
terraform_data.juju_bootstrap (local-exec): 10:38:51 DEBUG juju.environs.simplestreams simplestreams.go:463 looking for data index using URL https://repo.example.com/simple-streams-lxd-ova/streams/v1/index2.sjson
terraform_data.juju_bootstrap (local-exec): 10:38:51 DEBUG juju.environs.simplestreams simplestreams.go:466 streams/v1/index2.sjson not accessed, actual error: [{github.com/juju/retry.Call:188: } {github.com/juju/juju/environs/sim
plestreams.(*urlDataSource).fetch:219: "https://repo.example.com/simple-streams-lxd-ova/streams/v1/index2.sjson" not found}]
terraform_data.juju_bootstrap (local-exec): 10:38:51 DEBUG juju.environs.simplestreams simplestreams.go:467 streams/v1/index2.sjson not accessed, trying legacy index path: streams/v1/index.sjson
terraform_data.juju_bootstrap (local-exec): 10:38:51 DEBUG juju.environs.simplestreams simplestreams.go:482 cannot load index "https://repo.example.com/simple-streams-lxd-ova/streams/v1/index.sjson": "https://repo.example.com/simple-streams-lxd-ova/streams/v1/index.sjson" not found
terraform_data.juju_bootstrap (local-exec): 10:38:51 DEBUG juju.environs.simplestreams simplestreams.go:421 falling back to search for unsigned metadata in datasource "image-metadata-url"
terraform_data.juju_bootstrap (local-exec): 10:38:51 DEBUG juju.environs.simplestreams simplestreams.go:451 looking for data index using path streams/v1/index2.json
terraform_data.juju_bootstrap (local-exec): 10:38:51 DEBUG juju.environs.simplestreams simplestreams.go:463 looking for data index using URL https://repo.example.com/simple-streams-lxd-ova/streams/v1/index2.json
terraform_data.juju_bootstrap (local-exec): 10:38:51 DEBUG juju.environs.simplestreams simplestreams.go:466 streams/v1/index2.json not accessed, actual error: [{github.com/juju/retry.Call:188: } {github.com/juju/juju/environs/simp
lestreams.(*urlDataSource).fetch:219: "https://repo.example.com/simple-streams-lxd-ova/streams/v1/index2.json" not found}]
terraform_data.juju_bootstrap (local-exec): 10:38:51 DEBUG juju.environs.simplestreams simplestreams.go:467 streams/v1/index2.json not accessed, trying legacy index path: streams/v1/index.json
terraform_data.juju_bootstrap (local-exec): 10:38:51 DEBUG juju.environs.simplestreams simplestreams.go:486 read metadata index at "https://repo.example.com/simple-streams-lxd-ova/streams/v1/index.json"
terraform_data.juju_bootstrap (local-exec): 10:38:51 DEBUG juju.environs.simplestreams simplestreams.go:490 skipping index "https://repo.example.com/simple-streams-lxd-ova/streams/v1/index.json" because of missing information:
index file has no data for cloud {TestRegion 10.246.152.100} not found
terraform_data.juju_bootstrap (local-exec): 10:38:51 DEBUG juju.environs.bootstrap bootstrap.go:992 ignoring image metadata in image-metadata-url: index file has no data for cloud {TestRegion 10.246.152.100} not found
terraform_data.juju_bootstrap (local-exec): 10:38:51 DEBUG juju.environs.simplestreams simplestreams.go:417 searching for signed metadata in datasource "default ubuntu cloud images"
terraform_data.juju_bootstrap (local-exec): 10:38:51 DEBUG juju.environs.simplestreams simplestreams.go:451 looking for data index using path streams/v1/index2.sjson
terraform_data.juju_bootstrap (local-exec): 10:38:51 WARN juju.environs.simplestreams datasource.go:212 Got error requesting "http://cloud-images.ubuntu.com/releases/streams/v1/index2.sjson": Get "http://cloud-images.ubuntu.com/releases/streams/v1/index2.sjson": dial tcp [2620:2d:4000:1::1a]:80: connect: network is unreachable
terraform_data.juju_bootstrap (local-exec): 10:38:52 WARN juju.environs.simplestreams datasource.go:212 Got error requesting "http://cloud-images.ubuntu.com/releases/streams/v1/index2.sjson": Get "http://cloud-images.ubuntu.com/releases/streams/v1/index2.sjson": dial tcp [2620:2d:4000:1::17]:80: connect: network is unreachable
terraform_data.juju_bootstrap (local-exec): 10:38:54 WARN juju.environs.simplestreams datasource.go:212 Got error requesting "http://cloud-images.ubuntu.com/releases/streams/v1/index2.sjson": Get "http://cloud-images.ubuntu.com/releases/streams/v1/index2.sjson": dial tcp [2620:2d:4000:1::17]:80: connect: network is unreachable
terraform_data.juju_bootstrap (local-exec): 10:38:54 DEBUG juju.environs.simplestreams simplestreams.go:463 looking for data index using URL http://cloud-images.ubuntu.com/releases/streams/v1/index2.sjson
terraform_data.juju_bootstrap (local-exec): 10:38:54 INFO cmd bootstrap.go:978 bootstrap failed but --keep-broken was specified.
terraform_data.juju_bootstrap (local-exec): This means that cloud resources are left behind, but not registered to
terraform_data.juju_bootstrap (local-exec): your local client, as the controller was not successfully created.
terraform_data.juju_bootstrap (local-exec): However, you should be able to ssh into the machine using the user "ubuntu" and
terraform_data.juju_bootstrap (local-exec): their IP address for diagnosis and investigation.
terraform_data.juju_bootstrap (local-exec): When you are ready to clean up the failed controller, use your cloud console or
terraform_data.juju_bootstrap (local-exec): equivalent CLI tools to terminate the instances and remove remaining resources.

terraform_data.juju_bootstrap (local-exec): See `juju kill-controller`.
terraform_data.juju_bootstrap (local-exec): ERROR failed to bootstrap model: failed looking for image metadata in default ubuntu cloud images: cannot read index data, attempt count exceeded: cannot access URL "http://cloud-images.ubuntu.com/releases/streams/v1/index2.sjson": Get "http://cloud-images.ubuntu.com/releases/streams/v1/index2.sjson": dial tcp [2620:2d:4000:1::17]:80: connect: network is unreachable
terraform_data.juju_bootstrap (local-exec): 10:38:54 DEBUG cmd supercommand.go:549 error stack:
terraform_data.juju_bootstrap (local-exec): cannot read index data, attempt count exceeded: cannot access URL "http://cloud-images.ubuntu.com/releases/streams/v1/index2.sjson": Get "http://cloud-images.ubuntu.com/releases/streams/
v1/index2.sjson": dial tcp [2620:2d:4000:1::17]:80: connect: network is unreachable
terraform_data.juju_bootstrap (local-exec): github.com/juju/juju/environs/bootstrap.bootstrapImageMetadata:997: failed looking for image metadata in default ubuntu cloud images
terraform_data.juju_bootstrap (local-exec): github.com/juju/juju/environs/bootstrap.bootstrapIAAS:418:
terraform_data.juju_bootstrap (local-exec): github.com/juju/juju/environs/bootstrap.Bootstrap:738:
terraform_data.juju_bootstrap (local-exec): github.com/juju/juju/cmd/juju/commands.(*bootstrapCommand).Run:1088: failed to bootstrap model

If I point cloud-images.ubuntu.com/releases to my simple-streams mirror through host file and haproxy, then the bootstrapping works.

Once the first controller is up, if I try to enable HA, the same failure happens as the juju controller tries to contact cloud-images.ubuntu.com.

Revision history for this message
Bayani Carbone (bcarbone) wrote :

subscribed field high as it is affecting a customer deployment.

Revision history for this message
Bayani Carbone (bcarbone) wrote :

Are we sure this is a duplicate of https://bugs.launchpad.net/bugs/2053003?
That bug is supposed to be fixed in Juju 3.3.3 and I tested on 3.3.3 and faced the issue.
So I suppose this is either a new bug or the original bug is not solved.

Revision history for this message
Harry Pidcock (hpidcock) wrote :

try bootstrapping with --config image-metadata-defaults-disabled=true and --config container-image-metadata-defaults-disabled=true

Revision history for this message
Bayani Carbone (bcarbone) wrote :

it's now working fine, thanks!

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.