bootstrap on gke fails and unable to cancel bootstrap process

Bug #1884266 reported by Richard Harding
This bug report is a duplicate of:  Bug #1840367: Impossible to cancel bootstrap. Edit Remove
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Canonical Juju
Triaged
High
Unassigned

Bug Description

juju bootstrap gke-k8s gke-controller
Creating Juju controller "gke-controller" on gke-k8s/us-central1
Creating k8s resources for controller "controller-gke-controller"
^CInterrupt signalled: waiting for bootstrap to exit
j^CInterrupt signalled: waiting for bootstrap to exit
^CInterrupt signalled: waiting for bootstrap to exit
^CInterrupt signalled: waiting for bootstrap to exit
^CInterrupt signalled: waiting for bootstrap to exit
^CInterrupt signalled: waiting for bootstrap to exit
^CInterrupt signalled: waiting for bootstrap to exit
^CInterrupt signalled: waiting for bootstrap to exit
^CInterrupt signalled: waiting for bootstrap to exit
^CInterrupt signalled: waiting for bootstrap to exit
^CInterrupt signalled: waiting for bootstrap to exit
^CInterrupt signalled: waiting for bootstrap to exit
^CInterrupt signalled: waiting for bootstrap to exit
^CInterrupt signalled: waiting for bootstrap to exit
^CInterrupt signalled: waiting for bootstrap to exit
^CInterrupt signalled: waiting for bootstrap to exit
^CInterrupt signalled: waiting for bootstrap to exit
^CInterrupt signalled: waiting for bootstrap to exit
^CInterrupt signalled: waiting for bootstrap to exit
^CInterrupt signalled: waiting for bootstrap to exit
^CInterrupt signalled: waiting for bootstrap to exit
^CInterrupt signalled: waiting for bootstrap to exit
^CInterrupt signalled: waiting for bootstrap to exit
^CInterrupt signalled: waiting for bootstrap to exit
^CInterrupt signalled: waiting for bootstrap to exit
^CInterrupt signalled: waiting for bootstrap to exit
^CInterrupt signalled: waiting for bootstrap to exit
^CInterrupt signalled: waiting for bootstrap to exit
^CInterrupt signalled: waiting for bootstrap to exit
^CInterrupt signalled: waiting for bootstrap to exit
^CInterrupt signalled: waiting for bootstrap to exit
^CInterrupt signalled: waiting for bootstrap to exit
^CInterrupt signalled: waiting for bootstrap to exit
^CInterrupt signalled: waiting for bootstrap to exit
^CInterrupt signalled: waiting for bootstrap to exit
^CInterrupt signalled: waiting for bootstrap to exit
^CInterrupt signalled: waiting for bootstrap to exit
^CInterrupt signalled: waiting for bootstrap to exit

I had to kill it and now it's left some cruft behind to manually clean up.
ERROR controller "gke-controller" already exists
10:58:14 DEBUG cmd supercommand.go:537 error stack:
/build/snapcraft-juju-03af7d/parts/juju/src/environs/bootstrap/prepare.go:87: controller "gke-controller" already exists

When I went into the pods there was an error about a "unschedulable" pod and in that pod "controller-0" there was an error that it was unschedule-able because "insufficient memory".

I used the default cluster setup so perhaps there's missing documentation and error handling that helps detect this situation?

I created another node-pool with 8gb of ram to try to test it out with.

Revision history for this message
Richard Harding (rharding) wrote :

I've unregistered the controller locally, removed the workloads, service and ingress, and storage from the gke webui. It still won't let me reuse the controller name so trying to find the list of what it created.

Revision history for this message
Richard Harding (rharding) wrote :

I did get it to bootstrap after adding a new node pool with 8gb of ram instances. It seems our default needs don't jive with the gke default cluster set up.

Revision history for this message
Ian Booth (wallyworld) wrote :

I've found on all clouds that ^C sometimes isn't able to interrupt the bootstrap process. I can't recall or find an existing bug right now.

I think in the past I've just run gcloud container clusters create and the defaults have worked. If that's no longer the case, we should document that.

I'll add the bug to a milestone as we at least should verify what works at the moment.

Changed in juju:
milestone: none → 2.8-next
status: New → Triaged
importance: Undecided → High
Revision history for this message
Heather Lanigan (hmlanigan) wrote :

We have a pile of bugs on unable to cancel bootstrap, the current count with this one is now 6? See 1840367. Not marking as a duplicate as there may be addition work to be done here.

Revision history for this message
Ian Booth (wallyworld) wrote :

Marking as dupe as bootstrap improvements being done as a separate piece of feature work.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.