kill-controller panic after invalid credential error

Bug #1876779 reported by Tim Van Steenburgh
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Canonical Juju
Fix Released
High
Ian Booth

Bug Description

tvansteenburgh@slipgate:~$ juju kill-controller cdk-azure -y
Destroying controller "cdk-azure"
Waiting for resources to be reclaimed
Waiting on 2 models, 11 machines, 7 applications
Waiting on 2 models, 11 machines, 7 applications
Waiting on 2 models, 11 machines, 7 applications
Waiting on 2 models, 11 machines, 7 applications
Waiting on 2 models, 11 machines, 7 applications
Waiting on 2 models, 11 machines, 7 applications
Waiting on 2 models, 11 machines, 7 applications, will kill machines directly in 4m29s
Waiting on 2 models, 11 machines, 7 applications, will kill machines directly in 4m24s
Waiting on 2 models, 11 machines, 7 applications, will kill machines directly in 4m18s
Waiting on 2 models, 11 machines, 7 applications, will kill machines directly in 4m13s
Waiting on 2 models, 11 machines, 7 applications, will kill machines directly in 4m8s
Waiting on 2 models, 11 machines, 7 applications, will kill machines directly in 4m3s
Waiting on 2 models, 11 machines, 7 applications, will kill machines directly in 3m58s
Waiting on 2 models, 11 machines, 7 applications, will kill machines directly in 3m53s
Waiting on 2 models, 11 machines, 7 applications, will kill machines directly in 3m47s
Waiting on 2 models, 11 machines, 7 applications, will kill machines directly in 3m42s
Waiting on 2 models, 11 machines, 7 applications, will kill machines directly in 3m37s
Waiting on 2 models, 11 machines, 7 applications, will kill machines directly in 3m32s
Waiting on 2 models, 11 machines, 7 applications, will kill machines directly in 3m27s
Waiting on 2 models, 11 machines, 7 applications, will kill machines directly in 3m22s
Waiting on 2 models, 11 machines, 7 applications, will kill machines directly in 3m17s
Waiting on 2 models, 11 machines, 7 applications, will kill machines directly in 3m11s
Waiting on 2 models, 11 machines, 7 applications, will kill machines directly in 3m6s
Waiting on 2 models, 11 machines, 7 applications, will kill machines directly in 3m1s
Waiting on 2 models, 11 machines, 7 applications, will kill machines directly in 2m56s
Waiting on 2 models, 11 machines, 7 applications, will kill machines directly in 2m51s
Waiting on 2 models, 11 machines, 7 applications, will kill machines directly in 2m46s
Waiting on 2 models, 11 machines, 7 applications, will kill machines directly in 2m41s
Waiting on 2 models, 11 machines, 7 applications, will kill machines directly in 2m35s
Waiting on 2 models, 11 machines, 7 applications, will kill machines directly in 2m30s
Waiting on 2 models, 11 machines, 7 applications, will kill machines directly in 2m25s
Waiting on 2 models, 11 machines, 7 applications, will kill machines directly in 2m20s
Waiting on 2 models, 11 machines, 7 applications, will kill machines directly in 2m15s
Waiting on 2 models, 11 machines, 7 applications, will kill machines directly in 2m10s
Waiting on 2 models, 11 machines, 7 applications, will kill machines directly in 2m4s
Waiting on 2 models, 11 machines, 7 applications, will kill machines directly in 1m59s
Waiting on 2 models, 11 machines, 7 applications, will kill machines directly in 1m54s
Waiting on 2 models, 11 machines, 7 applications, will kill machines directly in 1m49s
Waiting on 2 models, 11 machines, 7 applications, will kill machines directly in 1m44s
Waiting on 2 models, 11 machines, 7 applications, will kill machines directly in 1m39s
Waiting on 2 models, 11 machines, 7 applications, will kill machines directly in 1m34s
Waiting on 2 models, 11 machines, 7 applications, will kill machines directly in 1m28s
Waiting on 2 models, 11 machines, 7 applications, will kill machines directly in 1m23s
Waiting on 2 models, 11 machines, 7 applications, will kill machines directly in 1m18s
Waiting on 2 models, 11 machines, 7 applications, will kill machines directly in 1m13s
Waiting on 2 models, 11 machines, 7 applications, will kill machines directly in 1m8s
Waiting on 2 models, 11 machines, 7 applications, will kill machines directly in 1m3s
Waiting on 2 models, 11 machines, 7 applications, will kill machines directly in 57s
Waiting on 2 models, 11 machines, 7 applications, will kill machines directly in 52s
Waiting on 2 models, 11 machines, 7 applications, will kill machines directly in 47s
Waiting on 2 models, 11 machines, 7 applications, will kill machines directly in 42s
Waiting on 2 models, 11 machines, 7 applications, will kill machines directly in 37s
Waiting on 2 models, 11 machines, 7 applications, will kill machines directly in 32s
Waiting on 2 models, 11 machines, 7 applications, will kill machines directly in 27s
Waiting on 2 models, 11 machines, 7 applications, will kill machines directly in 21s
Waiting on 2 models, 11 machines, 7 applications, will kill machines directly in 16s
Waiting on 2 models, 11 machines, 7 applications, will kill machines directly in 11s
Waiting on 2 models, 11 machines, 7 applications, will kill machines directly in 6s
Waiting on 2 models, 11 machines, 7 applications, will kill machines directly in 1s
Killing admin/cdk-azure directly
ERROR deleting resource group "juju-cdk-azure-model-b8481914-7ad8-4922-875c-4ad79281948c": azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://management.azure.com/subscriptions/0b63d338-88c1-4ebb-b0f3-d08eb320ea09/resourcegroups/juju-cdk-azure-model-b8481914-7ad8-4922-875c-4ad79281948c?api-version=2018-05-01: StatusCode=401 -- Original Error: adal: Refresh request failed. Status Code = '401'. Response body: {"error":"invalid_client","error_description":"AADSTS7000222: The provided client secret keys are expired. Visit the Azure Portal to create new keys for your app, or consider using certificate credentials for added security: https://aka.ms/certCreds\r\nTrace ID: 17b568be-0533-43cf-a017-cb38c6b09200\r\nCorrelation ID: 863c6917-ce45-49dc-989c-2154d3ddaf8f\r\nTimestamp: 2020-05-04 17:49:14Z","error_codes":[7000222],"timestamp":"2020-05-04 17:49:14Z","trace_id":"17b568be-0533-43cf-a017-cb38c6b09200","correlation_id":"863c6917-ce45-49dc-989c-2154d3ddaf8f","error_uri":"https://login.windows.net/error?code=7000222"}
Killing admin/default directly
ERROR deleting resource group "juju-default-model-a498d470-3c95-41d2-872e-b1af91a3cc2a": azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://management.azure.com/subscriptions/0b63d338-88c1-4ebb-b0f3-d08eb320ea09/resourcegroups/juju-default-model-a498d470-3c95-41d2-872e-b1af91a3cc2a?api-version=2018-05-01: StatusCode=401 -- Original Error: adal: Refresh request failed. Status Code = '401'. Response body: {"error":"invalid_client","error_description":"AADSTS7000222: The provided client secret keys are expired. Visit the Azure Portal to create new keys for your app, or consider using certificate credentials for added security: https://aka.ms/certCreds\r\nTrace ID: 81d39217-e385-4244-b9c3-1a702dd59800\r\nCorrelation ID: 95c310ed-c6c5-4623-b0d2-aa026450da72\r\nTimestamp: 2020-05-04 17:49:15Z","error_codes":[7000222],"timestamp":"2020-05-04 17:49:15Z","trace_id":"81d39217-e385-4244-b9c3-1a702dd59800","correlation_id":"95c310ed-c6c5-4623-b0d2-aa026450da72","error_uri":"https://login.windows.net/error?code=7000222"}
ERROR there were problems destroying some models, manual intervention may be necessary to ensure resources are released
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1b8678d]

goroutine 1 [running]:
github.com/juju/juju/cmd/juju/controller.cloudCallContext.func1(0x36fc7a7, 0x19, 0x0, 0x0)
        /build/juju/parts/juju/go/src/github.com/juju/juju/cmd/juju/controller/destroy.go:658 +0x2d
github.com/juju/juju/environs/context.(*CloudCallContext).InvalidateCredential(0xc420d31e30, 0x36fc7a7, 0x19, 0x683, 0x1)
        /build/juju/parts/juju/go/src/github.com/juju/juju/environs/context/callcontext.go:54 +0x3d
github.com/juju/juju/provider/azure/internal/errorutils.MaybeInvalidateCredential(0x3c8b6a0, 0xc420e96b40, 0x3c9a020, 0xc420d31e30, 0xc4203c1180)
        /build/juju/parts/juju/go/src/github.com/juju/juju/provider/azure/internal/errorutils/errors.go:55 +0x7d
github.com/juju/juju/provider/azure/internal/errorutils.HandleCredentialError(0x3c8b6a0, 0xc420e96b40, 0x3c9a020, 0xc420d31e30, 0x3c8b6a0, 0xc420e96b40)
        /build/juju/parts/juju/go/src/github.com/juju/juju/provider/azure/internal/errorutils/errors.go:41 +0x49
github.com/juju/juju/provider/azure.(*azureEnviron).deleteControllerManagedResourceGroups(0xc420891400, 0x3c9a020, 0xc420d31e30, 0xc421059b60, 0x24, 0x0, 0x0)
        /build/juju/parts/juju/go/src/github.com/juju/juju/provider/azure/environ.go:1667 +0xa49
github.com/juju/juju/provider/azure.(*azureEnviron).DestroyController(0xc420891400, 0x3c9a020, 0xc420d31e30, 0xc421059b60, 0x24, 0x0, 0x3269b00)
        /build/juju/parts/juju/go/src/github.com/juju/juju/provider/azure/environ.go:1649 +0x10d
github.com/juju/juju/environs.Destroy(0x7fff290fec25, 0x9, 0x7f2a7dceb790, 0xc420891400, 0x3c9a020, 0xc420d31e30, 0x7f2a7dd2bed0, 0xc4202212f0, 0x7f2a7dd302b8, 0xc420891400)
        /build/juju/parts/juju/go/src/github.com/juju/juju/environs/open.go:49 +0xda
github.com/juju/juju/cmd/juju/controller.(*killCommand).Run(0xc42081fb20, 0xc42087d040, 0x0, 0x0)
        /build/juju/parts/juju/go/src/github.com/juju/juju/cmd/juju/controller/kill.go:151 +0x655
github.com/juju/juju/cmd/modelcmd.(*controllerCommandWrapper).Run(0xc4207bbbc0, 0xc42087d040, 0xc4207bbbc0, 0xc4205aa750)
        /build/juju/parts/juju/go/src/github.com/juju/juju/cmd/modelcmd/controller.go:371 +0xda
github.com/juju/juju/cmd/modelcmd.(*baseCommandWrapper).Run(0xc420220c60, 0xc42087d040, 0x0, 0x0)
        /build/juju/parts/juju/go/src/github.com/juju/juju/cmd/modelcmd/base.go:471 +0xab
github.com/juju/juju/vendor/github.com/juju/cmd.(*SuperCommand).Run(0xc420233540, 0xc42087d040, 0xc42087d040, 0x0)
        /build/juju/parts/juju/go/src/github.com/juju/juju/vendor/github.com/juju/cmd/supercommand.go:516 +0x2cc
github.com/juju/juju/vendor/github.com/juju/cmd.Main(0x3cd9640, 0xc420233540, 0xc42087d040, 0xc4200e8050, 0x3, 0x3, 0x326b860)
        /build/juju/parts/juju/go/src/github.com/juju/juju/vendor/github.com/juju/cmd/cmd.go:379 +0x2d9
github.com/juju/juju/cmd/juju/commands.main.Run(0x385d9e0, 0xc4200e8040, 0x4, 0x4, 0x0)
        /build/juju/parts/juju/go/src/github.com/juju/juju/cmd/juju/commands/main.go:143 +0x1af
github.com/juju/juju/cmd/juju/commands.Main(0xc4200e8040, 0x4, 0x4, 0xc4200d2120)
        /build/juju/parts/juju/go/src/github.com/juju/juju/cmd/juju/commands/main.go:89 +0x4b
main.main()
        /build/juju/parts/juju/go/src/github.com/juju/juju/cmd/juju/main.go:37 +0x8c

Revision history for this message
Heather Lanigan (hmlanigan) wrote :

Client: 2.7.6-bionic-amd64
Controller: 2.5.4

Ian Booth (wallyworld)
Changed in juju:
milestone: none → 2.8-rc1
assignee: nobody → Ian Booth (wallyworld)
status: New → In Progress
importance: Undecided → High
summary: - Azure kill-controller panic
+ kill-controller panic after invalid credential error
Revision history for this message
Ian Booth (wallyworld) wrote :

The kill-controller command was not being initiaised with the callback to handle invalid creds.

Changed in juju:
status: In Progress → Fix Committed
Harry Pidcock (hpidcock)
Changed in juju:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.