On wendigo we have a staging model in which we were using the 2.1.2 client, while the controller was updated to 2.5.1. Any juju client operation returns an error, like:
stg-cdo-ubuntu-esm@wendigo:~$ juju status --debug --logging-config trace
08:51:20 INFO juju.cmd supercommand.go:63 running juju [2.1.2 gc go1.6]
08:51:20 DEBUG juju.cmd supercommand.go:64 args: []string{"juju", "status", "--debug", "--logging-config", "trace"}
08:51:20 INFO juju.juju api.go:72 connecting to API addresses: [10.25.2.111:17070 10.25.2.110:17070 10.25.2.109:17070]
08:51:20 TRACE juju.api certpool.go:49 cert dir "/etc/juju/certs.d" does not exist
08:51:20 DEBUG juju.api apiclient.go:590 dialing "wss://10.25.2.111:17070/model/d937da9d-d568-4dab-8f65-43d47c25e499/api"
08:51:20 DEBUG juju.api apiclient.go:590 dialing "wss://10.25.2.110:17070/model/d937da9d-d568-4dab-8f65-43d47c25e499/api"
08:51:20 DEBUG juju.api apiclient.go:593 successfully dialed "wss://10.25.2.111:17070/model/d937da9d-d568-4dab-8f65-43d47c25e499/api"
08:51:20 DEBUG juju.api apiclient.go:593 successfully dialed "wss://10.25.2.110:17070/model/d937da9d-d568-4dab-8f65-43d47c25e499/api"
08:51:20 INFO juju.api apiclient.go:521 connection established to "wss://10.25.2.111:17070/model/d937da9d-d568-4dab-8f65-43d47c25e499/api"
08:51:20 TRACE juju.rpc.jsoncodec codec.go:225 -> {"request-id":1,"type":"Admin","version":3,"request":"Login","params":{"auth-tag":"user-stg-cdo-ubuntu-esm","credentials":"","nonce":"","macaroons":null,"user-data":""}}
08:51:20 TRACE juju.rpc.jsoncodec codec.go:120 <- {"request-id":1,"response":{"discharge-required":{"caveats":[{"cid":"eyJUaGlyZFBhcnR5UHVibGljS2V5IjoiYjFCUDJiU3YwS1hWODFscWNHVHR2YStmbHQ0L1BTZDErMnRoY3N2SDdTUT0iLCJGaXJzdFBhcnR5UHVibGljS2V5IjoiNFBhQitlK2E0SmovZ01HRkFsTVlrM0dmQS9UcnBiNC94Y1ZnNkIwLzFDcz0iLCJOb25jZSI6IlFSZ21pMGNEZEZZVnBuQzF2aVFmc0hVUERHUmJieGg1IiwiSWQiOiJEb3d4Si8vVkorOXVVck4xRHVvM1NuTTZnRUN1L2hURjdEd2s0eUJIV2FuQ1pRcVZ1OGFqTEQ2Umg2NHhqMTRoRWNBK3VsYU9NL1o0VFhlR2hFWC9lNnoxY1BYWDlqT2loY0oxUEY2Z1pRWExOeUl4bWU5a1lacEZ5Z0ZLN0hDOG9YakI4a1I0TGhZY3hDbXZXK0tqQVhoWDI4UTVYN1JSM3AyTWkvc0tySDZnd1YxdlpJVDk3T1ZaUklZPSJ9","vid":"tq3oOiZkYRXmd5bzHtxcnDgGEevEG2SEysI9eRtjskFp1acYNKXvlYfILPgboM7FkllBQcsaUozuVUPH6H4p00BI3UWKkumA","cl":"https://10.25.2.111:17070/auth"},{"cid":"time-before 2019-03-20T08:51:20.930842434Z"}],"location":"juju model 4da59b22-9710-4e69-840a-be49ee864a97","identifier":"114c572c217de4c40a138b91e9d56452-623dd22a48a1fac7769453ba68922591","signature":"6b9979e3c7a171f83341e579be3166a1e384eaf1ddf8e5c81321a40d05478640"},"discharge-required-error":"verification failed: no macaroons"}}
08:51:20 DEBUG httpbakery client.go:592 postFormJSON to https://10.25.2.111:17070/auth/discharge; vals: url.Values{"id":[]string{"eyJUaGlyZFBhcnR5UHVibGljS2V5IjoiYjFCUDJiU3YwS1hWODFscWNHVHR2YStmbHQ0L1BTZDErMnRoY3N2SDdTUT0iLCJGaXJzdFBhcnR5UHVibGljS2V5IjoiNFBhQitlK2E0SmovZ01HRkFsTVlrM0dmQS9UcnBiNC94Y1ZnNkIwLzFDcz0iLCJOb25jZSI6IlFSZ21pMGNEZEZZVnBuQzF2aVFmc0hVUERHUmJieGg1IiwiSWQiOiJEb3d4Si8vVkorOXVVck4xRHVvM1NuTTZnRUN1L2hURjdEd2s0eUJIV2FuQ1pRcVZ1OGFqTEQ2Umg2NHhqMTRoRWNBK3VsYU9NL1o0VFhlR2hFWC9lNnoxY1BYWDlqT2loY0oxUEY2Z1pRWExOeUl4bWU5a1lacEZ5Z0ZLN0hDOG9YakI4a1I0TGhZY3hDbXZXK0tqQVhoWDI4UTVYN1JSM3AyTWkvc0tySDZnd1YxdlpJVDk3T1ZaUklZPSJ9"}, "location":[]string{"juju model 4da59b22-9710-4e69-840a-be49ee864a97"}}
08:51:20 DEBUG httpbakery client.go:244 client do POST https://10.25.2.111:17070/auth/discharge {
08:51:20 DEBUG httpbakery client.go:246 } -> error [{/build/juju-core-w9qlrW/juju-core-2.1.2/src/gopkg.in/macaroon-bakery.v1/httpbakery/client.go:283: } {cannot discharge: interaction required}]
08:51:20 DEBUG httpbakery client.go:244 client do GET https://10.25.2.111:17070/auth/login?waitid=c7744762f24c5cc89126c46c {
08:51:20 DEBUG httpbakery client.go:246 } -> error <nil>
please enter password for stg-cdo-ubuntu-esm on prodstack-is:
08:51:34 TRACE juju.rpc.jsoncodec codec.go:225 -> {"request-id":2,"type":"Admin","version":3,"request":"Login","params":{"auth-tag":"user-stg-cdo-ubuntu-esm","credentials":"","nonce":"","macaroons":[[{"caveats":[{"cid":"eyJUaGlyZFBhcnR5UHVibGljS2V5IjoiYjFCUDJiU3YwS1hWODFscWNHVHR2YStmbHQ0L1BTZDErMnRoY3N2SDdTUT0iLCJGaXJzdFBhcnR5UHVibGljS2V5IjoiNFBhQitlK2E0SmovZ01HRkFsTVlrM0dmQS9UcnBiNC94Y1ZnNkIwLzFDcz0iLCJOb25jZSI6IlFSZ21pMGNEZEZZVnBuQzF2aVFmc0hVUERHUmJieGg1IiwiSWQiOiJEb3d4Si8vVkorOXVVck4xRHVvM1NuTTZnRUN1L2hURjdEd2s0eUJIV2FuQ1pRcVZ1OGFqTEQ2Umg2NHhqMTRoRWNBK3VsYU9NL1o0VFhlR2hFWC9lNnoxY1BYWDlqT2loY0oxUEY2Z1pRWExOeUl4bWU5a1lacEZ5Z0ZLN0hDOG9YakI4a1I0TGhZY3hDbXZXK0tqQVhoWDI4UTVYN1JSM3AyTWkvc0tySDZnd1YxdlpJVDk3T1ZaUklZPSJ9","vid":"tq3oOiZkYRXmd5bzHtxcnDgGEevEG2SEysI9eRtjskFp1acYNKXvlYfILPgboM7FkllBQcsaUozuVUPH6H4p00BI3UWKkumA","cl":"https://10.25.2.111:17070/auth"},{"cid":"time-before 2019-03-20T08:51:20.930842434Z"}],"location":"juju model 4da59b22-9710-4e69-840a-be49ee864a97","identifier":"114c572c217de4c40a138b91e9d56452-623dd22a48a1fac7769453ba68922591","signature":"6b9979e3c7a171f83341e579be3166a1e384eaf1ddf8e5c81321a40d05478640"},{"caveats":[{"cid":"declared username stg-cdo-ubuntu-esm"},{"cid":"time-before 2019-03-20T08:51:34.981839799Z"}],"location":"","identifier":"eyJUaGlyZFBhcnR5UHVibGljS2V5IjoiYjFCUDJiU3YwS1hWODFscWNHVHR2YStmbHQ0L1BTZDErMnRoY3N2SDdTUT0iLCJGaXJzdFBhcnR5UHVibGljS2V5IjoiNFBhQitlK2E0SmovZ01HRkFsTVlrM0dmQS9UcnBiNC94Y1ZnNkIwLzFDcz0iLCJOb25jZSI6IlFSZ21pMGNEZEZZVnBuQzF2aVFmc0hVUERHUmJieGg1IiwiSWQiOiJEb3d4Si8vVkorOXVVck4xRHVvM1NuTTZnRUN1L2hURjdEd2s0eUJIV2FuQ1pRcVZ1OGFqTEQ2Umg2NHhqMTRoRWNBK3VsYU9NL1o0VFhlR2hFWC9lNnoxY1BYWDlqT2loY0oxUEY2Z1pRWExOeUl4bWU5a1lacEZ5Z0ZLN0hDOG9YakI4a1I0TGhZY3hDbXZXK0tqQVhoWDI4UTVYN1JSM3AyTWkvc0tySDZnd1YxdlpJVDk3T1ZaUklZPSJ9","signature":"8a4dfae6a59555db16efc51466df51576210d76b35b4dc34532965662510cc2e"}]],"user-data":""}}
08:51:35 TRACE juju.rpc.jsoncodec codec.go:123 <- error: unexpected end of JSON input (closing false)
08:51:35 ERROR cmd supercommand.go:458 codec.ReadHeader error: error receiving message: unexpected end of JSON input
08:51:35 DEBUG cmd supercommand.go:459 (error details: [{github.com/juju/juju/cmd/juju/status/status.go:139: } {github.com/juju/juju/cmd/modelcmd/modelcommand.go:171: } {github.com/juju/juju/juju/api.go:77: } {github.com/juju/juju/api/apiclient.go:242: } {github.com/juju/juju/api/state.go:109: } {github.com/juju/juju/api/apiclient.go:670: } {github.com/juju/retry/retry.go:187: } {github.com/juju/juju/rpc/client.go:149: } {github.com/juju/juju/rpc/server.go:354: codec.ReadHeader error} {github.com/juju/juju/rpc/jsoncodec/codec.go:131: error receiving message} {unexpected end of JSON input}])
Logging out and in again does not help. Updating the client version fixes the issue.
Tim reminded me that this is expected with performance improvements that went into 2.5.1. Clients older than 2.2 will not be able to handle the websocket buffer sizes changed in:
https:/
This is a won't fix as the correct path forward is to update to a client 2.2 or greater. This has been done to work around this situation and is required as a path to improved performance and goes back far enough we felt ok making this change for clients.