Resources can't be downloaded

[Charles Butler]

As a thought experiment I bootstrapped a manual controller, enlisted 2 additional machines and attempted to deploy the Canonical Distribution of Kubernetes.

After deploying the charms from the store, it didn't appear that the resources made it to the charms. I attributed this due to beta characteristics of resources. When attempting to manually attach the resources however this did not function as expected.

$ juju attach kubernetes-master kubernetes=kubernetes-master.tar.gz
ERROR failed to upload resource "kubernetes": PUT https://198.211.98.33:17070/model/0c3e7734-2a24-452d-83cd-493bcc162852/applications/kubernetes-master/resources/kubernetes: Put https://198.211.98.33:17070/model/0c3e7734-2a24-452d-83cd-493bcc162852/applications/kubernetes-master/resources/kubernetes: write tcp 10.0.5.36:45190->198.211.98.33:17070: write: connection reset by peer

$ juju attach kubernetes-master kubernetes=kubernetes-master.tar.gz --debug
01:31:55 INFO juju.cmd supercommand.go:63 running juju [2.0-rc1 gc go1.6.2]
01:31:55 DEBUG juju.cmd supercommand.go:64 args: []string{"juju", "attach", "kubernetes-master", "kubernetes=kubernetes-master.tar.gz", "--debug"}
01:31:55 INFO juju.juju api.go:72 connecting to API addresses: [198.211.98.33:17070 [2604:a880:400:d0::a79:e001]:17070 10.10.0.5:17070 10.136.25.134:17070]
01:31:55 INFO juju.api apiclient.go:507 dialing "wss://198.211.98.33:17070/model/0c3e7734-2a24-452d-83cd-493bcc162852/api"
01:31:55 INFO juju.api apiclient.go:507 dialing "wss://[2604:a880:400:d0::a79:e001]:17070/model/0c3e7734-2a24-452d-83cd-493bcc162852/api"
01:31:55 INFO juju.api apiclient.go:507 dialing "wss://10.10.0.5:17070/model/0c3e7734-2a24-452d-83cd-493bcc162852/api"
01:31:55 INFO juju.api apiclient.go:507 dialing "wss://10.136.25.134:17070/model/0c3e7734-2a24-452d-83cd-493bcc162852/api"
01:31:55 INFO juju.api apiclient.go:302 connection established to "wss://198.211.98.33:17070/model/0c3e7734-2a24-452d-83cd-493bcc162852/api"
01:31:55 DEBUG juju.juju api.go:263 API hostnames unchanged - not resolving
01:31:56 DEBUG httpbakery client.go:244 client do PUT https://198.211.98.33:17070/model/0c3e7734-2a24-452d-83cd-493bcc162852/applications/kubernetes-master/resources/kubernetes {
01:31:56 DEBUG httpbakery client.go:246 } -> error <nil>
01:31:56 DEBUG juju.api apiclient.go:558 health ping failed: connection is shut down
01:31:56 ERROR cmd supercommand.go:458 failed to upload resource "kubernetes": PUT https://198.211.98.33:17070/model/0c3e7734-2a24-452d-83cd-493bcc162852/applications/kubernetes-master/resources/kubernetes: resource#kubernetes-master/kubernetes not found
01:31:56 DEBUG cmd supercommand.go:459 (error details: [{github.com/juju/juju/resource/cmd/upload.go:118: failed to upload resource "kubernetes"} {github.com/juju/juju/resource/cmd/upload.go:132: } {github.com/juju/juju/resource/api/client/client.go:99: } {/build/juju-core-uGAy74/juju-core-2.0-rc1/src/github.com/juju/httprequest/client.go:232: PUT https://198.211.98.33:17070/model/0c3e7734-2a24-452d-83cd-493bcc162852/applications/kubernetes-master/resources/kubernetes} {resource#kubernetes-master/kubernetes not found}])

I was able to upgrade the charm using a a version received from the charm store

charm pull cs:~containers/kubernetes-master-2 and push this to the model.

Afterwords I was able to append the resources. It seems like it just completely missed processing the resources initially, which in turn caused problems.

I don't see anything relevant in the controller logs. Advice on how to further diagnose and report helpful findings would be helpful.

[Nate Finch (natefinch)]

This is not just manual, it's presumably due to a change in the charm store, which is no longer sending the content length.

ERROR juju.resource.resourceadapters (attempt 14) retrying resource download from charm store due to error: no content length found in response

[Nate Finch (natefinch)]

Seems to be a charmstore problem, doesn't happen for already-zipped resources like
https://api.jujucharms.com/charmstore/v5/~containers/kubernetes-master-6/resource/kubernetes/4

but does happen with plaintext resources like
https://api.jujucharms.com/charmstore/v5/~natefinch/starsay-4/resource/data/0

An easy test:

go get npf.io/gorram
gorram -i {{.ContentLength}} net/http Get <url>

Should print the content length, but for resources that aren't gzipped, it prints -1.

you can also use

gorram -i '{{printf "%#v" .}}' net/http Get <url>

to show the full response data (minus body)

Note that curl seems to do some extra processing to get the correct content-length, even though it's not actually in the response.

[Nate Finch (natefinch)]

in conversation with the charmstore team, it seems like this is an expected response. We don't technically need to check the content-length, since we can check the fingerprint, and that would change if the length changed.

[Nate Finch (natefinch)]

Fixed here: https://github.com/juju/juju/pull/6572