Juju fails to deploy allocate node due to bad http response: 400 Bad Request

[Charles Butler]

I've deployed nexenta-edge charms as local charms and verified this issue is only reproduceable when polling the store, using juju 2.0-beta3 (possibly other versions, unverified)

This looks like some interaction with the charm store failed?

I'm not certain which interaction, however - as the charms look fine from a functional / permissions standpoint
You'll see that the permissions (thinking 401) - look similiar to the following:

perm:
  Read:
  - everyone
  Write:
  - charmers
  - lazypower

Aside from the snippet below, i've attached unit/machine logs from one of the failed units, nexentaedge-iscsi-gw here

https://gist.github.com/anonymous/522ddaab0c69f92ab7ba5b7a7f3d5962

unit-nexentaedge-iscsi-gw-0: 2016-04-19 20:34:54 ERROR juju.worker.dependency engine.go:509 "uniter" manifold worker returned unexpected error: preparing operation "install cs:~nexenta-charmers/trusty/nexentaedge-iscsi-gw-1": failed to download charm "cs:~nexenta-charmers/trusty/nexentaedge-iscsi-gw-1" from ["https://10.245.0.224:17070/model/14f42adc-d51e-4e69-8307-b61e4b17b631/charms?file=%2A&url=cs%3A~nexenta-charmers%2Ftrusty%2Fnexentaedge-iscsi-gw-1"]: cannot download "https://10.245.0.224:17070/model/14f42adc-d51e-4e69-8307-b61e4b17b631/charms?file=%2A&url=cs%3A~nexenta-charmers%2Ftrusty%2Fnexentaedge-iscsi-gw-1": bad http response: 400 Bad Request
unit-nexentaedge-mgmt-0: 2016-04-19 20:34:54 ERROR juju.worker.dependency engine.go:509 "uniter" manifold worker returned unexpected error: preparing operation "install cs:~nexenta-charmers/trusty/nexentaedge-mgmt-2": failed to download charm "cs:~nexenta-charmers/trusty/nexentaedge-mgmt-2" from ["https://10.245.0.224:17070/model/14f42adc-d51e-4e69-8307-b61e4b17b631/charms?file=%2A&url=cs%3A~nexenta-charmers%2Ftrusty%2Fnexentaedge-mgmt-2"]: cannot download "https://10.245.0.224:17070/model/14f42adc-d51e-4e69-8307-b61e4b17b631/charms?file=%2A&url=cs%3A~nexenta-charmers%2Ftrusty%2Fnexentaedge-mgmt-2": bad http response: 400 Bad Request
unit-nexentaedge-data-0: 2016-04-19 20:34:54 ERROR juju.worker.dependency engine.go:509 "uniter" manifold worker returned unexpected error: preparing operation "install cs:~nexenta-charmers/trusty/nexentaedge-data-1": failed to download charm "cs:~nexenta-charmers/trusty/nexentaedge-data-1" from ["https://10.245.0.224:17070/model/14f42adc-d51e-4e69-8307-b61e4b17b631/charms?file=%2A&url=cs%3A~nexenta-charmers%2Ftrusty%2Fnexentaedge-data-1"]: cannot download "https://10.245.0.224:17070/model/14f42adc-d51e-4e69-8307-b61e4b17b631/charms?file=%2A&url=cs%3A~nexenta-charmers%2Ftrusty%2Fnexentaedge-data-1": bad http response: 400 Bad Request
unit-nexentaedge-swift-gw-0: 2016-04-19 20:34:55 ERROR juju.worker.dependency engine.go:509 "uniter" manifold worker returned unexpected error: preparing operation "install cs:~nexenta-charmers/trusty/nexentaedge-swift-gw-1": failed to download charm "cs:~nexenta-charmers/trusty/nexentaedge-swift-gw-1" from ["https://10.245.0.224:17070/model/14f42adc-d51e-4e69-8307-b61e4b17b631/charms?file=%2A&url=cs%3A~nexenta-charmers%2Ftrusty%2Fnexentaedge-swift-gw-1"]: cannot download "https://10.245.0.224:17070/model/14f42adc-d51e-4e69-8307-b61e4b17b631/charms?file=%2A&url=cs%3A~nexenta-charmers%2Ftrusty%2Fnexentaedge-swift-gw-1": bad http response: 400 Bad Request
unit-cinder-nexentaedge-0: 2016-04-19 20:34:55 ERROR juju.worker.dependency engine.go:509 "uniter" manifold worker returned unexpected error: preparing operation "install cs:~nexenta-charmers/trusty/cinder-nexentaedge-0": failed to download charm "cs:~nexenta-charmers/trusty/cinder-nexentaedge-0" from ["https://10.245.0.224:17070/model/14f42adc-d51e-4e69-8307-b61e4b17b631/charms?file=%2A&url=cs%3A~nexenta-charmers%2Ftrusty%2Fcinder-nexentaedge-0"]: cannot download "https://10.245.0.224:17070/model/14f42adc-d51e-4e69-8307-b61e4b17b631/charms?file=%2A&url=cs%3A~nexenta-charmers%2Ftrusty%2Fcinder-nexentaedge-0": bad http response: 400 Bad Request

[Charles Butler (lazypower)]

I've found further data of whats going wrong in the controller

2016-04-19 20:53:49 ERROR juju.apiserver charms.go:114 returning error from GET /model/14f42adc-d51e-4e69-8307-b61e4b17b631/charms?%3Amodeluuid=14f42adc-d51e-4e69-8307-b61e4b17b631&file=%2A&url=cs%3A~nexenta-charmers%2Ftrusty%2Fcinder-nexentaedge-0: [{github.com/juju/juju/apiserver/charms.go:89: } {github.com/juju/juju/apiserver/charms.go:421: unable to retrieve and save the charm} {github.com/juju/juju/apiserver/charms.go:456: cannot get charm from model storage} {gopkg.in/juju/blobstore.v2/gridfs.go:46: failed to open GridFS file "a1d2da13-e1cb-4849-8f60-0a3b33da69c0"} {not found}]

For the unabridged log:
https://gist.github.com/b1f3e90a1d8d21b6326e0f88dabe2360